Control device and control method

Abstract

To provide a control apparatus that can be controlled so as to obtain a desired image when changing a subject in photographing by a plurality of cameras.SOLUTION: A control apparatus includes: acquisition means of acquiring information indicating a photographing object of a first imaging device; and control means of controlling a photographing direction of a plurality of second imaging devices so as to be directed to the photographing object corresponded to the photographing object of the first imaging device. The control means does not change a photographing direction of the second imaging device included in a first group from a direction directed to a third subject corresponded to the first subject from the plurality of second imaging devices in the case where the photographing direction of the first imaging device is changed to a direction directed to a second subject from a direction directed to the first subject, and changes the photographing direction of the second imaging device included in a second group from the plurality of second imaging devices from a direction directed to the third subject to a direction directed to a fourth subject corresponded to the second subject.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a control device and a control method for controlling photography by a plurality of cameras. [Background technology]

[0002] In recent years, systems that use multiple cameras for photography have become known. By arranging a main camera and multiple sub-cameras different from the main camera among the multiple cameras, it becomes possible to obtain multiple shooting angles. In photography using multiple cameras, there is a demand for reducing the number of cameramen required to operate the cameras in order to reduce photography costs.

[0003] To reduce manpower, it is possible to use cameras called PTZ cameras, which allow remote adjustment of pan, tilt, and zoom. PTZ cameras use technology to automatically track targets detected from captured images, and can automatically control pan, tilt, and zoom so that the target is positioned as desired within the camera's field of view.

[0004] As a technology for automatic control of multiple cameras, Patent Document 1 discloses a technology for determining the shooting direction of a camera other than a parent camera based on the position of a subject being photographed by the parent camera. Patent Document 2 discloses a technology for transmitting subject information from a main imaging device to a sub-imaging device, which then automatically photographs based on the subject information. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-348428 [Patent Document 2] Japanese Patent Publication No. 2020-025248 Summary of the Invention [Problem to be solved by the invention]

[0006] However, while the subject changes on the main camera and each camera changes direction, it may not be possible to obtain an image of the desired subject. For example, when shooting a live event, while the shooting direction of multiple sub-cameras is changed in accordance with the change in subject on the main camera, the images of each sub-camera are moving images, and the user may not be able to obtain the image they want.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a control device that can perform control so as to obtain a desired image when changing the subject in photographing using a plurality of cameras. [Means for solving the problem]

[0008] The control device according to the present invention includes an acquisition means for acquiring information indicating a subject of imaging of a first imaging device, and a control means for controlling the imaging direction of a plurality of second imaging devices so that the second imaging devices are oriented toward a subject of imaging corresponding to the subject of imaging of the first imaging device, and the control means controls the imaging direction of the first imaging device when the imaging direction of the first imaging device is changed from a direction facing a first subject to a direction facing a second subject. ,before Among the plurality of second imaging devices, the imaging directions of the second imaging devices included in a second group of , corresponding to the first subject Change the direction from facing the third subject to facing the fourth subject corresponding to the second subject. and then changing the imaging direction of the second imaging devices included in the first group from a direction facing the third subject to a direction facing the fourth subject. It is characterized by: [Effects of the Invention]

[0009] According to the present invention, when changing the subject in photographing with a plurality of cameras, the desired image is It can be controlled to obtain [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an imaging system. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a control device and an overhead camera. [Figure 3]FIG. 2 is a diagram illustrating an example of the configuration of a sub camera, a main camera, and a role control device. [Figure 4] FIG. 2 is a block diagram illustrating a functional configuration of a control device. [Figure 5] FIG. 10 is a diagram illustrating role information. [Figure 6] FIG. 10 is a diagram illustrating transformation of a coordinate system. [Figure 7] FIG. 10 is a diagram illustrating a method for detecting the position of a subject. [Figure 8] 10A and 10B are diagrams showing specific examples of changing the subject to be photographed by the sub-camera. [Figure 9] FIG. 10 is a diagram illustrating grouping of sub-cameras. [Figure 10] FIG. 10 is a diagram illustrating a target position of panning. [Figure 11] FIG. 10 is a diagram illustrating a target tilt position. [Figure 12A] 10 is a flowchart illustrating a control process for the imaging direction of the sub-camera. [Figure 12B] 10 is a flowchart illustrating a control process for the imaging direction of the sub-camera. DETAILED DESCRIPTION OF THE INVENTION

[0011] <Embodiment> Hereinafter, a photography system according to an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram illustrating the configuration of the photography system. The photography system includes a control device 100, an overhead camera 300, multiple sub-cameras 400, a main camera 500, and a role control device 600. Each component of the photography system is connected via, for example, a LAN (Local Area Network) 700, and each device can communicate with each other.

[0012] The overhead camera 300 and the main camera 500 transmit images of the subject to the control device 100 via a video line (not shown). The overhead camera 300 and the main camera 500 also transmit imaging information to the control device 100 via the LAN 700. The overhead camera 300 is a camera that captures images with a wide angle of view so that the imaging environment can be captured from a bird's-eye view. The overhead camera 300 may be a PTZ camera. The main camera 500 is a camera that is manually operated by the user to capture images, or a PTZ camera. The sub-camera 400 is a PTZ camera that operates in cooperation with the main camera 500 and is driven based on pan, tilt, and zoom values ​​calculated by the control device 100. The imaging system includes multiple sub-cameras 400.

[0013] The control device 100 controls the shooting directions of the multiple sub-cameras 400 so that they face a target corresponding to the target of shooting of the main camera 500. The control device 100 calculates pan, tilt, and zoom values ​​for driving the sub-cameras 400 using role information set by the role control device 600 and information acquired from images shot by the overhead camera 300 and the main camera 500.

[0014] The role information is information about the subject of the sub-camera 400 that corresponds to the subject that is the subject of photography by the main camera 500. The role information may include information about whether the camera included in the photography system is the main camera 500 or the sub-camera 400. The information acquired from the image captured by the overhead camera 300 includes position information about the main camera 500 and the sub-camera 400. The information acquired from the image captured by the main camera 500 includes information about the subject that is the subject of photography by the main camera 500. The control device 100 transmits the calculated pan / tilt / zoom values ​​to the sub-camera 400. The control device 100 is a workstation, an edge AI device, or the like.

[0015] The role control device 600 controls the sub-camera 4 which is controlled in conjunction with the operation of the main camera 500. The role control device 600 sets the role of the sub-camera 400. The role control device 600 sets the role of the sub-camera 400 as to which subject the sub-camera 400 is to capture, corresponding to the subject that is the subject of the main camera 500's capture. The role control device 600 can set the role of the sub-camera 400, for example, in response to a user's instruction. The role control device 600 is an electronic device such as a laptop PC (personal computer) or a switch controller (device that can assign macros to buttons). The user can set the role of the sub-camera 400, for example, via a user interface (UI) of the laptop PC or by operating the buttons on the switch controller.

[0016] The configurations of the control device 100, overhead camera 300, sub-camera 400, main camera 500, and role control device 600 will be described in detail using FIGS. 2(A), 2(B), and 3(A) to 3(C).

[0017] 2(A) is a diagram showing an example of the configuration of the control device 100. The control device 100 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, and a ROM (Read Only Memory) 103. The control device 100 also includes an inference unit 104, a network interface (I / F) 105, and an input unit 106. The components included in the control device 100 are connected to each other via an internal bus 110.

[0018] The CPU 101 controls the entire control device 100. The RAM 102 is a high-speed storage device such as a DRAM (Dynamic Random Access Memory). The CPU 101 temporarily loads the OS, various programs, and various data into the RAM 102 and executes various processes. The RAM 102 is also used as a work area when the CPU 101 executes the OS and various programs.

[0019] The ROM 103 is a non-volatile storage device such as a flash memory, HDD, SSD, or SD card. The ROM 103 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0020] The inference unit 104 reads images captured by the overhead camera 300, the sub-camera 400, and the main camera 500 from the RAM 102, and estimates the position and presence or absence of objects such as the camera and the subject from the read-out captured images. The inference unit 104 is, for example, a computing device specialized for image processing and inference processing, such as a GPU (Graphics Processing Unit). A GPU is effective as a computing device used for learning processing. The inference unit 104 may be a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array). The processing of the inference unit 104 may also be implemented by the CPU 101.

[0021] The network I / F 105 is an interface for connecting to the LAN 700, and controls communication with external devices such as the sub-camera 400 via a communication medium such as Ethernet (registered trademark). The network I / F 105 may also be a serial communication interface.

[0022] 2(B) is a diagram showing an example of the configuration of overhead camera 300. Overhead camera 300 includes a CPU 301, RAM 302, ROM 303, network I / F 305, image processing unit 306, and image sensor 307. The components included in overhead camera 300 are connected to each other via internal bus 310.

[0023] The CPU 301 controls the entire overhead camera 300. The RAM 302 is a memory such as a DRAM. It is a high-speed storage device. The CPU 301 temporarily loads the OS, various programs, and various data into the RAM 102 and executes various processes. The RAM 302 is also used as a work area when the CPU 301 executes the OS and various programs.

[0024] The ROM 303 is a non-volatile storage device such as a flash memory, HDD, SSD, or SD card. The ROM 303 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0025] The image processing unit 306 is connected to an image sensor 307 such as a CCD or CMOS. The image processing unit 306 converts image data acquired from the image sensor 307 into a predetermined format and transfers the converted image data to the RAM 302. The image processing unit 306 may compress the converted image data and transfer it to the RAM 302.

[0026] The image processing unit 306 can perform image processing such as color correction, exposure control, and sharpness correction on the image when acquiring the image from the image sensor 307. The image processing unit 306 may also perform cropping to cut out a partial area of ​​the image data. The processing by the image processing unit 306 may be performed based on instructions received from an external device such as the control device 100 via the network I / F 305.

[0027] The network I / F 305 is an interface for connecting to the LAN 700, and controls communication with external devices such as the control device 100 via a communication medium such as Ethernet (registered trademark). The network I / F 305 may be a serial communication interface or the like.

[0028] The photography system includes multiple sub-cameras 400, and the main camera 500 takes pictures in cooperation with the multiple sub-cameras 400. FIG. 3(A) is a diagram showing an example of the configuration of the sub-camera 400. Each of the multiple sub-cameras 400 included in the photography system has the same configuration as that shown in FIG. 3(A). The sub-camera 400 includes a CPU 401, RAM 402, ROM 403, a network I / F 405, an image processing unit 406, an image sensor 407, a drive I / F 408, a drive unit 409, and a focus control unit 411. The components included in the sub-camera 400 are connected to each other via an internal bus 410.

[0029] The CPU 401 controls the entire sub-camera 400. The RAM 402 is a high-speed storage device such as a DRAM. The CPU 401 temporarily loads the OS, various programs, and various data into the RAM 402 and executes various processes. The RAM 402 is also used as a work area when the CPU 401 executes the OS and various programs.

[0030] The ROM 403 is a non-volatile storage device such as a flash memory, HDD, SSD, or SD card. The ROM 403 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0031] The image processing unit 406 is connected to an image sensor 407 such as a CCD or CMOS. The image processing unit 406 converts image data acquired from the image sensor 407 into a predetermined format and transfers the converted image data to the RAM 402. The image processing unit 406 may compress the converted image data and transfer it to the RAM 402.

[0032] When acquiring an image from the image sensor 407, the image processing unit 406 can perform image processing such as color correction, exposure control, and sharpness correction on the image. The image processing unit 406 may also perform crop processing to cut out a partial area of ​​the image data. The processing by the image processing unit 406 is performed in response to a request from a device such as the control device 100 via the network I / F 405. It may be performed based on instructions received from an external device.

[0033] The network I / F 405 is an interface for connecting to the LAN 700, and controls communication with external devices such as the control device 100 via a communication medium such as Ethernet (registered trademark). The network I / F 405 may be a serial communication interface or the like.

[0034] The drive I / F 408 is a connection unit with the drive unit 409, and transmits and receives control signals and the like to the drive unit 409. The drive unit 409 is a rotation mechanism for changing the shooting direction of the sub-camera 400, and includes a mechanical drive system, a drive motor, and the like. Based on instructions received from the CPU 401 via the drive I / F 408, the drive unit 409 drives rotations such as a pan operation that changes the shooting angle of view in the horizontal direction and a tilt operation that changes the shooting angle of view in the vertical direction, and a zoom operation that optically changes the shooting angle of view.

[0035] A focus control unit 411 controls the focus when the image sensor 407 acquires a captured image. As a focusing method, known techniques such as a contrast detection method or a phase difference detection method can be used.

[0036] FIG. 3(B) is a diagram showing an example of the configuration of main camera 500. In the following description, it is assumed that main camera 500 is manually operated by a user. Specifically, the user can use a camera controller (not shown) to perform PTZ control of main camera 500. Main camera 500 is not limited to a PTZ camera, and may be a camera configured to capture images by being manually and physically moved by the user.

[0037] The control device 100 can acquire orientation information of the main camera 500 (angles of pan and tilt based on the angle when the camera is installed facing directly at the camera) via the network I / F 505 and the LAN 700.

[0038] Main camera 500 includes a CPU 501, RAM 502, ROM 503, network I / F 505, image processing unit 506, image sensor 507, drive I / F 508, and drive unit 509. The components included in main camera 500 are connected to each other via internal bus 510.

[0039] The CPU 501 controls the entire main camera 500. The RAM 502 is a high-speed storage device such as a DRAM. The CPU 501 temporarily loads the OS, various programs, and various data into the RAM 502 and executes various processes. The RAM 502 is also used as a work area when the CPU 501 executes the OS and various programs.

[0040] The ROM 503 is a non-volatile storage device such as a flash memory, HDD, SSD, or SD card. The ROM 503 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0041] The image processing unit 506 is connected to an image sensor 507 such as a CCD or CMOS. The image processing unit 506 converts image data acquired from the image sensor 507 into a predetermined format and transfers the converted image data to the RAM 502. The image processing unit 506 may compress the converted image data and transfer it to the RAM 502.

[0042] The image processing unit 506 can perform image processing such as color correction, exposure control, and sharpness correction on the image when acquiring the image from the image sensor 507. The image processing unit 506 may also perform cropping to cut out a partial area of ​​the image data. The processing by the image processing unit 506 may be performed based on an instruction received from an external device such as the control device 100 via the network I / F 505.

[0043] The network I / F 505 is an interface for connecting to the LAN 700, and controls communication with external devices such as the control device 100 via a communication medium such as Ethernet (registered trademark). The network I / F 505 may be a serial communication interface or the like.

[0044] Drive I / F 508 is a connection unit with drive unit 509, and transmits and receives control signals and the like to drive unit 509. Drive unit 509 is a rotation mechanism for changing the shooting direction of sub-camera 400, and includes a mechanical drive system, a drive source motor, and the like. Based on instructions received from CPU 501 via drive I / F 508, drive unit 509 drives rotations such as panning to change the shooting angle of view in the horizontal direction and tilting to change the shooting angle of view in the vertical direction, and zooming to optically change the shooting angle of view.

[0045] 3(C) is a diagram showing an example of the configuration of the role control device 600. The role control device 600 includes a CPU 601, a RAM 602, a ROM 603, a network I / F 605, and an input unit 611. The components included in the role control device 600 are connected to each other via an internal bus 610.

[0046] The CPU 601 controls the entire role control device 600. The RAM 602 is a high-speed storage device such as a DRAM. The CPU 601 temporarily loads the OS, various programs, and various data into the RAM 602 and executes various processes. The RAM 602 is also used as a work area when the CPU 601 executes the OS and various programs.

[0047] The ROM 603 is a non-volatile storage device such as a flash memory, HDD, SSD, or SD card. The ROM 603 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0048] The network I / F 605 is an interface for connecting to the LAN 700, and controls communication with external devices such as the control device 100 via a communication medium such as Ethernet (registered trademark). The network I / F 605 may be a serial communication interface or the like.

[0049] The input unit 611 includes operation members such as a button, a mouse, and a keyboard. The user can input role information by controlling the GUI using the operation members. The CPU 601 transmits the role information designated by the user to the control device 100. Note that the role control device 600 may be configured as a single device together with the control device 100.

[0050] Specific processing by the control device 100 will be described with reference to Fig. 4. The control device 100 calculates the pan / tilt / zoom values ​​for each of the multiple sub-cameras 400 and controls the shooting direction of each sub-camera 400 so that the sub-cameras 400 face a subject that corresponds to the subject that the main camera 500 is shooting. In the example of Fig. 4, the multiple sub-cameras 400 are sub-camera 400a and sub-camera 400b, and are also collectively referred to as sub-camera 400.

[0051] Fig. 4 is a block diagram illustrating an example of the functional configuration of the control device 100. The processing of each functional unit of the control device 100 is realized using the hardware resources shown in Fig. 2(A). A detailed description of the processing of software that controls communication functions and general-purpose software such as an operating system will be omitted.

[0052] Control device 100 includes role setting unit 120, first recognition unit 121, main subject determination unit 122, tracking subject determination unit 123, shooting direction acquisition unit 124, and second recognition unit 125. Programs for realizing the processing of each functional unit are stored in ROM 103. CPU 101 can realize the processing of each functional unit by reading the programs stored in ROM 103 into RAM 102 and executing them.

[0053] The role setting unit 120 sets a subject to be photographed by the sub camera 400 based on the role information set by the role control device 600. The role information includes information on the subject to be photographed by the sub camera 400 (tracking subject) corresponding to the subject to be photographed by the main camera 500 (main subject).

[0054] The user can set role information via the input unit 611 of the role control device 600. Specifically, the user sets the subject to be photographed by the main camera 500 and the subject to be photographed by the sub-camera 400 that corresponds to the subject to be photographed by the main camera 500, by operating an operation member such as a keyboard on the GUI of the role control device 600.

[0055] Fig. 5 is a diagram illustrating role information. A user can register multiple patterns of role information in advance in the role control device 600 as shown in Fig. 5. For example, in pattern 1, the subject captured by the sub-camera 400 corresponding to subject A, the subject captured before the change of the main camera 500, is subject A. Also, the subject captured by the sub-camera 400 corresponding to subject B, the subject captured after the change of the main camera 500, is subject B. Pattern 1 is a pattern in which the subject captured by the main camera 500 and the subject captured by the sub-camera 400 are the same.

[0056] On the other hand, in pattern 5, the subject captured by sub camera 400 corresponding to subject A, which is the subject captured by main camera 500 before the change, is subject B. Also, the subject captured by sub camera 400 corresponding to subject C, which is the subject captured by main camera 500 after the change, is subject D. Pattern 5 is a pattern in which the subject captured by main camera 500 and the subject captured by sub camera 400 are different.

[0057] Furthermore, as in patterns 1 and 2, the subject of the main camera 500 before the change and the subject of the sub-camera 400 before the change may be the same subject. As in patterns 4 and 7, the subject of the main camera 500 before the change and the subject of the sub-camera 400 after the change may be the same subject. As in patterns 4 and 6, the subject of the main camera 500 after the change and the subject of the sub-camera 400 before the change may be the same subject. As in patterns 1 and 3, the subject of the main camera 500 after the change and the subject of the sub-camera 400 after the change may be the same subject.

[0058] As in patterns 3 to 7, the subject of main camera 500 before the change and the subject of sub camera 400 before the change may be different subjects. As in patterns 2 and 4 to 7, the subject of main camera 500 after the change and the subject of sub camera 400 after the change may be different subjects.

[0059] The role control device 600 transmits role information ROLE designated by the user from among the multiple patterns illustrated in Fig. 5 to the role setting unit 120 of the control device 100. The role setting unit 120 sets a correspondence relationship between the subject photographed by the main camera 500 and the subject photographed by the sub-camera 400 based on the role information ROLE received from the role control device 600. The role setting unit 120 transmits the set role information ROLE to the tracking subject determination unit 123.

[0060] If the user specifies incompatible patterns such as pattern 1 and pattern 2, the role setting unit 120 determines the shooting of the sub-camera 400 based on the pattern registered later. Furthermore, in cases where the subject of main camera 500 is changed from subject A to subject D, which is not included in the pattern specified by the user, role setting unit 120 may set the subject of sub-camera 400 to the same subject as that of main camera 500, for example.

[0061] The first recognition unit 121 reads from RAM 102 the image IMG captured by the overhead camera 300 and reference position information REF_POSI, which includes information on the installation position of each camera and information on marker coordinates. The first recognition unit 121 uses the information read from RAM 102 to detect a person (subject) to be photographed and acquires position information POSITION of the detected subject. If multiple subjects are detected, the position information POSITION includes position information for each subject. The first recognition unit 121 outputs the position information POSITION of the detected subjects and the identification information ID of each subject.

[0062] The installation position of each camera can be measured manually or using a sensor (not shown) as the camera's position in a plane coordinate system when the shooting area is viewed from directly above. Information on the measured installation position of each camera is input to the control device 100 by, for example, a user via input unit 106. CPU 101 writes the input information on the installation position of each camera into RAM 102. Alternatively, the measured information on the installation position of each camera may be transmitted from the sensor to the control device 100 and written into RAM 102 by CPU 101.

[0063] The markers are installed so that they can be recognized when viewed from directly above the shooting area in order to calculate the homography transformation matrix used for coordinate transformation. Marker coordinates in the coordinate system of the shooting area can be measured manually or using a sensor (not shown). Markers are marks of a color different from the color of the floor or ground, but any form is acceptable as long as they can be measured manually or by a sensor. When the sensor that measures the markers is a camera, by installing marks of a specific color as markers, the camera can extract the color of the marker from the captured image of the shooting area and obtain the marker coordinates. Information about the marker coordinates is input to the control device 100, for example, by a user via the input unit 106. The CPU 101 writes the input information about the marker coordinates to the RAM 102. Alternatively, information about the measured marker coordinates may be transmitted from the sensor to the control device 100 and written to the RAM 102 by the CPU 101.

[0064] The reference position information REF_POSI and the subject position information POSITION are expressed as coordinates in a planar coordinate system of the shooting area viewed from directly above. For example, if the first recognition unit 121 detects three subjects, it outputs the position information POSITION and identification information ID for the three subjects. The first recognition unit 121 writes the output information to RAM 102.

[0065] A method for detecting the position of a subject present in a shooting area and acquiring position information POS of the detected subject will be described with reference to Figures 6(A) and 6(B). Figure 6(A) shows the coordinate system of the image captured by overhead camera 300. Figure 6(B) shows the coordinate system of the shooting area viewed from directly above. Based on the relationship between the coordinate system of the image captured by overhead camera 300 and the coordinate system of the shooting area viewed from directly above, first recognition unit 121 acquires the coordinates of the subject in the coordinate system of the shooting area viewed from directly above as position information POS.

[0066] When calculating the pan value so that sub-camera 400a and sub-camera 400b face the direction of the subject to be tracked, the angle calculation is simplified if the calculation is performed in a plane coordinate system perpendicular to the axis of the panning operation. For example, if sub-camera 400a and sub-camera 400b are installed perpendicular to the floor or ground, the coordinate plane perpendicular to the axis of the panning operation is parallel to the floor or ground and is a coordinate plane that views the space in which the subject exists from directly above.

[0067] In the following description, sub-camera 400a and sub-camera 400b are installed perpendicular to the floor or ground, and the pan value of each sub-camera 400 is calculated in the coordinate system of the shooting area viewed from directly above. The first recognition unit 121 converts the coordinates of the subject in the coordinate system of the image captured by overhead camera 300 shown in Fig. 6(A) (hereinafter referred to as overhead camera coordinate system) into coordinates in the coordinate system of the shooting area viewed from directly above (hereinafter referred to as planar coordinate system) shown in Fig. 6(B). This allows the first recognition unit 121 to calculate the pan value of each sub-camera 400 in the coordinate system of the shooting area viewed from directly above.

[0068] The coordinates of the subject in the overhead camera coordinate system are transformed into coordinates in the planar coordinate system using the homography transformation matrix H according to the following equation 1.

number

[0069] The first recognition unit 121 reads out reference position information REF_POSI from RAM 102. The marker coordinates of markers Mark_A to Mark_D included in REF_POSI are coordinates in a planar coordinate system of the shooting area viewed from directly above, and are substituted into the left side of Equation 1. The first recognition unit 121 also acquires the marker coordinates in the overhead camera coordinate system from the image IMG captured by the overhead camera 300, and substitutes them into the right side of Equation 1. By substituting the marker coordinates in the overhead camera coordinate system and the marker coordinates in the planar coordinate system into Equation 1, a homography transformation matrix H can be obtained.

[0070] Once the homography transformation matrix H is calculated, the first recognition unit 121 can use Equation 1 to map the overhead camera coordinate system shown in FIG. 6(A) to the planar coordinate system shown in FIG. 6(B). That is, the first recognition unit 121 can convert any coordinate in the overhead camera coordinate system into the corresponding coordinate in the planar coordinate system. Therefore, the first recognition unit 121 can determine the positions of subjects A, B, and C captured in the image IMG captured by the overhead camera 300 in the planar coordinate system of FIG. 6(B). The first recognition unit 121 writes the homography transformation matrix H calculated using the marker coordinates to RAM 102.

[0071] 7(A) and 7(B), a method for detecting the position of a subject from a captured image IMG input from the overhead camera 300 using an inference model for subject detection will be described. The inference model for subject detection is stored in advance in ROM 103. When detecting a person as the subject, for example, the first recognition unit 121 can detect the person from the image IMG captured by the overhead camera 300 using a trained person detection inference model created by a machine learning method such as deep learning. The person detection inference model takes an image as input and outputs the coordinates within the image of the person appearing in the image.

[0072] The first recognition unit 121 (inference unit 104) inputs the captured image IMG input from the overhead camera 300 into a person detection inference model and detects people present in the image. FIG. 7(A) shows people detected by the person detection inference model surrounded by a rectangle. In the example of FIG. 7(A), subjects A, B, and C are detected from the image IMG captured by the overhead camera 300. The first recognition unit 121 writes information on the positions (coordinates) of the people detected in the image captured by the overhead camera 300 to RAM 102.

[0073] The first recognition unit 121 is not limited to a machine learning method, and may use, for example, a SIFT (Scale-Invariant Feature Transform) method that detects a person as a subject by matching local feature points in an image. Alternatively, the first recognition unit 121 may detect a person using a template matching method that detects a subject by calculating the similarity with a template image.

[0074] The first recognition unit 121 determines a point on a rectangle surrounding the subject detected in the overhead camera coordinate system shown in Fig. 7(A), for example, the midpoint of the bottom side (the side on the foot side) of the rectangle, as the subject detection position. The first recognition unit 121 converts the coordinates of the subject detection position into coordinates in the planar coordinate system shown in Fig. 7(B) using Equation 1.

[0075] Specifically, the first recognition unit 121 reads the homography transformation matrix H from RAM 102 and can convert the coordinates (xa, ya) of the detection position of subject A in the overhead camera coordinate system into coordinates (XA, YA) in the planar coordinate system by substituting them for x and y in Equation 1. Similarly, the first recognition unit 121 can convert the coordinates (xb, yb) and (xc, yc) of the detection positions of subjects B and C into coordinates (XB, YB) and (XC, YC) in the planar coordinate system by substituting them for x and y in Equation 1. The first recognition unit 121 writes the converted coordinates of subjects A, B, and C into RAM 102 as position information POSITION of the detected subjects.

[0076] The first recognition unit 121 outputs the identification information ID of the detected subject in addition to the position information POSITION of the detected subject. The first recognition unit 121 can acquire the identification information ID of the detected subject using a trained model for person identification stored in the ROM 103. The trained model is a trained model that is generated using a machine learning method such as deep learning to learn from past captured images and detection results in the past captured images, and is configured to output the identification information ID of the subject.

[0077] The first recognition unit 121 can acquire an identification ID by inputting the image captured by the overhead camera 300 and the position information of the subject detected in the image captured by the overhead camera 300 into a trained model for person identification read from ROM 103. The trained model for person identification can be generated using known technology. For example, the trained model for person identification can be generated by training the model so that past frame information and the detection results in the past frame information are input and an identification ID is output. The first recognition unit 121 can acquire the identification ID of the subject detected in the current frame by inputting current frame information into the trained model for person identification.

[0078] Main subject determination unit 122 determines a main subject, which is the subject of image capture by main camera 500, based on the orientation of main camera 500 and position information of the subject. Main subject determination unit 122 acquires information on the determined main subject as information indicating the subject of image capture. Main subject determination unit 122 determines a main subject, which is the subject of image capture by main camera 500, based on position information POSITION of the subject acquired by first recognition unit 121 and direction information ANGLE input from main camera 500. For example, main subject determination unit 122 can determine, as the main subject, a subject that is located closer to the orientation (on the optical axis) of main camera 500, among the subjects detected by first recognition unit 121. Main subject determination unit 122 writes identification information MAIN_SUBJECT_ID of the determined main subject to RAM 102.

[0079] Note that main subject determination section 122 may detect the main subject from the image captured by main camera 500. Main subject determination section 122 can, for example, determine a subject that exists closer to the center of the image captured by main camera 500 as the main subject.

[0080] The tracking subject determination unit 123 determines a tracking subject that is a subject to be photographed by the sub camera 400, based on the main subject and the correspondence (role information ROLE) between the subject to be photographed by the main camera 500 and the subject to be photographed by the sub camera 400. The tracking subject determination unit 123 transmits information on the determined tracking subject to the photographing direction acquisition unit 124.

[0081] The tracking subject determination unit 123 can determine the tracking subjects of sub-camera 400a and sub-camera 400b, for example, as follows: The tracking subject determination unit 123 acquires subject information DETECT_INFO, which is information on the subjects detected by the second recognition unit 125 from the images captured by sub-camera 400a and sub-camera 400. The tracking subject determination unit 123 determines the tracking subject based on the identification information MAIN_SUBJECT_ID of the main subject, the subject information DETCT_INFO, and the role information ROLE. The tracking subject determination unit 123 transmits the identification information SUBJECT_ID of the determined tracking subject to the shooting direction acquisition unit 124.

[0082] When the subject being photographed by main camera 500 is changed, tracking subject determination unit 123 receives identification information MAIN_SUBJECT_ID of the main subject of the changed photographing subject from main subject determination unit 122. Tracking subject determination unit 123 acquires identification information SUBJECT_ID of the tracking subject corresponding to the changed main subject, and transmits it to shooting direction acquisition unit 124.

[0083] The tracking subject determination unit 123 determines whether the tracking subject is closer to the sub-camera 400a or the sub-camera 400b from the image IMG captured by the overhead camera 300. The tracking subject determination unit 123 transmits identification information SUBJECT_ID of the tracking subject to the imaging direction acquisition unit 124, and instructs it to change the imaging direction of the sub-camera 400 (here, sub-camera 400a) that is closer to the tracking subject.

[0084] After the shooting direction of the sub-camera 400a, which is closer to the tracking subject, is changed, the tracking subject determination unit 123 receives subject information DETECT_INFO of the subject detected from the image SUB_IMG captured by the sub-camera 400a. The tracking subject determination unit 123 confirms that the tracking subject with identification information SUBJECT_ID is captured in the image SUB_IMG captured by the sub-camera 400a. After confirmation, the tracking subject determination unit 123 transmits the identification information SUBJECT_ID of the tracking subject to the shooting direction acquisition unit 124, and instructs the shooting direction acquisition unit 124 to change the shooting direction of the sub-camera 400b, whose shooting direction has not been changed.

[0085] 8(A) to 8(D), a specific example will be described in which the subject of shooting of main camera 500 is changed, and the subject of shooting of sub camera 400 is changed accordingly. The shooting directions of sub camera 400a and sub camera 400b are changed in the order of FIG. 8(A) to FIG. 8(D).

[0086] In the examples of Figures 8(A) to 8(D), when main camera 500 is capturing an image of subject B as the main subject, sub-camera 400a and sub-camera 400b capture an image of subject D. When the subject being captured by main camera 500 is changed from subject B to subject A, the subject being captured by sub-camera 400a and sub-camera 400b is changed to subject C. However, the subject being captured by sub-camera 400a and sub-camera 400b is changed to subject C at different times.

[0087] 8(A) shows a state in which main camera 500 is photographing subject B, and sub-camera 400a and sub-camera 400b are photographing subject D corresponding to subject B. In FIG. 8(B), the subject being photographed by main camera 500 is changed from subject B to subject A, and the photographing direction of main camera 500 is changed from facing subject B to facing subject A. The subject being photographed by sub-camera 400 corresponding to subject A is subject C.

[0088] 8(C), the shooting direction of sub-camera 400a, which is closer to subject C among sub-cameras 400, is changed from a direction facing subject D to a direction facing subject C corresponding to subject A. At this point, the shooting direction of sub-camera 400b remains unchanged, facing subject D corresponding to subject B.

[0089] In FIG. 8(D), the shooting direction of sub-camera 400b is changed from the direction facing subject D to the direction facing subject C. In this way, after the shooting direction of sub-camera 400a is first changed to the direction facing subject C, the shooting direction of sub-camera 400b is changed from the direction facing subject D to the direction facing subject C. Therefore, while the shooting direction of sub-camera 400a is changed from the direction facing subject D to the direction facing subject C, sub-camera 400b can record a shot image of subject D. Also, while the shooting direction of sub-camera 400b is changed from the direction facing subject D to the direction facing subject C, sub-camera 400a can record a shot image of subject C.

[0090] 8(C) shows an example in which the shooting direction of the sub-camera 400a that is closer to the subject C is changed first, but which sub-camera 400 has its shooting direction changed first is not necessarily determined based on the distance to the tracking subject. The tracking subject determination unit 123 may determine which sub-camera 400 has its shooting direction changed first based on, for example, a user operation.

[0091] Furthermore, the tracking subject determination unit 123 may determine which sub-camera 400 to change the shooting direction of first, based on the position of the sub-camera 400, the movable range of the shooting direction (optical axis direction) of the sub-camera 400, and the position of the tracking subject. For example, the tracking subject determination unit 123 may first change the shooting direction of the sub-camera 400 whose shooting direction includes the changed tracking subject within the movable range of the shooting direction. Furthermore, the tracking subject determination unit 123 may later change the shooting direction of the sub-camera 400 that can shoot the tracking subject before the change at the center of the screen (either one of the sub-cameras 400a and 400b, if both can shoot at the center of the screen).

[0092] 8(A) to 8(D) show an example in which there are two sub-cameras 400, but there may be three or more sub-cameras 400. When there are three or more sub-cameras 400 and the shooting direction of the main camera 500 is changed, the tracking subject determination unit 123 divides these multiple sub-cameras 400 into multiple groups that have different timings for changing the shooting direction.

[0093] 8(B) includes an additional sub-camera 400, the tracking subject determination unit 123 divides the multiple sub-cameras 400 into a first group including sub-camera 400b and a second group including sub-camera 400a. When the shooting direction of the main camera 500 is changed from subject B to subject A, the tracking subject determination unit 123 does not change the shooting direction of the sub-cameras 400 included in the first group from the direction facing subject D corresponding to subject B. On the other hand, when the shooting direction of the main camera 500 is changed, the tracking subject determination unit 123 changes the shooting direction of the sub-cameras 400 included in the second group from the direction facing subject D to the direction facing subject C corresponding to subject A.

[0094] The tracking subject determination unit 123 can divide the multiple sub-cameras 400 into a first group and a second group using various methods. For example, the tracking subject determination unit 123 can divide the multiple sub-cameras 400 into a first group and a second group based on the positions of the multiple sub-cameras 400.

[0095] More specifically, the tracking subject determination unit 123 determines the distance between each position of the plurality of sub-cameras 400 and the tracking subject corresponding to the subject before the shooting direction of the main camera 500 is changed. 8B, the tracking subject determination unit 123 later changes the shooting direction of the sub-cameras 400 in the first group that are closer to the subject D, for example.

[0096] Furthermore, the tracking subject determination unit 123 may group the sub-cameras 400 based on the position of each of the sub-cameras 400 and the distance to the tracking subject corresponding to the subject after the shooting direction of the main camera 500 has been changed. In the example of FIG. 8(B), the tracking subject determination unit 123 first changes the shooting direction of the sub-cameras 400 in the second group that are closer to subject C, for example. Note that the tracking subject determination unit 123 may group the sub-cameras 400 based on both the distance to subject D and the distance to subject C.

[0097] The tracking subject determination unit 123 may group the multiple sub-cameras 400 based not only on the distance to the tracking subject but also on the relative position to the tracking subject. The relative position to the tracking subject is represented by the distance from the sub-camera 400 to the tracking subject and the direction of the tracking subject with respect to the optical axis of the sub-camera 400. The tracking subject determination unit 123 can group the multiple sub-cameras 400 based on the relative position to the tracking subject before or after the change in the shooting direction of the main camera 500. The tracking subject determination unit 123 may group the multiple sub-cameras 400 based on the relative position to the tracking subject before and after the change in the shooting direction of the main camera 500.

[0098] The tracking subject determination unit 123 may divide the multiple sub-cameras 400 into a first group and a second group so that adjacent sub-cameras 400 are not included in the same group. By preventing adjacent sub-cameras 400 from being included in the same group, the control device 100 can capture images of the tracking subject from various angles before and after changing the shooting direction.

[0099] The tracking subject determination unit 123 may group the multiple sub-cameras 400 based not only on the distance to the tracking subject or the relative position to the tracking subject, but also on the movable range of the shooting direction of the sub-cameras 400. The movable range of the shooting direction indicates the range in which the orientation of the optical axis of the sub-camera 400 can be changed.

[0100] 9(A) to 9(C), the grouping based on the movable area of ​​the shooting direction of the sub-camera 400 will be described. The tracking subject determination unit 123 groups the multiple sub-cameras 400 based on the position of the sub-camera 400, the movable area of ​​the shooting direction of the sub-camera 400, and the position of the tracking subject. The tracking subject 903 shown in FIGS. 9(A) to 9(C) is the tracking subject corresponding to the subject after the shooting target of the main camera 500 has been changed.

[0101] 9(A) shows sub-camera 400a, a movable area 901 in the shooting direction of sub-camera 400a, sub-camera 400b, a movable area 902 in the shooting direction of sub-camera 400b, and a tracking subject 903. Tracking subject 903 exists within movable area 902 in the shooting direction of sub-camera 400b. Tracking subject determination unit 123 sets sub-cameras 400 in which tracking subject 903 exists within the movable area in the shooting direction as a second group, and first changes the shooting direction of the sub-cameras 400 in the second group.

[0102] 9(B), like FIG. 9(A), shows the sub camera 400a, the movable area 901 in the shooting direction of the sub camera 400a, the sub camera 400b, the movable area 902 in the shooting direction of the sub camera 400b, and the tracking subject 903. However, the tracking subject 903 is not included in the movable area 901 in the shooting direction of the sub camera 400a or the movable area 902 in the shooting direction of the sub camera 400b. The tracking subject determination unit 123, for example, classifies the sub cameras 400 whose distance from the position of the tracking subject 903 to the movable area is shorter than a predetermined threshold into a second group, and The shooting direction of the sub-camera 400 can be changed first.

[0103] 9(A), Fig. 9(C) shows sub camera 400a, a movable area 901 in the shooting direction of sub camera 400a, sub camera 400b, a movable area 902 in the shooting direction of sub camera 400b, and a tracking subject 903. However, tracking subject 903 is included in both movable area 901 in the shooting direction of sub camera 400a and movable area 902 in the shooting direction of sub camera 400b. In this case, tracking subject determination unit 123 may group multiple sub cameras 400 based on conditions other than the movable areas.

[0104] 9(A) to 9(C), tracking subject 903 is assumed to be a tracking subject corresponding to a subject after the change in the imaging target of main camera 500, but it may also be a tracking subject corresponding to a subject before the change in the imaging target of main camera 500. That is, tracking subject determination unit 123 may group multiple sub-cameras 400 based on the positions of the sub-cameras 400, the movable range of the imaging direction of the sub-cameras 400, and the position of the tracking subject before the change. For example, tracking subject determination unit 123 sets sub-cameras 400 (or some of them) whose pan / tilt direction can be controlled as a first group so that the tracking subject before the change is positioned in the optical axis direction, and later changes the imaging direction of the sub-cameras 400 in the first group.

[0105] The tracking subject determination unit 123 may also divide the multiple sub-cameras 400 into a first group and a second group based on a user operation. The tracking subject determination unit 123 may acquire grouping information stored in advance in ROM 103 based on a user operation, or may acquire grouping information by accepting an operation from the user during shooting.

[0106] The above-mentioned various methods can be combined as appropriate to group the multiple sub-cameras 400. Furthermore, the multiple sub-cameras 400 are not limited to two groups, and may be divided into three or more groups depending on the number of subjects to be tracked, etc.

[0107] The imaging direction acquisition unit 124 receives information about the sub camera 400 whose imaging direction is to be changed and the identification information SUBJECT_ID of the tracking subject from the tracking subject determination unit 123, and calculates the pan and tilt control values ​​for the sub camera 400.

[0108] The shooting direction acquisition unit 124 reads out from RAM 102 the coordinates of the sub camera 400 in a planar coordinate system contained in the reference position information REF_POSI and the coordinates of the tracked subject contained in the position information POSITION of the detected subject. Based on the coordinates of the sub camera 400 and the coordinates of the tracked subject, the shooting direction acquisition unit 124 calculates pan / tilt control values ​​for directing the sub camera 400 toward the tracked subject. A method for calculating the pan / tilt target positions as pan / tilt control values ​​will be described using Figures 10 and 11.

[0109] 10 is a diagram illustrating the target position of panning. Angle θ, which indicates the target position of panning, is the angle formed between a line extending from the center of the optical axis of sub-camera 400 and a line connecting sub-camera 400 and the tracking subject. Angle θ can be calculated using the following equation 2.

number

[0110] 11 is a diagram illustrating the target position of tilt. Angle ρ, which indicates the target position of tilt, is the angle formed by a line extending from the optical axis of sub-camera 400 at height h1 of the optical axis of sub-camera 400 and a line extending from sub-camera 400 toward the top of the head of the tracking subject at height h2. Angle ρ can be calculated using the following equations 3 and 4.

number

[0111] The pan / tilt control value may be an angular velocity value for rotating the sub-camera 400 in the direction of the tracking subject. An example of a method for calculating the pan / tilt angular velocity value will be described. The shooting direction acquisition unit 124 acquires the current pan / tilt position of the sub-camera 400 via the network I / F 105. The shooting direction acquisition unit 124 calculates a pan angular velocity proportional to the difference between the current pan position and the pan target position. The shooting direction acquisition unit 124 also calculates a tilt angular velocity proportional to the difference between the current tilt position and the tilt target position. The shooting direction acquisition unit 124 writes the calculated pan and tilt control values ​​to the RAM 102.

[0112] The second recognition unit 125 determines whether or not a tracking subject has been detected from the image SUB_IMG captured by the sub camera 400, using an inference model for identifying a person stored in the ROM 103, similar to the first recognition unit 121. The second recognition unit 125 transmits the determination result to the tracking subject determination unit 123 as subject information DETECT_INFO.

[0113] 12(A) and 12(B), a process for controlling the shooting directions of multiple sub-cameras 400 so that they face a target corresponding to a target shot by the main camera 500 will be described. FIGS. 12(A) and 12(B) are flowcharts illustrating an example of the process for controlling the shooting directions of the sub-cameras 400. FIGS. 12(A) and 12(B) show an example of a process in which, when the main camera 500 shoots target A, the sub-cameras 400a and 400b shoot target C, and when the main camera 500 shoots target B, the sub-cameras 400a and 400b shoot target D. The sub-cameras 400a and 400b are also collectively referred to as sub-cameras 400.

[0114] In step S1201 of FIG. 12(A), CPU 101 detects a main subject that is an object to be photographed by main camera 500. CPU 101 acquires information about the detected main subject as information indicating the object to be photographed by main camera 500. Specifically, first recognition unit 121 acquires an image photographed by main camera 500 and detects the subject. First recognition unit 121 transmits position information (including coordinate information) and identification information of the detected subject to main subject determination unit 122. Main subject determination unit 122 determines the main subject based on the orientation of main camera 500, the position at which the subject appears in the image photographed by main camera 500, or the like. The main subject determined by main subject determination unit 122 is the main subject detected in step S1201.

[0115] In step S1202, the CPU 101 detects the main object It is determined whether the detected main subject is subject A. If the detected main subject is subject A, the process proceeds to step S1203. If the detected main subject is not subject A, the process proceeds to step S1302 in FIG. 12(B).

[0116] In step S1203, CPU 101 acquires the coordinates of subject C, which is the subject being photographed by sub-camera 400 and corresponds to subject A, the main subject. Specifically, main subject determination unit 122 transmits identification information and position information of the main subject to tracking subject determination unit 123. Furthermore, role setting unit 120 transmits information on the correspondence between the subject being photographed by main camera 500 and the subject being photographed by sub-camera 400 to tracking subject determination unit 123. Tracking subject determination unit 123 determines subject C as the tracking subject. Tracking subject determination unit 123 acquires position information (including coordinate information) of subject C from the image photographed by overhead camera 300. Tracking subject determination unit 123 transmits the acquired position information of subject C to shooting direction acquisition unit 124.

[0117] In step S1204, CPU 101 points sub-cameras 400a and 400b toward subject C. Specifically, imaging direction acquisition unit 124 acquires pan / tilt control values ​​for each of sub-cameras 400a and 400b to control the imaging directions of sub-cameras 400a and 400b so that they point toward subject C, the tracking subject. Imaging direction acquisition unit 124 transmits the pan / tilt control values ​​to sub-cameras 400a and 400b. Based on the pan / tilt control values ​​received from imaging direction acquisition unit 124, sub-cameras 400a and 400b point their imaging directions toward subject C.

[0118] In step S1205, similarly to step S1201, CPU 101 detects the main subject that is the subject of image capture by main camera 500. In step S1206, CPU 101 determines whether or not the main subject detected in step S1205 is subject B.

[0119] If the detected main subject is subject B, it is determined that the shooting direction of main camera 500 has changed from a direction facing subject A to a direction facing subject B, and processing proceeds to step S1207. In this way, whether or not the shooting direction of main camera 500 has changed can be determined based on a change in the subject detected from the image captured by main camera 500. If the main subject detected in step S1205 is not subject B, processing returns to step S1203.

[0120] In step S1207, CPU 101 acquires the coordinates of subject D, which is the subject of image capture by sub-camera 400 and corresponds to main subject subject B. CPU 101 can acquire the coordinates of subject D from the image captured by overhead camera 300, similar to step S1203.

[0121] In step S1208, CPU 101 determines whether subject D is closer to sub-camera 400a or sub-camera 400b (the distance to which is shorter). If subject D is closer to sub-camera 400a than to sub-camera 400b, the process proceeds to step S1209. If subject D is closer to sub-camera 400b than to sub-camera 400a, the process proceeds to step S1214.

[0122] In step S1209, CPU 101 acquires information on the movable range of each of the imaging directions of sub-cameras 400a and 400b. The movable range of the imaging direction of sub-camera 400 indicates the range in which the optical axis of sub-camera 400 can be directed.

[0123] In step S1210, CPU 101 determines whether or not a line (optical axis) extending from the center of the lens in a forward direction can be superimposed on subject D by panning sub-camera 400a. In other words, CPU 101 determines whether or not subject D can be photographed so that it is positioned approximately at the center of the image photographed by sub-camera 400a. If the optical axis of sub-camera 400a can be superimposed on subject D, the process proceeds to step S1211. If the axis cannot be superimposed on the subject D, the process proceeds to step S1213.

[0124] In step S1211, CPU 101 points sub-camera 400a toward subject D. Specifically, imaging direction acquisition unit 124 acquires pan / tilt control values ​​for controlling the imaging direction of sub-camera 400a so that it faces subject D, which is the tracking target. Imaging direction acquisition unit 124 transmits the acquired pan / tilt control values ​​to sub-camera 400a. Sub-camera 400a faces the imaging direction toward subject D based on the pan / tilt control values ​​received from imaging direction acquisition unit 124. Second recognition unit 125 acquires an image captured by sub-camera 400a and confirms that subject D has been detected from the acquired captured image. By the processing of step S1211, CPU 101 can confirm that the imaging direction of sub-camera 400a has been changed to face subject D and that subject D is being tracked.

[0125] In step S1212, CPU 101 points sub-camera 400b toward subject D. Specifically, imaging direction acquisition unit 124 acquires pan / tilt control values ​​for controlling the imaging direction of sub-camera 400b so that it faces subject D, which is the tracking target. Imaging direction acquisition unit 124 transmits the acquired pan / tilt control values ​​to sub-camera 400b. Sub-camera 400b faces its imaging direction toward subject D based on the pan / tilt control values ​​received from imaging direction acquisition unit 124. Second recognition unit 125 acquires an image captured by sub-camera 400b and confirms that subject D has been detected from the acquired captured image. By the processing of step S1212, CPU 101 can confirm that the imaging direction of sub-camera 400b has been changed to face subject D and that subject D is being tracked.

[0126] In step S1213, CPU 101 determines which of sub-camera 400a and sub-camera 400b can capture subject D at a position closer to the center of the screen. For example, when the optical axes of sub-cameras 400a and 400b are brought closer to subject D within the range of their movable ranges, CPU 101 can determine that the sub-camera 400 whose optical axis is closer to subject D can capture subject D at a position closer to the center of the screen. If sub-camera 400a can capture subject D at a position closer to the center of the screen, the process proceeds to step S1211. If sub-camera 400b can capture subject D at a position closer to the center of the screen, the process proceeds to step S1214.

[0127] In step S1214, CPU 101 directs sub-camera 400b toward subject D, similar to step S1212. In step S1215, CPU 101 directs sub-camera 400a toward subject D, similar to step S1211.

[0128] In step S1216, CPU 101 points sub-cameras 400a and 400b toward subject D. Specifically, imaging direction acquisition unit 124 acquires pan / tilt control values ​​for each of sub-cameras 400a and 400b to control the imaging directions of sub-cameras 400a and 400b so that they point toward subject D, the tracking subject. Imaging direction acquisition unit 124 transmits the pan / tilt control values ​​to sub-cameras 400a and 400b. Based on the pan / tilt control values ​​received from imaging direction acquisition unit 124, sub-cameras 400a and 400b point their imaging directions toward subject D.

[0129] In step S1217, CPU 101 detects a main subject that is the object of shooting by main camera 500, as in step S1201. In step S1218, CPU 101 determines whether or not the main subject detected in step S1217 is subject A. If the main subject detected in step S1217 is subject A, it is determined that the shooting direction of main camera 500 has changed from a direction facing subject B to a direction facing subject A, and processing proceeds to step S1307 in FIG. 12(B). If the detected main subject is not subject A, processing returns to step S1216.

[0130] In step S1302, CPU 101 determines whether the main subject detected in step S1201 of Fig. 12A is subject B. If the detected main subject is subject B, the process proceeds to step S1303. If the detected main subject is not subject B, the process proceeds to step S1201 of Fig. 12A.

[0131] In step S1303, CPU 101 acquires the coordinates of subject D, which is the subject captured by sub-camera 400 and corresponds to subject B, the main subject. Specifically, main subject determination unit 122 transmits identification information and position information of the main subject to tracking subject determination unit 123. Furthermore, role setting unit 120 transmits information on the correspondence between the subject captured by main camera 500 and the subject captured by sub-camera 400 to tracking subject determination unit 123. Tracking subject determination unit 123 determines subject D as the tracking subject. Tracking subject determination unit 123 acquires position information (including coordinate information) of subject D from the image captured by overhead camera 300. Tracking subject determination unit 123 transmits the acquired position information of subject D to shooting direction acquisition unit 124.

[0132] In step S1304, CPU 101 points sub-cameras 400a and 400b toward subject D. Specifically, imaging direction acquisition unit 124 acquires pan / tilt control values ​​for each of sub-cameras 400a and 400b to control the imaging directions of sub-cameras 400a and 400b so that they point toward subject D, the tracking subject. Imaging direction acquisition unit 124 transmits the pan / tilt control values ​​to sub-cameras 400a and 400b. Based on the pan / tilt control values ​​received from imaging direction acquisition unit 124, sub-cameras 400a and 400b point their imaging directions toward subject D.

[0133] In step S1305, similarly to step S1201, CPU 101 detects the main subject that is the subject of image capture by main camera 500. In step S1306, CPU 101 determines whether the main subject detected in step S1305 is subject A or not.

[0134] If the detected main subject is subject A, it is determined that the shooting direction of main camera 500 has changed from a direction facing subject B to a direction facing subject A, and the process proceeds to step S1307. If the main subject detected in step S1305 is not subject A, the process returns to step S1303.

[0135] In step S1307, CPU 101 acquires the coordinates of subject C, which is the subject of image capture by sub-camera 400 and corresponds to main subject subject A. CPU 101 can acquire the coordinates of subject C from the image captured by overhead camera 300, similar to step S1203.

[0136] In step S1308, CPU 101 determines whether subject C is closer to sub-camera 400a or sub-camera 400b (the distance to which is shorter). If subject C is closer to sub-camera 400a than to sub-camera 400b, the process proceeds to step S1309. If subject C is closer to sub-camera 400b than to sub-camera 400a, the process proceeds to step S1314.

[0137] In step S1309, CPU 101 acquires information on the movable range of each of the imaging directions of sub-cameras 400a and 400b. The movable range of the imaging direction of sub-camera 400 indicates the range in which the optical axis of sub-camera 400 can be directed.

[0138] In step S1310, CPU 101 determines whether or not, by panning sub-camera 400a, a line (optical axis) extending from the center of the lens in a forward direction can be superimposed on subject C. In other words, CPU 101 determines whether or not subject C can be photographed so that it is positioned approximately in the center of the image photographed by sub-camera 400a. If the optical axis of sub-camera 400a can be superimposed on subject C, the process proceeds to step S1311. If the optical axis of sub-camera 400a cannot be superimposed on subject C, the process proceeds to step S1313.

[0139] In step S1311, CPU 101 points sub-camera 400a toward subject C. Specifically, imaging direction acquisition unit 124 acquires pan / tilt control values ​​for controlling the imaging direction of sub-camera 400a so that it faces subject C, the tracking target. Imaging direction acquisition unit 124 transmits the acquired pan / tilt control values ​​to sub-camera 400a. Sub-camera 400a faces the imaging direction toward subject C based on the pan / tilt control values ​​received from imaging direction acquisition unit 124. Second recognition unit 125 acquires an image captured by sub-camera 400a and confirms that subject C has been detected from the acquired captured image. By the processing of step S1311, CPU 101 can confirm that the imaging direction of sub-camera 400a has been changed to face subject D and that subject C is being tracked.

[0140] In step S1312, CPU 101 points sub-camera 400b toward subject C. Specifically, imaging direction acquisition unit 124 acquires pan / tilt control values ​​for controlling the imaging direction of sub-camera 400b so that it faces subject C, the tracking target. Imaging direction acquisition unit 124 transmits the acquired pan / tilt control values ​​to sub-camera 400b. Sub-camera 400b faces its imaging direction toward subject C based on the pan / tilt control values ​​received from imaging direction acquisition unit 124. Second recognition unit 125 acquires an image captured by sub-camera 400b and confirms that subject C has been detected from the acquired captured image. By the processing of step S1312, CPU 101 can confirm that the imaging direction of sub-camera 400b has been changed to face subject C and that subject C is being tracked.

[0141] In step S1313, CPU 101 determines which of sub-camera 400a and sub-camera 400b can capture subject C at a position closer to the center of the screen. For example, when the optical axes of sub-cameras 400a and 400b are brought closer to subject C within the movable range, CPU 101 can determine that the sub-camera 400 whose optical axis is closer to subject C can capture subject C at a position closer to the center of the screen. If sub-camera 400a can capture subject C at a position closer to the center of the screen, the process proceeds to step S1311. If sub-camera 400b can capture subject C at a position closer to the center of the screen, the process proceeds to step S1314.

[0142] In step S1314, similar to step S1312, CPU 101 directs sub-camera 400b toward subject C. In step S1315, CPU 101 directs sub-camera 400a toward subject C, similar to step S1311.

[0143] In step S1316, CPU 101 points sub-cameras 400a and 400b toward subject C. Specifically, imaging direction acquisition unit 124 acquires pan / tilt control values ​​for each of sub-cameras 400a and 400b to control the imaging directions of sub-cameras 400a and 400b so that they point toward subject C, the tracking subject. Imaging direction acquisition unit 124 transmits the pan / tilt control values ​​to sub-cameras 400a and 400b. Based on the pan / tilt control values ​​received from imaging direction acquisition unit 124, sub-cameras 400a and 400b point their imaging directions toward subject C.

[0144] In step S1317, CPU 101 detects a main subject that is the object of shooting by main camera 500, similarly to step S1201 in Fig. 12(A). In step S1318, CPU 101 determines whether or not the main subject detected in step S1317 is subject B. If the main subject detected in step S1317 is subject B, it is determined that the shooting direction of main camera 500 has changed from a direction facing subject A to a direction facing subject B, and processing proceeds to step S1207 in Fig. 12(A). If the detected main subject is not subject B, processing returns to step S1316.

[0145] 12(A) and 12(B) illustrate an example in which there are two sub-cameras 400. However, there may be three or more sub-cameras 400. In this case, the control device 100 may divide the multiple sub-cameras 400 into a group that changes the shooting direction at the same timing as sub-camera 400a and a group that changes the shooting direction at the same timing as sub-camera 400b. The control device 100 may divide the multiple sub-cameras 400 into groups before the processing of steps S1211 and S1214 in Fig. 12(A) and steps S1311 and S1314 in Fig. 12(B).

[0146] The control device 100 can group the multiple sub-cameras 400 based on, for example, the relative position between the sub-camera 400 and the tracked subject, the distance between the sub-camera 400 and the tracked subject, or the movable range of the shooting direction of the sub-camera 400. The control device 100 may also group the multiple sub-cameras 400 so that adjacent sub-cameras 400 are not included in the same group. The control device 100 may also group the multiple sub-cameras 400 based on a user operation. The control device 100 may group the multiple sub-cameras 400 by arbitrarily combining these various conditions.

[0147] 12(A) and 12(B), the process that takes into account the movable area in the shooting direction of the sub-camera 400 may be omitted. That is, the processes of steps S1209, S1210, and S1213 in Fig. 12(A) and the processes of steps S1309, S1310, and S1313 in Fig. 12(B) may be omitted.

[0148] Furthermore, when the shooting direction of main camera 500 is facing subject A and subject A is no longer detected in the image captured by main camera 500, control device 100 controls sub-camera 400, which changes its shooting direction first, not to change its shooting direction from subject C. Furthermore, when the shooting direction of main camera 500 is changed to face subject B but subject B is not detected in the image captured by main camera 500, control device 100 controls sub-camera 400, which changes its shooting direction first, not to change its shooting direction from subject C.

[0149] In the above embodiment, when the control device 100 controls the shooting directions of the multiple sub-cameras 400 so that they face a target corresponding to that of the main camera 500, it first changes the shooting directions of some of the sub-cameras 400 and does not change the shooting directions of the other sub-cameras 400. After changing the shooting directions of some of the sub-cameras 400, the control device 100 changes the shooting directions of the other sub-cameras 400. This makes it possible to capture effective angles of view for each of the targets before and after the change using multiple cameras when changing the target in multi-camera shooting. Therefore, during post-shooting editing and live broadcasting, the user can obtain the desired image from images captured at multiple angles of view, even in scenes where the target changes.

[0150] The above-described embodiment is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configuration of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0151] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0152] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an acquisition means for acquiring information indicating a subject to be photographed by the first imaging device; a control means for controlling the photographing directions of the plurality of second photographing devices so that the second photographing devices are oriented toward a photographing target corresponding to the photographing target of the first photographing device; and When the photographing direction of the first imaging device is changed from a direction facing a first subject to a direction facing a second subject, the control means among the plurality of second imaging devices, the imaging direction of the second imaging devices included in a first group is not changed from a direction facing a third subject corresponding to the first subject, The imaging direction of the second imaging devices included in a second group among the plurality of second imaging devices is changed from a direction facing the third subject to a direction facing a fourth subject corresponding to the second subject. A control device characterized by: (Configuration 2) The control means changes the imaging direction of the second imaging device included in the first group from a direction facing the third subject to a direction facing the fourth subject after the imaging direction of the second imaging device included in the second group is changed to a direction facing the fourth subject. 2. The control device according to configuration 1, (Configuration 3) Whether or not the photographing direction of the first imaging device has been changed from a direction facing the first subject to a direction facing the second subject is determined based on a change in the subject detected from the photographed image of the first imaging device. 3. The control device according to configuration 1 or 2. (Configuration 4) When the imaging direction of the first imaging device is a direction facing the first subject and the first subject is no longer detected from the image captured by the first imaging device, or when the imaging direction of the first imaging device is changed to a direction facing the second subject but the second subject is not detected from the image captured by the first imaging device, the control means does not change the imaging direction of the second imaging device included in the second group from a direction facing the third subject. 4. The control device according to any one of configurations 1 to 3. (Configuration 5) The first subject and the third subject are the same subject. 5. The control device according to any one of configurations 1 to 4. (Configuration 6) The first object and the fourth object are the same object. 5. The control device according to any one of configurations 1 to 4. (Configuration 7) The second subject and the third subject are the same subject. 7. The control device according to any one of configurations 1 to 6, (Configuration 8) The second subject and the fourth subject are the same subject. 7. The control device according to any one of configurations 1 to 6, (Configuration 9) The third subject is a subject different from the first subject. 5. The control device according to any one of configurations 1 to 4. (Configuration 10) The fourth subject is a subject different from the second subject. 5. The control device according to any one of configurations 1 to 4. (Configuration 11) The imaging device further includes a dividing unit that divides the plurality of second imaging devices into the first group and the second group based on the positions of the plurality of second imaging devices. 11. The control device according to any one of configurations 1 to 10. (Configuration 12) The dividing means divides the second imaging devices into the first group and the second group based on at least one of a relative position of each of the second imaging devices and the fourth object, and a relative position of each of the second imaging devices and the third object. 12. The control device according to configuration 11, (Configuration 13) The dividing means divides the second imaging devices into the first group and the second group based on at least one of a distance between the position of each of the second imaging devices and the fourth object and a distance between the position of each of the second imaging devices and the third object. 12. The control device according to configuration 11, (Configuration 14) The dividing means divides the plurality of second imaging devices into the first group and the second group so that adjacent second imaging devices are not included in the same group. 12. The control device according to configuration 11, (Configuration 15) The dividing means divides the second imaging devices into the first group and the second group based on the positions of the second imaging devices, the movable ranges of the imaging directions of the second imaging devices, and the position of the third subject. 12. The control device according to configuration 11, (Configuration 16) The dividing means divides the second imaging devices into the first group and the second group based on the positions of the second imaging devices, the movable ranges of the imaging directions of the second imaging devices, and the position of the fourth subject. 12. The control device according to configuration 11, (Configuration 17) The imaging device further includes a dividing unit that divides the plurality of second imaging devices into the first group and the second group based on a user operation. 11. The control device according to any one of configurations 1 to 10. (method) an acquisition step of acquiring information indicating a subject to be photographed by the first imaging device; a control step of controlling the photographing directions of the plurality of second photographing devices so that the second photographing devices are oriented toward a photographing target corresponding to the photographing target of the first photographing device; and In the control step, when the photographing direction of the first imaging device is changed from a direction facing a first object to a direction facing a second object, among the plurality of second imaging devices, the imaging direction of the second imaging devices included in a first group is not changed from a direction facing a third subject corresponding to the first subject, The imaging direction of the second imaging devices included in a second group among the plurality of second imaging devices is changed from a direction facing the third subject to a direction facing a fourth subject corresponding to the second subject. A control method comprising: (program) A program for causing a computer to function as each means of the control device according to any one of configurations 1 to 17. [Explanation of symbols]

[0153] 100: control device, 101: CPU

Claims

1. an acquisition means for acquiring information indicating a subject to be photographed by the first imaging device; a control means for controlling the photographing directions of the plurality of second photographing devices so that the second photographing devices are oriented toward a photographing target corresponding to the photographing target of the first photographing device; and When the photographing direction of the first imaging device is changed from a direction facing a first subject to a direction facing a second subject, the control means changing the imaging direction of the second imaging devices included in a second group from a direction facing a third subject corresponding to the first subject to a direction facing a fourth subject corresponding to the second subject, changing the imaging direction of the second imaging devices included in a first group from a direction facing the third subject to a direction facing the fourth subject; A control device characterized by:

2. Whether or not the photographing direction of the first imaging device has been changed from a direction facing the first subject to a direction facing the second subject is determined based on a change in the subject detected from the photographed image of the first imaging device.

2. The control device according to claim 1.

3. When the imaging direction of the first imaging device is a direction facing the first subject and the first subject is no longer detected from the image captured by the first imaging device, or when the imaging direction of the first imaging device is changed to a direction facing the second subject but the second subject is not detected from the image captured by the first imaging device, the control means does not change the imaging direction of the second imaging device included in the second group from a direction facing the third subject.

2. The control device according to claim 1.

4. The first subject and the third subject are the same subject.

2. The control device according to claim 1.

5. The first object and the fourth object are the same object.

2. The control device according to claim 1.

6. The second subject and the third subject are the same subject.

2. The control device according to claim 1.

7. The second subject and the fourth subject are the same subject.

2. The control device according to claim 1.

8. The third subject is a subject different from the first subject.

2. The control device according to claim 1.

9. The fourth subject is a subject different from the second subject.

2. The control device according to claim 1.

10. The imaging device further includes a dividing unit that divides the plurality of second imaging devices into the first group and the second group based on the positions of the plurality of second imaging devices.

2. The control device according to claim 1.

11. The dividing means divides the second imaging devices into the first group and the second group based on at least one of a relative position of each of the second imaging devices and the fourth object and a relative position of each of the second imaging devices and the third object. The control device according to claim 10 .

12. The dividing means divides the second imaging devices into the first group and the second group based on at least one of a distance between the position of each of the second imaging devices and the fourth object and a distance between the position of each of the second imaging devices and the third object. The control device according to claim 10 .

13. The dividing means divides the second imaging devices into the first group and the second group so that adjacent second imaging devices are not included in the same group. The control device according to claim 10 .

14. The dividing means divides the second imaging devices into the first group and the second group based on the positions of the second imaging devices, the movable areas of the imaging directions of the second imaging devices, and the position of the third subject. The control device according to claim 10 .

15. The dividing means divides the second imaging devices into the first group and the second group based on the positions of the second imaging devices, the movable areas of the imaging directions of the second imaging devices, and the position of the fourth subject. The control device according to claim 10 .

16. The camera further includes a dividing unit that divides the plurality of second image capturing devices into the first group and the second group based on a user operation.

2. The control device according to claim 1.

17. an acquisition step of acquiring information indicating a subject to be photographed by the first imaging device; a control step of controlling the photographing directions of the plurality of second photographing devices so that the second photographing devices are oriented toward a photographing target corresponding to the photographing target of the first photographing device; and In the control step, when the photographing direction of the first imaging device is changed from a direction facing a first subject to a direction facing a second subject, changing the imaging direction of the second imaging devices included in a second group from a direction facing a third subject corresponding to the first subject to a direction facing a fourth subject corresponding to the second subject, changing the imaging direction of the second imaging devices included in a first group from a direction facing the third subject to a direction facing the fourth subject; A control method comprising:

18. A program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 16.

Images