Control apparatus and control method

Abstract

To provide a control apparatus that can be controlled so as to obtain a desired image when changing a subject in photographing by a plurality of cameras.SOLUTION: A control apparatus includes: acquisition means of acquiring information indicating a photographing object of a first imaging device; and control means of controlling a photographing direction of a plurality of second imaging devices so as to be directed to the photographing object corresponded to the photographing object of the first imaging device. The control means does not change a photographing direction of the second imaging device included in a first group from a direction directed to a third subject corresponded to the first subject from the plurality of second imaging devices in the case where the photographing direction of the first imaging device is changed to a direction directed to a second subject from a direction directed to the first subject, and changes the photographing direction of the second imaging device included in a second group from the plurality of second imaging devices from a direction directed to the third subject to a direction directed to a fourth subject corresponded to the second subject.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to a control device and a control method for controlling photography by a plurality of cameras. [Background technology]

[0002] In recent years, a system that uses multiple cameras for shooting has become known. By arranging a main camera and multiple sub-cameras different from the main camera among the multiple cameras, it becomes possible to obtain multiple shooting angles of view. In shooting with multiple cameras, it is desired to reduce the number of cameramen who shoot the cameras in order to reduce shooting costs.

[0003] To reduce manpower, it is possible to use cameras called PTZ cameras, which allow remote adjustment of pan, tilt, and zoom. PTZ cameras use technology to automatically track targets detected from captured images, and can automatically control pan, tilt, and zoom so that the target is located at the desired position within the camera's field of view.

[0004] As a technology related to automatic control of multiple cameras, Patent Document 1 discloses a technology for determining the shooting direction of a camera other than a parent camera based on the position of a subject photographed by the parent camera. Patent Document 2 discloses a technology for transmitting subject information in a main imaging device to a sub imaging device, and the sub imaging device automatically photographs based on the subject information. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2003-348428 A [Patent Document 2] JP 2020-025248 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, while the subject is changed by the main camera and each camera is changing its direction, it may not be possible to obtain an image of the desired subject. For example, when shooting a live performance, while the shooting direction of the multiple sub-cameras is changed in accordance with the change in subject of the main camera, the images of each sub-camera are moving images, so the user may not be able to obtain the desired image.

[0007] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide a control device capable of performing control so as to obtain a desired image when changing the subject in photographing with multiple cameras. [Means for solving the problem]

[0008] The control device of the present invention has an acquisition means for acquiring information indicating a subject to be photographed by a first imaging device, and a control means for controlling the shooting directions of a plurality of second imaging devices so that they face a subject to be photographed corresponding to the subject to be photographed by the first imaging device, and when the shooting direction of the first imaging device is changed from a direction facing a first subject to a direction facing a second subject, the control means is characterized in that, among the plurality of second imaging devices, the shooting direction of the second imaging device included in a first group is not changed from a direction facing a third subject corresponding to the first subject, and the shooting direction of the second imaging device included in a second group among the plurality of second imaging devices is changed from a direction facing the third subject to a direction facing a fourth subject corresponding to the second subject. Effect of the Invention

[0009] According to the present invention, when changing the subject in photographing with multiple cameras, the desired image is It can be controlled to obtain. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an imaging system. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a control device and an overhead camera. [Diagram 3] FIG. 2 is a diagram illustrating an example of the configuration of a sub camera, a main camera, and a role control device. [Figure 4] FIG. 2 is a block diagram illustrating a functional configuration of a control device. [Diagram 5] FIG. 11 is a diagram illustrating role information. [Figure 6] FIG. 1 is a diagram illustrating transformation of a coordinate system. [Figure 7] FIG. 4 is a diagram illustrating a method for detecting the position of a subject. [Figure 8] 11A and 11B are diagrams showing a specific example of changing a subject to be photographed by a sub camera. [Figure 9] FIG. 13 is a diagram illustrating grouping of sub-cameras. [Figure 10] FIG. 13 is a diagram illustrating a target position of panning. [Figure 11] FIG. 13 is a diagram illustrating a target position of tilt. [Figure 12A] 11 is a flowchart illustrating a control process of the shooting direction of the sub camera. [Figure 12B] 11 is a flowchart illustrating a control process of the shooting direction of the sub camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Embodiment> Hereinafter, an imaging system according to an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram illustrating the configuration of the imaging system. The imaging system includes a control device 100, an overhead camera 300, a plurality of sub-cameras 400, a main camera 500, and a role control device 600. Each component of the imaging system is connected via, for example, a LAN (Local Area Network) 700, and each device can communicate with each other.

[0012] The overhead camera 300 and the main camera 500 transmit images of the subject to the control device 100 via a video line (not shown). The overhead camera 300 and the main camera 500 also transmit shooting information to the control device 100 via a LAN 700. The overhead camera 300 is a camera that shoots with a wide angle of view so that the shooting environment can be shot from a bird's-eye view. The overhead camera 300 may be a PTZ camera. The main camera 500 is a camera that shoots by manually operating it by the user, or a PTZ camera. The sub-camera 400 is a PTZ camera that operates in cooperation with the main camera 500, and is driven based on the pan-tilt-zoom value calculated by the control device 100. The shooting system includes a plurality of sub-cameras 400.

[0013] The control device 100 controls the shooting directions of the multiple sub-cameras 400 so that they face a subject corresponding to the subject of shooting of the main camera 500. The control device 100 calculates pan, tilt, and zoom values ​​for driving the sub-cameras 400 using role information set by the role control device 600 and information acquired from images shot by the overhead camera 300 and the main camera 500.

[0014] The role information is information on the subject of the sub-camera 400 corresponding to the subject that is the subject of the main camera 500. The role information may include information on whether the camera included in the shooting system is the main camera 500 or the sub-camera 400. The information acquired from the image captured by the overhead camera 300 includes position information of the main camera 500 and the sub-camera 400. The information acquired from the image captured by the main camera 500 includes information on the subject that is the subject of the main camera 500. The control device 100 transmits the calculated pan / tilt / zoom values ​​to the sub-camera 400. The control device 100 is a workstation or an edge AI device, etc.

[0015] The role control device 600 controls the sub-camera 4 in cooperation with the operation of the main camera 500. The role control device 600 sets the role of the sub camera 400 as to which subject the sub camera 400 is to capture, corresponding to the subject that is the subject of the main camera 500's capture. The role control device 600 can set the role of the sub camera 400, for example, in response to a user's instruction. The role control device 600 is an electronic device such as a notebook PC (personal computer) or a switch controller (equipment that can assign macros to buttons). The user can set the role of the sub camera 400, for example, via a user interface (UI) of the notebook PC or by operating the buttons of the switch controller.

[0016] The configurations of the control device 100, the overhead camera 300, the sub-camera 400, the main camera 500, and the role control device 600 will be described in detail with reference to FIGS. 2(A), 2(B) and 3(A) to 3(C).

[0017] 2(A) is a diagram showing an example of the configuration of the control device 100. The control device 100 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, and a ROM (Read Only Memory) 103. The control device 100 also includes an inference unit 104, a network interface (I / F) 105, and an input unit 106. The components included in the control device 100 are connected to each other via an internal bus 110.

[0018] The CPU 101 controls the entire control device 100. The RAM 102 is a high-speed storage device such as a dynamic random access memory (DRAM). The CPU 101 temporarily loads the OS, various programs, and various data into the RAM 102 and executes various processes. The RAM 102 is also used as a working area when the CPU 101 executes the OS and various programs.

[0019] The ROM 103 is a non-volatile storage device such as a flash memory, an HDD, an SSD, or an SD card. The ROM 103 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0020] The inference unit 104 reads out images captured by the overhead camera 300, the sub camera 400, and the main camera 500 from the RAM 102, and estimates the positions and presence or absence of objects such as the camera and the subject from the read-out captured images. The inference unit 104 is, for example, a calculation device specialized for image processing and inference processing, such as a GPU (Graphics Processing Unit). The GPU is effective as a calculation device used for learning processing. The inference unit 104 may be a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array). The processing of the inference unit 104 may be realized by the CPU 101.

[0021] The network I / F 105 is an interface for connecting to the LAN 700, and controls communication with external devices such as the sub-camera 400 via a communication medium such as Ethernet (registered trademark). The network I / F 105 may be a serial communication interface or the like.

[0022] 2(B) is a diagram showing an example of the configuration of the overhead camera 300. The overhead camera 300 includes a CPU 301, a RAM 302, a ROM 303, a network I / F 305, an image processing unit 306, and an image sensor 307. The components included in the overhead camera 300 are connected to each other via an internal bus 310.

[0023] The CPU 301 controls the entire overhead camera 300. The RAM 302 is a memory such as a DRAM. It is a high-speed storage device. The CPU 301 temporarily loads the OS, various programs, and various data into the RAM 102 to execute various processes. The RAM 302 is also used as a work area when the CPU 301 executes the OS and various programs.

[0024] The ROM 303 is a non-volatile storage device such as a flash memory, an HDD, an SSD, an SD card, etc. The ROM 303 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0025] The image processing unit 306 is connected to an image sensor 307 such as a CCD or a CMOS. The image processing unit 306 converts image data acquired from the image sensor 307 into a predetermined format and transfers it to the RAM 302. The image processing unit 306 may compress the converted image data and transfer it to the RAM 302.

[0026] When acquiring an image from the image sensor 307, the image processing unit 306 can perform image processing such as color correction, exposure control, and sharpness correction on the image. The image processing unit 306 may also perform cropping to cut out a partial area of ​​the image data. The processing by the image processing unit 306 may be performed based on an instruction received from an external device such as the control device 100 via the network I / F 305.

[0027] The network I / F 305 is an interface for connecting to the LAN 700, and controls communication with an external device such as the control device 100 via a communication medium such as Ethernet (registered trademark). The network I / F 305 may be a serial communication interface or the like.

[0028] The photographing system includes a plurality of sub-cameras 400, and the main camera 500 photographs in cooperation with the plurality of sub-cameras 400. FIG. 3(A) is a diagram showing an example of the configuration of the sub-camera 400. Each of the plurality of sub-cameras 400 included in the photographing system has a configuration similar to that shown in FIG. 3(A). The sub-camera 400 includes a CPU 401, a RAM 402, a ROM 403, a network I / F 405, an image processing unit 406, an image sensor 407, a driving I / F 408, a driving unit 409, and a focus control unit 411. The components included in the sub-camera 400 are connected to each other via an internal bus 410.

[0029] The CPU 401 performs overall control of the sub-camera 400. The RAM 402 is a high-speed storage device such as a DRAM. The CPU 401 temporarily loads the OS, various programs, and various data into the RAM 402 and executes various processes. The RAM 402 is also used as a working area when the CPU 401 executes the OS and various programs.

[0030] The ROM 403 is a non-volatile storage device such as a flash memory, an HDD, an SSD, or an SD card. The ROM 403 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0031] The image processing unit 406 is connected to an image sensor 407 such as a CCD or CMOS. The image processing unit 406 converts image data acquired from the image sensor 407 into a predetermined format and transfers it to the RAM 402. The image processing unit 406 may compress the converted image data and transfer it to the RAM 402.

[0032] When acquiring an image from the image sensor 407, the image processing unit 406 can perform image processing such as color correction, exposure control, and sharpness correction on the image. The image processing unit 406 may also perform crop processing to cut out a partial area of ​​the image data. The processing by the image processing unit 406 is performed in response to a command from the control device 100 or the like via the network I / F 405. It may be performed based on instructions received from an external device.

[0033] The network I / F 405 is an interface for connecting to the LAN 700, and controls communication with an external device such as the control device 100 via a communication medium such as Ethernet (registered trademark). The network I / F 405 may be a serial communication interface or the like.

[0034] The drive I / F 408 is a connection unit with the drive unit 409, and transmits and receives control signals and the like to the drive unit 409. The drive unit 409 is a rotation mechanism for changing the shooting direction of the sub camera 400, and includes a mechanical drive system and a drive source motor, etc. Based on an instruction received from the CPU 401 via the drive I / F 408, the drive unit 409 drives rotations such as a pan operation for changing the shooting angle of view in the horizontal direction and a tilt operation for changing the shooting angle of view in the vertical direction, and a zoom operation for optically changing the shooting angle of view.

[0035] The focus control unit 411 controls the focus when the image sensor 407 captures an image. As a focusing method, a known technique such as a contrast detection method or a phase difference detection method can be used.

[0036] Fig. 3(B) is a diagram showing an example of the configuration of main camera 500. In the following description, it is assumed that main camera 500 is manually operated by a user. Specifically, the user can perform PTZ control of main camera 500 using a camera controller (not shown). Main camera 500 is not limited to a PTZ camera, and may be a camera configured to capture an image by manually and physically moving it by the user.

[0037] The control device 100 can acquire orientation information of the main camera 500 (angles of pan and tilt based on a frontally facing installation) via the network I / F 505 and the LAN 700.

[0038] Main camera 500 includes a CPU 501, RAM 502, ROM 503, network I / F 505, image processing unit 506, image sensor 507, drive I / F 508, and drive unit 509. The components included in main camera 500 are connected to each other via internal bus 510.

[0039] The CPU 501 performs overall control of the main camera 500. The RAM 502 is a high-speed storage device such as a DRAM. The CPU 501 temporarily loads the OS, various programs, and various data into the RAM 502 and executes various processes. The RAM 502 is also used as a working area when the CPU 501 executes the OS and various programs.

[0040] The ROM 503 is a non-volatile storage device such as a flash memory, an HDD, an SSD, or an SD card. The ROM 503 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0041] The image processing unit 506 is connected to an image sensor 507 such as a CCD or a CMOS. The image processing unit 506 converts image data acquired from the image sensor 507 into a predetermined format and transfers the converted image data to the RAM 502. The image processing unit 506 may compress the converted image data and transfer it to the RAM 502.

[0042] The image processing unit 506 can perform image processing such as color correction, exposure control, and sharpness correction on the image when acquiring the image from the image sensor 507. The image processing unit 506 may also perform cropping to cut out a partial area of ​​the image data. The processing by the image processing unit 506 may be performed based on an instruction received from an external device such as the control device 100 via the network I / F 505.

[0043] The network I / F 505 is an interface for connecting to the LAN 700, and controls communication with an external device such as the control device 100 via a communication medium such as Ethernet (registered trademark). The network I / F 505 may be a serial communication interface or the like.

[0044] The drive I / F 508 is a connection unit with the drive unit 509, and transmits and receives control signals and the like to the drive unit 509. The drive unit 509 is a rotation mechanism for changing the shooting direction of the sub camera 400, and includes a mechanical drive system and a drive source motor, etc. Based on an instruction received from the CPU 501 via the drive I / F 508, the drive unit 509 drives rotations such as a pan operation for changing the shooting angle of view in the horizontal direction and a tilt operation for changing the shooting angle of view in the vertical direction, and a zoom operation for optically changing the shooting angle of view.

[0045] 3(C) is a diagram showing an example of the configuration of the role control device 600. The role control device 600 includes a CPU 601, a RAM 602, a ROM 603, a network I / F 605, and an input unit 611. The components included in the role control device 600 are connected to each other via an internal bus 610.

[0046] The CPU 601 controls the entire role control device 600. The RAM 602 is a high-speed storage device such as a DRAM. The CPU 601 temporarily loads the OS, various programs, and various data into the RAM 602 and executes various processes. The RAM 602 is also used as a working area when the CPU 601 executes the OS and various programs.

[0047] The ROM 603 is a non-volatile storage device such as a flash memory, an HDD, an SSD, or an SD card. The ROM 603 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0048] The network I / F 605 is an interface for connecting to the LAN 700, and controls communication with an external device such as the control device 100 via a communication medium such as Ethernet (registered trademark). The network I / F 605 may be a serial communication interface or the like.

[0049] The input unit 611 includes operation members such as a button, a mouse, and a keyboard. A user can input role information by controlling a GUI using the operation members. The CPU 601 transmits the role information designated by the user to the control device 100. Note that the role control device 600 may be configured together with the control device 100 as a single device.

[0050] Specific processing of the control device 100 will be described with reference to Fig. 4. The control device 100 calculates the pan / tilt / zoom values ​​of each of the multiple sub-cameras 400 and controls the shooting direction of each sub-camera 400 so that the sub-cameras 400 face a subject corresponding to the subject shot by the main camera 500. In the example of Fig. 4, the multiple sub-cameras 400 are sub-camera 400a and sub-camera 400b, and are also collectively referred to as sub-camera 400.

[0051] Fig. 4 is a block diagram illustrating a functional configuration of the control device 100. The processing of each functional unit of the control device 100 is realized using the hardware resources shown in Fig. 2(A). A detailed description of the processing of software that controls the communication function and general-purpose software such as an operating system will be omitted.

[0052] Control device 100 includes role setting section 120, first recognition section 121, main subject determination section 122, tracking subject determination section 123, shooting direction acquisition section 124, and second recognition section 125. Programs for realizing the processing of each functional section are stored in ROM 103. CPU 101 can realize the processing of each functional section by reading out the programs stored in ROM 103 into RAM 102 and executing them.

[0053] The role setting unit 120 sets a subject to be photographed by the sub camera 400 based on the role information set by the role control device 600. The role information includes information on a subject to be photographed by the sub camera 400 (a tracking subject) corresponding to a subject to be photographed by the main camera 500 (a main subject).

[0054] A user can set role information via the input unit 611 of the role control device 600. Specifically, the user operates the GUI of the role control device 600 with an operating member such as a keyboard to set a subject to be photographed by the main camera 500 and a subject to be photographed by the sub-camera 400 corresponding to the subject to be photographed by the main camera 500.

[0055] Fig. 5 is a diagram explaining role information. A user can register multiple patterns of role information in advance in the role control device 600 as shown in Fig. 5. For example, in pattern 1, the subject of shooting of the sub-camera 400 corresponding to subject A, which is the subject of shooting before the change of the main camera 500, is subject A. Also, the subject of shooting of the sub-camera 400 corresponding to subject B, which is the subject of shooting after the change of the main camera 500, is subject B. Pattern 1 is a pattern in which the subject of the main camera 500 and the subject of the sub-camera 400 are the same.

[0056] On the other hand, in pattern 5, the subject captured by the sub-camera 400 corresponding to subject A, which is the subject captured before the change of the main camera 500, is subject B. Also, the subject captured by the sub-camera 400 corresponding to subject C, which is the subject captured after the change of the main camera 500, is subject D. Pattern 5 is a pattern in which the subject of the main camera 500 and the subject of the sub-camera 400 are different.

[0057] Also, as in patterns 1 and 2, the subject of the main camera 500 before the change and the subject of the sub camera 400 before the change may be the same subject. As in patterns 4 and 7, the subject of the main camera 500 before the change and the subject of the sub camera 400 after the change may be the same subject. As in patterns 4 and 6, the subject of the main camera 500 after the change and the subject of the sub camera 400 before the change may be the same subject. As in patterns 1 and 3, the subject of the main camera 500 after the change and the subject of the sub camera 400 after the change may be the same subject.

[0058] As in patterns 3 to 7, the subject of main camera 500 before the change and the subject of sub camera 400 before the change may be different subjects. As in patterns 2 and patterns 4 to 7, the subject of main camera 500 after the change and the subject of sub camera 400 after the change may be different subjects.

[0059] The role control device 600 transmits role information ROLE designated by the user from among multiple patterns illustrated in Fig. 5 to the role setting unit 120 of the control device 100. The role setting unit 120 sets a correspondence relationship between the subject photographed by the main camera 500 and the subject photographed by the sub camera 400 based on the role information ROLE received from the role control device 600. The role setting unit 120 transmits the set role information ROLE to the tracking subject determination unit 123.

[0060] If the user specifies incompatible patterns such as pattern 1 and pattern 2, the role setting unit 120 determines the shooting mode of the sub-camera 400 based on the pattern registered later. In addition, in a case where the subject of the main camera 500 is changed from subject A to subject D, which is not included in the pattern specified by the user, the role setting unit 120 may set the subject of the sub camera 400 to the same subject as the main camera 500, for example.

[0061] The first recognition unit 121 reads out from the RAM 102 the image IMG captured by the overhead camera 300 and reference position information REF_POSI including information on the installation position of each camera and information on marker coordinates. The first recognition unit 121 uses the information read out from the RAM 102 to detect a person (subject) to be photographed and acquires position information POSITION of the detected subject. When multiple subjects are detected, the position information POSITION includes position information of each of the subjects. The first recognition unit 121 outputs the position information POSITION of the detected subjects and the identification information ID of each subject.

[0062] The installation position of each camera can be measured manually or by using a sensor (not shown) as the camera's position in a plane coordinate system when the shooting area is viewed from directly above. Information on the measured installation position of each camera is input to the control device 100 via the input unit 106 by, for example, a user. The CPU 101 writes the input information on the installation position of each camera in the RAM 102. Alternatively, information on the measured installation position of each camera may be transmitted from the sensor to the control device 100 and written to the RAM 102 by the CPU 101.

[0063] The marker is installed so as to be recognizable when viewed from directly above the photographed area in order to calculate a homography transformation matrix used in coordinate transformation. The marker coordinates in the coordinate system of the photographed area can be measured manually or by using a sensor (not shown). The marker is a mark of a color different from the color of the floor or the ground, but may be in any form as long as it can be measured manually or by a sensor. When the sensor that measures the marker is a camera, by installing a mark of a specific color as a marker, the camera can extract the color of the marker from the captured image of the photographed area and obtain the marker coordinates. Information on the marker coordinates is input to the control device 100 via the input unit 106 by, for example, a user. The CPU 101 writes the input information on the marker coordinates to the RAM 102. Information on the measured marker coordinates may also be transmitted from the sensor to the control device 100 and written to the RAM 102 by the CPU 101.

[0064] The reference position information REF_POSI and the subject position information POSITION are expressed by coordinates in a planar coordinate system of the shooting area viewed from directly above. For example, when the first recognition unit 121 detects three subjects, it outputs the position information POSITION and the identification information ID of the three subjects. The first recognition unit 121 writes the output information to the RAM 102.

[0065] A method for detecting the position of a subject present in a shooting area and acquiring position information POSITION of the detected subject will be described with reference to Figures 6(A) and 6(B). Figure 6(A) shows the coordinate system of the image captured by overhead camera 300. Figure 6(B) shows the coordinate system of the shooting area viewed from directly above. Based on the relationship between the coordinate system of the image captured by overhead camera 300 and the coordinate system of the shooting area viewed from directly above, first recognition unit 121 acquires the coordinates of the subject in the coordinate system of the shooting area viewed from directly above as position information POSITION.

[0066] When calculating the pan value so that the sub-cameras 400a and 400b face the direction of the subject to be tracked, the angle calculation is simplified if the calculation is performed in a plane coordinate system perpendicular to the axis of the panning motion. For example, when the sub-cameras 400a and 400b are installed perpendicular to the floor or ground, the coordinate plane perpendicular to the axis of the panning motion is parallel to the floor or ground and is a coordinate plane in which the space in which the subject exists is viewed from directly above.

[0067] In the following description, sub-camera 400a and sub-camera 400b are installed perpendicular to the floor or ground, and the pan value of each sub-camera 400 is calculated in the coordinate system of the shooting area viewed from directly above. The first recognition unit 121 converts the coordinates of the subject in the coordinate system of the image captured by overhead camera 300 shown in Fig. 6(A) (hereinafter referred to as overhead camera coordinate system) into coordinates in the coordinate system of the shooting area viewed from directly above (hereinafter referred to as planar coordinate system) shown in Fig. 6(B). This allows the first recognition unit 121 to calculate the pan value of each sub-camera 400 in the coordinate system of the shooting area viewed from directly above.

[0068] The coordinates of the subject in the overhead camera coordinate system are transformed into coordinates in the planar coordinate system using the homography transformation matrix H according to the following formula 1.

number

[0069] The first recognition unit 121 reads out the reference position information REF_POSI from the RAM 102. The marker coordinates of the markers Mark_A to Mark_D included in REF_POSI are coordinates in a planar coordinate system of the shooting area viewed from directly above, and are substituted into the left side of Equation 1. The first recognition unit 121 also acquires the marker coordinates in the overhead camera coordinate system from the image IMG captured by the overhead camera 300, and substitutes them into the right side of Equation 1. By substituting the marker coordinates in the overhead camera coordinate system and the marker coordinates in the planar coordinate system into Equation 1, a homography transformation matrix H is obtained.

[0070] If the homography transformation matrix H is obtained, the first recognition unit 121 can map the overhead camera coordinate system shown in FIG. 6(A) to the planar coordinate system shown in FIG. 6(B) using Equation 1. That is, the first recognition unit 121 can convert any coordinate in the overhead camera coordinate system to a corresponding coordinate in the planar coordinate system. Therefore, the first recognition unit 121 can grasp the positions of the subjects A, B, and C captured in the image IMG captured by the overhead camera 300 in the planar coordinate system of FIG. 6(B). The first recognition unit 121 writes the homography transformation matrix H obtained using the marker coordinates to the RAM 102.

[0071] 7(A) and 7(B), a method for detecting the position of a subject from a captured image IMG input from the overhead camera 300 using an inference model for subject detection will be described. The inference model for subject detection is stored in advance in ROM 103. When detecting a person as the subject, for example, the first recognition unit 121 can detect the person from the captured image IMG of the overhead camera 300 using a trained person detection inference model created by a machine learning method such as deep learning. The person detection inference model receives an image as input and outputs the coordinates within the image of a person appearing in the image.

[0072] The first recognition unit 121 (inference unit 104) inputs the captured image IMG input from the overhead camera 300 into a person detection inference model, and detects people present in the image. FIG. 7(A) shows people detected by the person detection inference model surrounded by a rectangle. In the example of FIG. 7(A), subjects A, B, and C are detected from the captured image IMG of the overhead camera 300. The first recognition unit 121 writes information on the positions (coordinates) of people detected in the image captured by the overhead camera 300 into the RAM 102.

[0073] The first recognition unit 121 is not limited to the machine learning method, and may use, for example, a Scale-Invariant Feature Transform (SIFT) method that detects a person as a subject by matching local feature points in an image. The first recognition unit 121 may detect a person using a template matching method that detects a subject by calculating the similarity with a template image.

[0074] The first recognition unit 121 determines a point on a rectangle surrounding the subject detected in the overhead camera coordinate system shown in Fig. 7(A), for example, the midpoint of the bottom side (the side on the foot side) of the rectangle, as the subject detection position. The first recognition unit 121 converts the coordinates of the subject detection position into coordinates in the plane coordinate system shown in Fig. 7(B) using Equation 1.

[0075] Specifically, the first recognition unit 121 reads out the homography transformation matrix H from the RAM 102, and can convert the coordinates (xa, ya) of the detection position of the subject A in the overhead camera coordinate system into coordinates (XA, YA) in the planar coordinate system by substituting them into x and y in the formula 1. Similarly, the first recognition unit 121 can convert the coordinates (xb, yb) and (xc, yc) of the detection positions of the subjects B and C into coordinates (XB, YB) and (XC, YC) in the planar coordinate system by substituting them into x and y in the formula 1. The first recognition unit 121 writes the converted coordinates of the subjects A, B, and C into the RAM 102 as position information POSITION of the detected subjects.

[0076] The first recognition unit 121 outputs the identification information ID of the detected subject in addition to the position information POSITION of the detected subject. The first recognition unit 121 can acquire the identification information ID of the detected subject by using a trained model for identifying a person stored in the ROM 103. The trained model is a trained model that is generated by using a machine learning method such as deep learning to train a past captured image and a detection result in the past captured image, and to output the identification information ID of the subject.

[0077] The first recognition unit 121 can input the image captured by the overhead camera 300 and the position information of the subject detected in the image captured by the overhead camera 300 to the trained model for person identification read from the ROM 103 to obtain the identification information ID. The trained model for person identification can be generated by a known technology. For example, the trained model for person identification can be generated by learning so that past frame information and the detection results in the past frame information are input and an identification information ID is output. The first recognition unit 121 can input the current frame information to the trained model for person identification to obtain the identification information ID of the subject detected in the current frame.

[0078] The main subject determination unit 122 determines a main subject to be photographed by the main camera 500 based on the orientation of the main camera 500 and the position information of the subject. The main subject determination unit 122 acquires information on the determined main subject as information indicating the photographing subject. The main subject determination unit 122 determines a main subject to be photographed by the main camera 500 based on the subject position information POSITION acquired by the first recognition unit 121 and the orientation information ANGLE input from the main camera 500. For example, the main subject determination unit 122 can determine a subject that is closer to the orientation (on the optical axis) of the main camera 500 as the main subject among the subjects detected by the first recognition unit 121. The main subject determination unit 122 writes the identification information MAIN_SUBJECT_ID of the determined main subject to the RAM 102.

[0079] Note that main subject determination section 122 may detect the main subject from an image captured by main camera 500. Main subject determination section 122 can determine, for example, a subject that exists closer to the center of the image captured by main camera 500 as the main subject.

[0080] The tracking subject determination unit 123 determines a tracking subject that is a subject to be photographed by the sub camera 400, based on the main subject and the correspondence (role information ROLE) between the subject to be photographed by the main camera 500 and the subject to be photographed by the sub camera 400. The tracking subject determination unit 123 transmits information on the determined tracking subject to the photographing direction acquisition unit 124.

[0081] The tracking subject determination unit 123 can determine the tracking subjects of the sub-cameras 400a and 400b, for example, as follows. The tracking subject determination unit 123 acquires subject information DETECT_INFO, which is information on the subject detected by the second recognition unit 125 from the captured images of the sub-cameras 400a and 400. The tracking subject determination unit 123 determines the tracking subject based on the identification information MAIN_SUBJECT_ID of the main subject, the subject information DETCT_INFO, and the role information ROLE. The tracking subject determination unit 123 transmits the identification information SUBJECT_ID of the determined tracking subject to the shooting direction acquisition unit 124.

[0082] When the subject of shooting of main camera 500 is changed, tracking subject determination unit 123 receives identification information MAIN_SUBJECT_ID of the main subject of the changed shooting subject from main subject determination unit 122. Tracking subject determination unit 123 acquires identification information SUBJECT_ID of the tracking subject corresponding to the changed main subject, and transmits it to shooting direction acquisition unit 124.

[0083] The tracking subject determination unit 123 determines whether the tracking subject is closer to the sub-camera 400a or the sub-camera 400b from the image IMG captured by the overhead camera 300. The tracking subject determination unit 123 transmits identification information SUBJECT_ID of the tracking subject to the imaging direction acquisition unit 124, and instructs it to change the imaging direction of the sub-camera 400 (here, sub-camera 400a) that is closer to the tracking subject.

[0084] After the shooting direction of the sub-camera 400a closer to the tracking subject is changed, the tracking subject determination unit 123 receives the subject information DETECT_INFO of the subject detected from the captured image SUB_IMG of the sub-camera 400a. The tracking subject determination unit 123 confirms that the tracking subject with the identification information SUBJECT_ID is captured in the captured image SUB_IMG of the sub-camera 400a. After confirmation, the tracking subject determination unit 123 transmits the identification information SUBJECT_ID of the tracking subject to the shooting direction acquisition unit 124, and instructs it to change the shooting direction of the sub-camera 400b, whose shooting direction has not been changed.

[0085] 8(A) to 8(D), a specific example will be described in which the shooting target of the main camera 500 is changed, and the shooting target of the sub-camera 400 is changed. The shooting directions of the sub-camera 400a and the sub-camera 400b are changed in the order from FIG. 8(A) to FIG. 8(D).

[0086] In the examples of Figures 8(A) to 8(D), when the main camera 500 is shooting subject B as the main subject, the sub-cameras 400a and 400b shoot subject D. When the subject of the main camera 500 is changed from subject B to subject A, the subject of the sub-cameras 400a and 400b is changed to subject C. However, the subject of the sub-cameras 400a and 400b is changed to subject C at different times.

[0087] Fig. 8(A) shows a state in which main camera 500 is photographing subject B, and sub-camera 400a and sub-camera 400b are photographing subject D corresponding to subject B. In Fig. 8(B), the subject photographed by main camera 500 is changed from subject B to subject A, and the photographing direction of main camera 500 is changed from a direction facing subject B to a direction facing subject A. The subject photographed by sub-camera 400 corresponding to subject A is subject C.

[0088] 8(C), the shooting direction of sub-camera 400a, which is closer to subject C among sub-cameras 400, is changed from a direction facing subject D to a direction facing subject C corresponding to subject A. At this point, the shooting direction of sub-camera 400b remains unchanged, facing subject D corresponding to subject B.

[0089] In FIG. 8(D), the shooting direction of the sub-camera 400b is changed from the direction facing the subject D to the direction facing the subject C. In this way, after the shooting direction of the sub-camera 400a is first changed to the direction facing the subject C, the shooting direction of the sub-camera 400b is changed from the direction facing the subject D to the direction facing the subject C. Therefore, while the shooting direction of the sub-camera 400a is changed from the direction facing the subject D to the direction facing the subject C, the sub-camera 400b can record the shot image of the subject D. Also, while the shooting direction of the sub-camera 400b is changed from the direction facing the subject D to the direction facing the subject C, the sub-camera 400a can record the shot image of the subject C.

[0090] 8C shows an example in which the shooting direction of the sub-camera 400a closer to the subject C is changed first, but which sub-camera 400 has its shooting direction changed first is not limited to being determined based on the distance to the tracking subject. The tracking subject determination unit 123 may determine which sub-camera 400 has its shooting direction changed first based on, for example, a user operation.

[0091] Furthermore, the tracking subject determination unit 123 may determine which of the sub-cameras 400 should have its shooting direction changed first, based on the position of the sub-camera 400, the movable area of ​​the shooting direction (optical axis direction) of the sub-camera 400, and the position of the tracking subject. For example, the tracking subject determination unit 123 may first change the shooting direction of the sub-camera 400 in which the tracking subject after the change is included in the movable area of ​​the shooting direction. Furthermore, the tracking subject determination unit 123 may later change the shooting direction of the sub-camera 400 that can shoot the tracking subject before the change at the center of the screen (either one of the sub-cameras 400a and 400b if both can shoot at the center of the screen).

[0092] 8(A) to 8(D) show an example in which there are two sub-cameras 400, but there may be three or more sub-cameras 400. When there are three or more sub-cameras 400 and the shooting direction of the main camera 500 is changed, the tracking subject determination unit 123 divides these multiple sub-cameras 400 into multiple groups that have different timings for changing the shooting direction.

[0093] For example, when an additional sub-camera 400 not shown in FIG. 8B is present, the tracking subject determination unit 123 divides the multiple sub-cameras 400 into a first group including the sub-camera 400b and a second group including the sub-camera 400a. When the shooting direction of the main camera 500 is changed from the subject B to the subject A, the tracking subject determination unit 123 does not change the shooting direction of the sub-cameras 400 included in the first group among the multiple sub-cameras 400 from the direction facing the subject D corresponding to the subject B. On the other hand, when the shooting direction of the main camera 500 is changed, the tracking subject determination unit 123 changes the shooting direction of the sub-cameras 400 included in the second group among the multiple sub-cameras 400 from the direction facing the subject D to the direction facing the subject C corresponding to the subject A.

[0094] The tracking subject determination unit 123 can divide the multiple sub-cameras 400 into a first group and a second group by various methods. For example, the tracking subject determination unit 123 can divide the multiple sub-cameras 400 into a first group and a second group based on the positions of each of the multiple sub-cameras 400.

[0095] More specifically, the tracking subject determination unit 123 determines the distance between each of the positions of the multiple sub-cameras 400 and the tracking subject corresponding to the subject before the shooting direction of the main camera 500 is changed. 8B, the tracking subject determination unit 123 later changes the shooting direction of the sub-cameras 400 in the first group that is closer to the subject D, for example.

[0096] Furthermore, the tracking subject determination unit 123 may group the sub-cameras 400 based on the position of each of the sub-cameras 400 and the distance to the tracking subject corresponding to the subject after the shooting direction of the main camera 500 is changed. In the example of Fig. 8(B), the tracking subject determination unit 123 first changes the shooting direction of the sub-cameras 400 of the second group that is closer to the subject C, for example. Note that the tracking subject determination unit 123 may group the sub-cameras 400 based on both the distance to the subject D and the distance to the subject C.

[0097] The tracking subject determination unit 123 may group the sub-cameras 400 based on the relative position to the tracking subject, not limited to the distance to the tracking subject. The relative position to the tracking subject is represented by the distance from the sub-camera 400 to the tracking subject and the direction of the tracking subject with respect to the optical axis of the sub-camera 400. The tracking subject determination unit 123 can group the sub-cameras 400 based on the relative position to the tracking subject before the shooting direction of the main camera 500 is changed or the tracking subject after the change. The tracking subject determination unit 123 may group the sub-cameras 400 based on the relative position to the tracking subject before the shooting direction of the main camera 500 is changed and the tracking subject after the change.

[0098] The tracking subject determination unit 123 may divide the multiple sub-cameras 400 into a first group and a second group so that adjacent sub-cameras 400 are not included in the same group. By making adjacent sub-cameras 400 not included in the same group, the control device 100 can capture images of the tracking subject from various angles before and after changing the shooting direction.

[0099] The tracking subject determination unit 123 may group the sub-cameras 400 based not only on the distance to the tracking subject or the relative position to the tracking subject, but also on the movable range of the shooting direction of the sub-cameras 400. The movable range of the shooting direction indicates the range in which the direction of the optical axis of the sub-camera 400 can be changed.

[0100] 9(A) to 9(C), the grouping based on the movable area in the shooting direction of the sub-camera 400 will be described. The tracking subject determination unit 123 groups the multiple sub-cameras 400 based on the position of the sub-camera 400, the movable area in the shooting direction of the sub-camera 400, and the position of the tracking subject. The tracking subject 903 shown in Figs. 9(A) to 9(C) is the tracking subject corresponding to the subject after the shooting target of the main camera 500 is changed.

[0101] 9(A) shows sub camera 400a, a movable area 901 in the shooting direction of sub camera 400a, sub camera 400b, a movable area 902 in the shooting direction of sub camera 400b, and a tracking subject 903. The tracking subject 903 exists within the movable area 902 in the shooting direction of sub camera 400b. The tracking subject determination unit 123 sets the sub cameras 400 in which the tracking subject 903 exists within the movable area in the shooting direction as a second group, and changes the shooting direction of the sub cameras 400 in the second group first.

[0102] 9(B), like FIG. 9(A), shows sub camera 400a, movable area 901 in the shooting direction of sub camera 400a, sub camera 400b, movable area 902 in the shooting direction of sub camera 400b, and tracking subject 903. However, tracking subject 903 is not included in either movable area 901 in the shooting direction of sub camera 400a or movable area 902 in the shooting direction of sub camera 400b. The tracking subject determination unit 123, for example, classifies sub cameras 400 whose distance from the position of tracking subject 903 to their movable areas is shorter than a predetermined threshold as a second group, and In this case, the shooting direction of the sub-camera 400 can be changed first.

[0103] 9(C), like FIG. 9(A), shows sub camera 400a, movable area 901 in the shooting direction of sub camera 400a, sub camera 400b, movable area 902 in the shooting direction of sub camera 400b, and tracking subject 903. However, tracking subject 903 is included in both movable area 901 in the shooting direction of sub camera 400a and movable area 902 in the shooting direction of sub camera 400b. In this case, tracking subject determination unit 123 may group multiple sub cameras 400 based on conditions other than the movable areas.

[0104] 9(A) to 9(C), the tracking subject 903 is assumed to be a tracking subject corresponding to a subject after the change of the shooting subject of the main camera 500, but it may be a tracking subject corresponding to a subject before the change of the shooting subject of the main camera 500. That is, the tracking subject determination unit 123 may group a plurality of sub-cameras 400 based on the positions of the sub-cameras 400, the movable area of ​​the shooting direction of the sub-cameras 400, and the position of the tracking subject before the change. For example, the tracking subject determination unit 123 sets the sub-cameras 400 (or a part thereof) that can control the pan / tilt direction so that the tracking subject before the change is positioned in the optical axis direction as a first group, and later changes the shooting direction of the sub-cameras 400 in the first group.

[0105] The tracking subject determination unit 123 may also divide the multiple sub-cameras 400 into a first group and a second group based on a user operation. The tracking subject determination unit 123 may obtain grouping information stored in advance in the ROM 103 based on a user operation, or may obtain the grouping information by accepting an operation from the user during shooting.

[0106] The method of grouping the multiple sub-cameras 400 can be applied by appropriately combining the above various methods. In addition, the multiple sub-cameras 400 are not limited to two groups, and may be divided into three or more groups depending on the number of subjects to be tracked.

[0107] The shooting direction acquisition unit 124 receives information on the sub camera 400 whose shooting direction is to be changed and the identification information SUBJECT_ID of the tracking subject from the tracking subject determination unit 123 , and calculates the pan and tilt control values ​​of the sub camera 400 .

[0108] The shooting direction acquisition unit 124 reads out the coordinates of the sub camera 400 in the planar coordinate system contained in the reference position information REF_POSI and the coordinates of the tracked subject contained in the position information POSITION of the detected subject from the RAM 102. The shooting direction acquisition unit 124 calculates pan / tilt control values ​​for directing the sub camera 400 toward the tracked subject based on the coordinates of the sub camera 400 and the coordinates of the tracked subject. A method of calculating the pan / tilt target position as the pan / tilt control value will be described with reference to Figs. 10 and 11.

[0109] 10 is a diagram explaining the target position of panning. Angle θ indicating the target position of panning is the angle between a line extending from the center of the optical axis of the sub-camera 400 and a line connecting the sub-camera 400 and the tracking subject. Angle θ can be calculated by the following formula 2.

number

[0110] 11 is a diagram explaining the target position of tilt. Angle ρ indicating the target position of tilt is the angle between a line extending from the optical axis of sub-camera 400 at height h1 of the optical axis of sub-camera 400 and a line extending from sub-camera 400 toward the top of the head of the tracking subject at height h2. Angle ρ can be calculated by the following formulas 3 and 4.

number

[0111] The pan / tilt control value may be an angular velocity value for rotating the sub camera 400 in the direction of the tracking subject. An example of a method for calculating the pan / tilt angular velocity value will be described. The shooting direction acquisition unit 124 acquires the current pan / tilt position of the sub camera 400 via the network I / F 105. The shooting direction acquisition unit 124 calculates a pan angular velocity proportional to the difference between the current pan position and the pan target position. The shooting direction acquisition unit 124 also calculates a tilt angular velocity proportional to the difference between the current tilt position and the tilt target position. The shooting direction acquisition unit 124 writes the calculated pan and tilt control values ​​to the RAM 102.

[0112] The second recognition unit 125 uses an inference model for identifying a person stored in the ROM 103 in the same manner as the first recognition unit 121 to determine whether or not a tracking subject has been detected from the image SUB_IMG captured by the sub camera 400. The second recognition unit 125 transmits the determination result to the tracking subject determination unit 123 as subject information DETECT_INFO.

[0113] 12(A) and 12(B), a process of controlling the shooting direction of the multiple sub-cameras 400 so that they face a shooting target corresponding to the shooting target of the main camera 500 will be described. FIGS. 12(A) and 12(B) are flowcharts illustrating an example of a control process of the shooting direction of the sub-cameras 400. FIGS. 12(A) and 12(B) show an example of a process in which, when the main camera 500 shoots a target A, the sub-cameras 400a and 400b shoot a target C, and, when the main camera 500 shoots a target B, the sub-cameras 400a and 400b shoot a target D. The sub-cameras 400a and 400b are also collectively referred to as the sub-cameras 400.

[0114] In step S1201 of FIG. 12(A), CPU 101 detects a main subject that is an object of shooting by main camera 500. CPU 101 acquires information on the detected main subject as information indicating an object of shooting by main camera 500. Specifically, first recognition unit 121 acquires an image shot by main camera 500 and detects the subject. First recognition unit 121 transmits position information (including coordinate information) and identification information of the detected subject to main subject determination unit 122. Main subject determination unit 122 determines the main subject based on the orientation of main camera 500, or the position where the subject appears in the image shot by main camera 500, etc. The main subject determined by main subject determination unit 122 is the main subject detected in step S1201.

[0115] In step S1202, the CPU 101 detects the main object It is determined whether the detected main subject is subject A. If the detected main subject is subject A, the process proceeds to step S1203. If the detected main subject is not subject A, the process proceeds to step S1302 in FIG. 12B.

[0116] In step S1203, CPU 101 acquires the coordinates of subject C, which is the subject of shooting of sub camera 400 corresponding to subject A, the main subject. Specifically, main subject determination unit 122 transmits identification information and position information of the main subject to tracking subject determination unit 123. Furthermore, role setting unit 120 transmits information on the correspondence relationship between the subject of shooting of main camera 500 and the subject of shooting of sub camera 400 to tracking subject determination unit 123. Tracking subject determination unit 123 determines subject C as the tracking subject. Tracking subject determination unit 123 acquires position information (including coordinate information) of subject C from the image captured by overhead camera 300. Tracking subject determination unit 123 transmits the acquired position information of subject C to shooting direction acquisition unit 124.

[0117] In step S1204, CPU 101 directs sub-cameras 400a, 400b toward subject C. Specifically, imaging direction acquisition unit 124 acquires pan / tilt control values ​​for each of sub-cameras 400a, 400b to control the imaging directions of sub-cameras 400a, 400b so that they face subject C, which is a tracking subject. Imaging direction acquisition unit 124 transmits the pan / tilt control values ​​to sub-cameras 400a, 400b. Sub-cameras 400a, 400b direct their imaging directions toward subject C based on the pan / tilt control values ​​received from imaging direction acquisition unit 124.

[0118] In step S1205, similarly to step S1201, CPU 101 detects a main subject that is an object to be photographed by main camera 500. In step S1206, CPU 101 determines whether or not the main subject detected in step S1205 is subject B.

[0119] If the detected main subject is subject B, it is determined that the shooting direction of main camera 500 has changed from a direction facing subject A to a direction facing subject B, and processing proceeds to step S1207. In this way, whether or not the shooting direction of main camera 500 has changed can be determined based on a change in the subject detected from the image captured by main camera 500. If the main subject detected in step S1205 is not subject B, processing returns to step S1203.

[0120] In step S1207, CPU 101 acquires the coordinates of subject D, which is the subject to be captured by sub camera 400 and corresponds to main subject subject B. CPU 101 can acquire the coordinates of subject D from the image captured by overhead camera 300, similar to step S1203.

[0121] In step S1208, CPU 101 determines whether subject D is closer to sub camera 400a or sub camera 400b (the distance to which is shorter). If subject D is closer to sub camera 400a than to sub camera 400b, the process proceeds to step S1209. If subject D is closer to sub camera 400b than to sub camera 400a, the process proceeds to step S1214.

[0122] In step S1209, CPU 101 acquires information on the movable area in the shooting direction of each of sub cameras 400a and 400b. The movable area in the shooting direction of sub camera 400 indicates the range in which the optical axis of sub camera 400 can be directed.

[0123] In step S1210, CPU 101 determines whether or not a line (optical axis) extending from the center of the lens in a forward direction can be superimposed on subject D by panning sub-camera 400a. In other words, CPU 101 determines whether or not subject D can be photographed so that it is positioned approximately at the center of the image photographed by sub-camera 400a. If the optical axis of sub-camera 400a can be superimposed on subject D, the process proceeds to step S1211. If the axis cannot be superimposed on subject D, the process proceeds to step S1213.

[0124] In step S1211, CPU 101 directs sub camera 400a toward subject D. Specifically, shooting direction acquisition unit 124 acquires a pan / tilt control value for controlling the shooting direction of sub camera 400a to face subject D, which is a tracking target. Shooting direction acquisition unit 124 transmits the acquired pan / tilt control value to sub camera 400a. Sub camera 400a directs the shooting direction toward subject D based on the pan / tilt control value received from shooting direction acquisition unit 124. Second recognition unit 125 acquires an image captured by sub camera 400a, and confirms that subject D has been detected from the acquired captured image. Through the process of step S1211, CPU 101 can confirm that the shooting direction of sub camera 400a has been changed to a direction facing subject D and that subject D can be tracked.

[0125] In step S1212, CPU 101 directs sub camera 400b toward subject D. Specifically, shooting direction acquisition unit 124 acquires a pan / tilt control value for controlling the shooting direction of sub camera 400b to face subject D, which is a tracking target. Shooting direction acquisition unit 124 transmits the acquired pan / tilt control value to sub camera 400b. Sub camera 400b directs the shooting direction toward subject D based on the pan / tilt control value received from shooting direction acquisition unit 124. Second recognition unit 125 acquires an image captured by sub camera 400b, and confirms that subject D has been detected from the acquired captured image. Through the process of step S1212, CPU 101 can confirm that the shooting direction of sub camera 400b has been changed to a direction facing subject D and that subject D can be tracked.

[0126] In step S1213, CPU 101 determines which of sub-camera 400a and sub-camera 400b can capture subject D at a position closer to the center of the screen. For example, when the optical axes of sub-camera 400a and sub-camera 400b are brought closer to subject D within the movable range, CPU 101 can determine that the sub-camera 400 whose optical axis is closer to subject D can capture subject D at a position closer to the center of the screen. If sub-camera 400a can capture subject D at a position closer to the center of the screen, the process proceeds to step S1211. If sub-camera 400b can capture subject D at a position closer to the center of the screen, the process proceeds to step S1214.

[0127] In step S1214, CPU 101 directs sub-camera 400b to subject D, similar to step S1212. In step S1215, CPU 101 directs sub-camera 400a to subject D, similar to step S1211.

[0128] In step S1216, CPU 101 directs sub-cameras 400a, 400b toward subject D. Specifically, shooting direction acquisition unit 124 acquires pan / tilt control values ​​for each of sub-cameras 400a, 400b to control the shooting directions of sub-cameras 400a, 400b to face subject D, which is the tracking subject. Shooting direction acquisition unit 124 transmits the pan / tilt control values ​​to sub-cameras 400a, 400b. Based on the pan / tilt control values ​​received from shooting direction acquisition unit 124, sub-cameras 400a, 400b direct their shooting directions toward subject D.

[0129] In step S1217, CPU 101 detects a main subject that is the subject of shooting by main camera 500, similarly to step S1201. In step S1218, CPU 101 determines whether or not the main subject detected in step S1217 is subject A. If the main subject detected in step S1217 is subject A, it is determined that the shooting direction of main camera 500 has been changed from a direction facing subject B to a direction facing subject A, and processing proceeds to step S1307 in FIG. 12(B). If the detected main subject is not subject A, processing returns to step S1216.

[0130] In step S1302, CPU 101 determines whether or not the main subject detected in step S1201 in Fig. 12A is subject B. If the detected main subject is subject B, processing proceeds to step S1303. If the detected main subject is not subject B, processing proceeds to step S1201 in Fig. 12A.

[0131] In step S1303, CPU 101 acquires the coordinates of subject D, which is the subject of shooting of sub camera 400 corresponding to subject B, the main subject. Specifically, main subject determination unit 122 transmits identification information and position information of the main subject to tracking subject determination unit 123. Furthermore, role setting unit 120 transmits information on the correspondence relationship between the subject of shooting of main camera 500 and the subject of shooting of sub camera 400 to tracking subject determination unit 123. Tracking subject determination unit 123 determines subject D as the tracking subject. Tracking subject determination unit 123 acquires position information (including coordinate information) of subject D from the image captured by overhead camera 300. Tracking subject determination unit 123 transmits the acquired position information of subject D to shooting direction acquisition unit 124.

[0132] In step S1304, CPU 101 directs sub-cameras 400a and 400b toward subject D. Specifically, shooting direction acquisition unit 124 acquires pan / tilt control values ​​for each of sub-cameras 400a and 400b to control the shooting directions of sub-cameras 400a and 400b to face subject D, which is the tracking subject. Shooting direction acquisition unit 124 transmits the pan / tilt control values ​​to sub-cameras 400a and 400b. Sub-cameras 400a and 400b direct their shooting directions toward subject D based on the pan / tilt control values ​​received from shooting direction acquisition unit 124.

[0133] In step S1305, similarly to step S1201, CPU 101 detects a main subject that is the subject being photographed by main camera 500. In step S1306, CPU 101 determines whether or not the main subject detected in step S1305 is subject A.

[0134] If the detected main subject is subject A, it is determined that the shooting direction of main camera 500 has been changed from a direction facing subject B to a direction facing subject A, and the process proceeds to step S1307. If the main subject detected in step S1305 is not subject A, the process returns to step S1303.

[0135] In step S1307, CPU 101 acquires the coordinates of subject C, which is the subject to be captured by sub camera 400 and corresponds to subject A, the main subject. CPU 101 can acquire the coordinates of subject C from the image captured by overhead camera 300, similar to step S1203.

[0136] In step S1308, CPU 101 determines whether subject C is closer to sub camera 400a or sub camera 400b (the distance to which is shorter). If subject C is closer to sub camera 400a than to sub camera 400b, the process proceeds to step S1309. If subject C is closer to sub camera 400b than to sub camera 400a, the process proceeds to step S1314.

[0137] In step S1309, CPU 101 acquires information on the movable area in the shooting direction of each of sub cameras 400a and 400b. The movable area in the shooting direction of sub camera 400 indicates the range in which the optical axis of sub camera 400 can be directed.

[0138] In step S1310, CPU 101 determines whether or not a line (optical axis) extending from the center of the lens in a forward direction can be superimposed on subject C by panning sub-camera 400a. In other words, CPU 101 determines whether or not subject C can be photographed so that it is positioned approximately at the center of the image photographed by sub-camera 400a. If the optical axis of sub-camera 400a can be superimposed on subject C, processing proceeds to step S1311. If the optical axis of sub-camera 400a cannot be superimposed on subject C, processing proceeds to step S1313.

[0139] In step S1311, CPU 101 directs sub camera 400a toward subject C. Specifically, shooting direction acquisition unit 124 acquires a pan / tilt control value for controlling the shooting direction of sub camera 400a to face subject C, which is a tracking target. Shooting direction acquisition unit 124 transmits the acquired pan / tilt control value to sub camera 400a. Sub camera 400a directs the shooting direction toward subject C based on the pan / tilt control value received from shooting direction acquisition unit 124. Second recognition unit 125 acquires an image captured by sub camera 400a, and confirms that subject C has been detected from the acquired captured image. Through the process of step S1311, CPU 101 can confirm that the shooting direction of sub camera 400a has been changed to a direction facing subject D and that subject C can be tracked.

[0140] In step S1312, CPU 101 directs sub camera 400b toward subject C. Specifically, shooting direction acquisition unit 124 acquires a pan / tilt control value for controlling the shooting direction of sub camera 400b to face subject C, which is a tracking target. Shooting direction acquisition unit 124 transmits the acquired pan / tilt control value to sub camera 400b. Sub camera 400b directs the shooting direction toward subject C based on the pan / tilt control value received from shooting direction acquisition unit 124. Second recognition unit 125 acquires an image captured by sub camera 400b, and confirms that subject C has been detected from the acquired captured image. Through the process of step S1312, CPU 101 can confirm that the shooting direction of sub camera 400b has been changed to a direction facing subject C and that subject C can be tracked.

[0141] In step S1313, CPU 101 determines which of sub-camera 400a and sub-camera 400b can capture subject C at a position closer to the center of the screen. For example, when the optical axes of sub-camera 400a and sub-camera 400b are brought closer to subject C within the movable range, CPU 101 can determine that the sub-camera 400 whose optical axis is closer to subject C can capture subject C at a position closer to the center of the screen. If sub-camera 400a can capture subject C at a position closer to the center of the screen, the process proceeds to step S1311. If sub-camera 400b can capture subject C at a position closer to the center of the screen, the process proceeds to step S1314.

[0142] In step S1314, CPU 101 directs sub-camera 400b to subject C, similar to step S1312. In step S1315, CPU 101 directs sub-camera 400a to subject C, similar to step S1311.

[0143] In step S1316, CPU 101 directs sub-cameras 400a, 400b toward subject C. Specifically, imaging direction acquisition unit 124 acquires pan / tilt control values ​​for each of sub-cameras 400a, 400b to control the imaging directions of sub-cameras 400a, 400b so that they face subject C, which is a tracking subject. Imaging direction acquisition unit 124 transmits the pan / tilt control values ​​to sub-cameras 400a, 400b. Sub-cameras 400a, 400b direct their imaging directions toward subject C based on the pan / tilt control values ​​received from imaging direction acquisition unit 124.

[0144] In step S1317, CPU 101 detects a main subject that is the subject of shooting by main camera 500, similarly to step S1201 in Fig. 12(A). In step S1318, CPU 101 determines whether or not the main subject detected in step S1317 is subject B. If the main subject detected in step S1317 is subject B, it is determined that the shooting direction of main camera 500 has changed from a direction facing subject A to a direction facing subject B, and processing proceeds to step S1207 in Fig. 12(A). If the detected main subject is not subject B, processing returns to step S1316.

[0145] In addition, in FIGS. 12(A) and 12(B), an example in which there are two sub-cameras 400 is described. However, the number of sub-cameras 400 may be three or more. In this case, the control device 100 may divide the sub-cameras 400 into a group that changes the shooting direction at the same timing as the sub-camera 400a and a group that changes the shooting direction at the same timing as the sub-camera 400b. The control device 100 may divide the sub-cameras 400 into groups before the processes of steps S1211 and S1214 in Fig. 12(A) and steps S1311 and S1314 in Fig. 12(B).

[0146] The control device 100 can group the sub-cameras 400 based on, for example, the relative position between the sub-camera 400 and the tracking subject, the distance between the sub-camera 400 and the tracking subject, or the movable area of ​​the shooting direction of the sub-camera 400. The control device 100 may also group the sub-cameras 400 so that adjacent sub-cameras 400 are not included in the same group. The control device 100 may also group the sub-cameras 400 based on a user operation. The control device 100 may group the sub-cameras 400 by arbitrarily combining these various conditions.

[0147] 12(A) and 12(B), the process considering the movable area in the shooting direction of the sub-camera 400 may be omitted. That is, the processes of steps S1209, S1210, and S1213 in Fig. 12(A) and the processes of steps S1309, S1310, and S1313 in Fig. 12(B) can be omitted.

[0148] Furthermore, when the shooting direction of the main camera 500 is a direction facing subject A and subject A is no longer detected from the image captured by the main camera 500, the control device 100 controls the shooting direction of the sub-camera 400, which changes its shooting direction first, not to change it from subject C. Furthermore, when the shooting direction of the main camera 500 is changed to a direction facing subject B but subject B is not detected from the image captured by the main camera 500, the control device 100 controls the shooting direction of the sub-camera 400, which changes its shooting direction first, not to change it from subject C.

[0149] In the above embodiment, when the control device 100 controls the shooting directions of the multiple sub-cameras 400 so that they face a shooting target corresponding to the shooting target of the main camera 500, the control device 100 changes the shooting directions of some of the sub-cameras 400 first, and does not change the shooting directions of the other sub-cameras 400. After the shooting directions of some of the sub-cameras 400 are changed, the control device 100 changes the shooting directions of the other sub-cameras 400. This makes it possible to shoot the respective subjects before and after the change at effective angles of view by the multiple cameras when changing the subject in shooting with the multi-cameras. Therefore, the user can obtain the desired image from the images shot at multiple angles of view even in a scene where the subject is changed during editing after shooting and live broadcasting.

[0150] The above-described embodiment is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0151] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0152] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An acquisition means for acquiring information indicating a subject to be photographed by the first imaging device; a control means for controlling the photographing directions of the second imaging devices so that the second imaging devices face a photographing target corresponding to the photographing target of the first imaging device; having When the photographing direction of the first imaging device is changed from a direction facing a first object to a direction facing a second object, the control means a photographing direction of the second imaging device included in a first group among the plurality of second imaging devices is not changed from a direction facing a third subject corresponding to the first subject, The photographing direction of the second imaging devices included in a second group among the plurality of second imaging devices is changed from a direction facing the third subject to a direction facing a fourth subject corresponding to the second subject. A control device comprising: (Configuration 2) The control means changes the imaging direction of the second imaging device included in the second group from a direction facing the third subject to a direction facing the fourth subject after the imaging direction of the second imaging device included in the second group is changed to a direction facing the fourth subject. 2. The control device according to configuration 1, (Configuration 3) Whether or not the photographing direction of the first imaging device has been changed from a direction facing the first subject to a direction facing the second subject is determined based on a change in the subject detected from the photographed image of the first imaging device. 3. The control device according to configuration 1 or 2. (Configuration 4) When the shooting direction of the first imaging device is a direction facing the first subject and the first subject is no longer detected from the image captured by the first imaging device, or when the shooting direction of the first imaging device is changed to a direction facing the second subject but the second subject is not detected from the image captured by the first imaging device, the control means does not change the shooting direction of the second imaging device included in the second group from a direction facing the third subject. 4. The control device according to any one of configurations 1 to 3. (Configuration 5) The first subject and the third subject are the same subject. 5. The control device according to any one of configurations 1 to 4. (Configuration 6) The first object and the fourth object are the same object. 5. The control device according to any one of configurations 1 to 4. (Configuration 7) The second subject and the third subject are the same subject. 7. The control device according to any one of configurations 1 to 6. (Configuration 8) The second subject and the fourth subject are the same subject. 7. The control device according to any one of configurations 1 to 6. (Configuration 9) The third subject is a subject different from the first subject. 5. The control device according to any one of configurations 1 to 4. (Configuration 10) The fourth subject is a subject different from the second subject. 5. The control device according to any one of configurations 1 to 4. (Configuration 11) The imaging device further includes a division unit that divides the second imaging devices into the first group and the second group based on the positions of the second imaging devices. 11. The control device according to any one of configurations 1 to 10. (Configuration 12) The division means divides the second imaging devices into the first group and the second group based on at least one of a relative position of each of the second imaging devices and the fourth object and a relative position of each of the second imaging devices and the third object. 12. The control device according to claim 11, (Configuration 13) The division means divides the second imaging devices into the first group and the second group based on at least one of a distance between a position of each of the second imaging devices and the fourth object and a distance between a position of each of the second imaging devices and the third object. 12. The control device according to claim 11, (Configuration 14) The division means divides the second imaging devices into the first group and the second group such that adjacent second imaging devices are not included in the same group. 12. The control device according to claim 11, (Configuration 15) The division means divides the second imaging devices into the first group and the second group based on the positions of the second imaging devices, the movable areas of the second imaging devices in the shooting directions, and the position of the third subject. 12. The control device according to claim 11, (Configuration 16) The division means divides the second imaging devices into the first group and the second group based on the positions of the second imaging devices, the movable areas of the second imaging devices in the shooting directions, and the position of the fourth subject. 12. The control device according to claim 11, (Configuration 17) The imaging device further includes a division unit that divides the plurality of second imaging devices into the first group and the second group based on a user operation. 11. The control device according to any one of configurations 1 to 10. (method) An acquisition step of acquiring information indicating a subject to be photographed by the first imaging device; a control step of controlling the photographing directions of the plurality of second imaging devices so that the second imaging devices face a photographing target corresponding to the photographing target of the first imaging device; having In the control step, when the photographing direction of the first imaging device is changed from a direction facing a first object to a direction facing a second object, a photographing direction of the second imaging device included in a first group among the plurality of second imaging devices is not changed from a direction facing a third subject corresponding to the first subject, The photographing direction of the second imaging devices included in a second group among the plurality of second imaging devices is changed from a direction facing the third subject to a direction facing a fourth subject corresponding to the second subject. A control method comprising: (program) A program for causing a computer to function as each of the means of the control device according to any one of configurations 1 to 17. [Explanation of symbols]

[0153] 100: control device, 101: CPU

Claims

1. an acquisition means for acquiring information indicating a subject to be photographed by the first imaging device; a control means for controlling the photographing directions of the second image capturing devices so that the second image capturing devices face a photographing target corresponding to the photographing target of the first image capturing device; having When the photographing direction of the first imaging device is changed from a direction facing a first subject to a direction facing a second subject, the control means a photographing direction of the second imaging device included in a first group among the plurality of second imaging devices is not changed from a direction facing a third object corresponding to the first object, The photographing direction of the second imaging devices included in a second group among the plurality of second imaging devices is changed from a direction facing the third subject to a direction facing a fourth subject corresponding to the second subject. A control device comprising:

2. The control means changes the imaging direction of the second imaging device included in the second group from a direction facing the third subject to a direction facing the fourth subject after the imaging direction of the second imaging device included in the second group is changed to a direction facing the fourth subject. The control device according to claim 1 .

3. Whether or not the photographing direction of the first imaging device has been changed from a direction facing the first subject to a direction facing the second subject is determined based on a change in the subject detected from the photographed image of the first imaging device. The control device according to claim 1 .

4. The control means controls the imaging direction of the second imaging device included in the second group not to change the direction toward the third subject when the imaging direction of the first imaging device is a direction toward the first subject and the first subject is no longer detected from the image captured by the first imaging device, or when the imaging direction of the first imaging device is changed to a direction toward the second subject but the second subject is not detected from the image captured by the first imaging device. The control device according to claim 1 .

5. The first subject and the third subject are the same subject. The control device according to claim 1 .

6. The first object and the fourth object are the same object. The control device according to claim 1 .

7. The second subject and the third subject are the same subject. The control device according to claim 1 .

8. The second object and the fourth object are the same object. The control device according to claim 1 .

9. The third subject is a subject different from the first subject. The control device according to claim 1 .

10. The fourth subject is a subject different from the second subject. The control device according to claim 1 .

11. The imaging device further includes a dividing unit that divides the plurality of second imaging devices into the first group and the second group based on the positions of the plurality of second imaging devices. The control device according to claim 1 .

12. The dividing means divides the second imaging devices into the first group and the second group based on at least one of a relative position of each of the second imaging devices to the fourth object and a relative position of each of the second imaging devices to the third object. The control device according to claim 11 .

13. The dividing means divides the second imaging devices into the first group and the second group based on at least one of a distance between a position of each of the second imaging devices and the fourth object and a distance between a position of each of the second imaging devices and the third object. The control device according to claim 11 .

14. The division means divides the second imaging devices into the first group and the second group such that adjacent second imaging devices are not included in the same group. The control device according to claim 11 .

15. The division means divides the second imaging devices into the first group and the second group based on the positions of the second imaging devices, the movable areas of the second imaging devices in the shooting directions, and the position of the third subject. The control device according to claim 11 .

16. The division means divides the second imaging devices into the first group and the second group based on the positions of the second imaging devices, the movable areas of the second imaging devices in the shooting directions, and the position of the fourth subject. The control device according to claim 11 .

17. The imaging device further includes a dividing unit that divides the plurality of second imaging devices into the first group and the second group based on a user operation. The control device according to claim 1 .

18. An acquisition step of acquiring information indicating a subject to be photographed by the first imaging device; a control step of controlling the photographing directions of the plurality of second image capturing devices so that the second image capturing devices face a photographing target corresponding to the photographing target of the first image capturing device; having In the control step, when the photographing direction of the first imaging device is changed from a direction facing a first object to a direction facing a second object, a photographing direction of the second imaging device included in a first group among the plurality of second imaging devices is not changed from a direction facing a third object corresponding to the first object, The photographing direction of the second imaging devices included in a second group among the plurality of second imaging devices is changed from a direction facing the third subject to a direction facing a fourth subject corresponding to the second subject. A control method comprising:

19. A program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 17.

Images