Imaging control device and imaging control method, and imaging system

Abstract

In a shooting control device and shooting control method that enables automatic shooting using multiple cameras in coordination, the objective is to enable automatic response when a camera fails to capture the expected image. [Solution] This is a shooting control device that controls the shooting direction and field of view of a sub-camera, which is one of a group of cameras including a main camera and two or more sub-cameras, based on the role set for the sub-camera and the subject of interest and field of view of the main camera. When the shooting control device determines that the main camera meets predetermined conditions, it selects one or more of the two or more sub-cameras and changes the control content of the selected sub-camera to track and shoot the subject of interest of the main camera with different settings than the main camera.

Description

Technical Field

[0001] The present invention relates to a shooting control device, a shooting control method, and an imaging system, and particularly to a technique for controlling a plurality of imaging devices.

Background Art

[0002] Patent Document 1 describes an imaging system that divides a plurality of cameras into a main camera and a sub-camera, and controls the sub-camera to photograph the same subject as the subject of the main camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An imaging system as described in Patent Document 1 enables automatic shooting by coordinating a plurality of cameras, thus enabling labor saving. On the other hand, when a camera that cannot capture the assumed video occurs due to a failure or the like, or when an accident occurs to the subject, there is a risk that appropriate countermeasures cannot be taken when an unexpected change in the shooting environment occurs.

[0005] In view of the problems of such conventional technologies, in one aspect of the present invention, in a shooting control device and a shooting control method that realize automatic shooting by coordinating a plurality of cameras, an automatic response when an unexpected change in the shooting environment occurs is realized.

Means for Solving the Problems

[0006] In one aspect, the present invention provides a shooting control device that is communicatively connected to a plurality of cameras, including a main camera whose shooting direction and field of view are controlled by an operator, and two or more sub-cameras whose shooting direction and field of view are controlled based on a set role, the control device comprising: a main camera information acquisition means for acquiring information of the main camera, including information of the subject of interest to be photographed by the main camera; a control means for controlling the shooting direction and field of view of the sub-cameras based on a role set for the sub-cameras; and a determination means for determining whether the main camera satisfies predetermined conditions, wherein the control means, when it is determined that the main camera satisfies the conditions, selects one or more of the two or more sub-cameras and sets a role for the selected sub-camera to photograph the subject of interest to be photographed by the main camera in parallel with the shooting by the main camera. [Effects of the Invention]

[0007] According to the present invention, in a shooting control device and shooting control method that realize automatic shooting by linking multiple cameras, it is possible to automatically respond when an unexpected change in the shooting environment occurs. [Brief explanation of the drawing]

[0008] [Figure 1] Schematic diagram of the imaging system according to the embodiment [Figure 2] Block diagram showing an example of the functional configuration of each device in the imaging system according to the embodiment. [Figure 3] Diagram showing an example of a role that can be set for a sub-camera in the embodiment. [Figure 4A] Flowchart relating to the operation of the imaging control device 100 according to the first embodiment [Figure 4B] Flowchart relating to the operation of the imaging control device 100 according to the first embodiment [Figure 4C] Flowchart relating to the operation of the imaging control device 100 according to the first embodiment [Figure 4D] Flowchart relating to the operation of the imaging control device 100 according to the first embodiment [Figure 5A] Flowchart showing the operation of the imaging control device 100 according to the second embodiment [Figure 5B] Flowchart showing the operation of the imaging control device 100 according to the second embodiment [Figure 5C] Flowchart showing the operation of the imaging control device 100 according to the second embodiment [Figure 5D] Flowchart showing the operation of the imaging control device 100 according to the second embodiment [Figure 6] Diagram showing an example of the shooting state using the imaging system according to the embodiment [Figure 7] Diagram showing an example of the role transition of the sub - camera in the second embodiment [Figure 8] Diagram showing the imaging control device 100 according to the embodiment focusing on the main operations and signal flow [Figure 9] Diagram showing an example of the roles and control contents that can be set for the sub - camera in the third embodiment [Figure 10] Flowchart regarding the role determination process in the third embodiment [Figure 11] Flowchart regarding the operations of each device of the imaging system according to the third embodiment [Figure 12] Diagram for explaining coordinate transformation in the embodiment [Figure 13] Diagram regarding subject detection and coordinate transformation in the embodiment [Figure 14] [[ID=3十七]]Schematic diagram of the operation control of the sub - camera in the third embodiment [Figure 15] Another schematic diagram of the operation control of the sub - camera in the third embodiment [Figure 16] Diagram for explaining the calculation of the pan value in the embodiment [[ID=四十五]] [Figure 17] Diagram for explaining the calculation of the tilt value in the embodiment [Figure 18] Diagram showing an example of the mapping of the zoom values between the main camera and the sub - camera in the embodiment [Figure 19] Flowchart regarding the determination process of the control contents according to the role of the sub - camera in the third embodiment [Figure 20] Schematic diagram of control according to the role of the sub-camera in the third embodiment [Figure 21] Schematic diagram of control according to the role of the sub-camera in the third embodiment [Figure 22] Schematic diagram of the tracking operation of multiple subjects in a modified example of the third embodiment

Embodiments for Carrying Out the Invention

[0009] Hereinafter, the present invention will be described in detail based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are essential for the invention, and multiple features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same reference numerals are assigned to the same or similar configurations, and redundant explanations are omitted.

[0010] ●<First Embodiment> (Overview of the multi-camera imaging system) FIG. 1 is a schematic diagram showing a configuration example of a multi-camera imaging system 10 (hereinafter simply referred to as an imaging system) according to the present embodiment. The imaging system 10 includes a plurality of cameras 300, 500, 600, 700, 800, 900, a photographing control device 100, and a role control device 400. The plurality of cameras 300, 500, 600, 700, 800, 900, the photographing control device 100, and the role control device 400 are communicably connected through a communication network 200.

[0011] The communication network 200 complies with known wired or wireless communication standards such as the IEEE802.3 series and the 1EEE802.11 series. In addition, each of the plurality of cameras 300, 500, 600, 700, 800, 900, the photographing control device 100, and the role control device 400 has a communication interface compliant with the standard of the communication network 200.

[0012] Camera 300 captures the entire predetermined shooting range. The shooting range is set to the area where the subject being filmed may be located, for example, in a studio. Therefore, the image from camera 300 captures all subjects within the shooting range.

[0013] The purpose of camera 300 is to capture images for detecting subjects within the shooting range. Therefore, the shooting direction and field of view of camera 300 are determined according to the position of camera 300 and the shooting range, and are basically fixed during shooting. Furthermore, in order to obtain a pan-focus image of the shooting range, the focusing distance may also be basically fixed. In addition, it is preferable that camera 300 captures the entire shooting range without it being obscured by objects outside the shooting range. For this reason, camera 300 is positioned to overlook the entire shooting range. To distinguish it from other cameras 400 and 500, whose shooting direction and field of view are not basically fixed during shooting, camera 300 will be referred to as the overhead camera below. However, the installation position of camera 300 is not limited to a position that overlooks the shooting range. The operation of the overhead camera 300 can be controlled from the shooting control device 100.

[0014] Cameras 500, 600, 700, 800, and 900 (hereinafter referred to as cameras 500-900) are, for example, PTZ cameras, and their operation, including shooting direction (pan and tilt angles) and field of view (zoom), can be controlled from an external device. The overhead camera 300 may also be a PTZ camera. Here, the operation of camera 500 is controlled by the user of the imaging system, and the operation of the remaining cameras 600-900 is controlled by the shooting control device 100. In the following, since the shooting control device 100 controls the operation of cameras 600-900 based on the state of camera 500, camera 500 will be referred to as the main camera, and cameras 600-900 as sub-cameras.

[0015] In the second embodiment described later, the main camera 500 is not required. If the main camera 500 is not present, the shooting direction and field of view of the sub-cameras 600 to 900 are controlled by the shooting control device 100 according to control content that is independent of the main camera and is associated with a role set by the role control device 400.

[0016] The number of sub-cameras is arbitrary and may be more or less than four, but there must be at least two. The main camera 500 may be operated directly by the user. Cameras 500-900 may also be configured to allow control of the shooting direction (pan and tilt angles) by mounting the camera body on a tripod head. Furthermore, cameras 500-900 may also be configured with interchangeable zoom lenses attached to the camera body.

[0017] In this embodiment, it is assumed that the role control device 400 has an operator. The shooting control device 100 may also have an operator (user), but this is not mandatory. The operator of the role control device 400 may also be the user of the shooting control device 100. Since the shooting control device 100 controls the shooting of the overhead camera 300 and the sub-camera 400, a photographer is not required. The main camera 500 is assumed to have an operator or photographer. In this way, labor savings can be achieved by configuring several devices without requiring operators or photographers.

[0018] Although Figure 1 shows all signals being communicated via the communication network 200, video signals and control signals may be communicated in different ways. For example, each of the multiple cameras 300, 500-900 may directly supply its video signal to the shooting control device 100 via a cable. Cameras 300, 500-900 and the shooting control device 100 have communication circuits corresponding to the video signal standard. Video signal standards include, but are not limited to, the SDI (Serial Digital Interface) standard and HDMI (High-Definition Multimedia Interface) (registered trademark).

[0019] The shooting control device 100 detects a subject from the video signal received from the overhead camera 300. Based on the subject detection result, the status of the main camera 500, and the role set for the sub-camera, the shooting direction and field of view of the sub-camera are determined. The shooting control device 100 transmits control commands including the determined shooting direction and field of view to the sub-cameras 600 to 900. By changing the role settings, the method of determining the shooting direction and field of view of the sub-cameras 600 to 900 can be made different, thereby increasing the degree of freedom in controlling the operation of the sub-cameras 600 to 900.

[0020] The role control device 400 sets roles and priorities for each of the sub-cameras 600 to 900. As will be described later, each role is associated with different control content and has a name that represents that control content. The priority for each of the sub-cameras 600 to 900 is pre-set considering the shooting scene, the role assigned to the sub-camera, and the shooting position, but the user can change it at any time. The priority can also be changed automatically depending on the shooting conditions.

[0021] Furthermore, when using the imaging system 10 for purposes such as live streaming, a switcher is connected to the network 200 to select and output one of the video feeds captured by the main camera 500 and sub-cameras 600-900.

[0022] The switcher is a selection device that receives video signals from both the main camera 500 and the sub-cameras 600-900, and selects the video signal from one sub-camera specified by the user to output as the main video signal.

[0023] The switcher also transmits control signals (tally information) via the network 200 to display tally messages on the camera capturing the main video feed. The shooting control device 100 can obtain the tally information via the network 200 and know which of the main camera 500 or sub-cameras 600-900 is selected as the main video feed.

[0024] Tally information, for example, associates each of the main camera 500 and sub-cameras 600-900 with one of the following statuses: "Streaming," "Previewing," or "Standby." "Streaming" and "Previewing" are each associated with one camera, while the remaining cameras are associated with "Standby." "Streaming" is associated with the camera currently capturing the main video feed. "Previewing" is associated with the camera currently capturing the video feed that will be selected as the main video feed next. Note that tally information only needs to identify which camera is associated with "Streaming."

[0025] The CPU (described below) of a camera associated with "Streaming" in its tally information can perform tally displays, such as illuminating a red tally lamp. However, cameras associated with "Streaming" in their tally information will not illuminate their tally lamps. Additionally, the CPU of a camera associated with "Previewing" in its tally information can perform tally displays, such as illuminating a tally lamp of a different color (e.g., green) than that of "Streaming".

[0026] (Examples of functional configurations for each device) Figure 2 is a block diagram showing an example of the functional configuration of each component constituting the multi-camera imaging system shown in Figure 1. The configurations represented as functional blocks in the diagram can be implemented using integrated circuits such as ASICs and FPGAs, discrete circuits, or a combination of memory and a processor that executes the program stored in memory. Furthermore, one functional block may be implemented using multiple integrated circuit packages, or multiple functional blocks may be implemented using a single integrated circuit package. Additionally, the same functional block may be implemented in different configurations depending on the operating environment and required capabilities.

[0027] (Shooting control device 100) First, an example of the functional configuration of the imaging control device 100 will be described. The imaging control device 100 may be a general-purpose computer device such as a personal computer or a workstation. The imaging control device 100 has a configuration in which a CPU 101, RAM 102, ROM 103, inference unit 104, network interface (I / F) 105, user input unit 106, and display unit 108 are interconnected via an internal bus 110.

[0028] The CPU 101 is a microprocessor capable of executing programmed instructions. For example, the CPU 101 realizes the functions of the imaging control device 100, which will be described later, by reading a program stored in the ROM 103 into the RAM 102 and executing it. The CPU 101 can also realize the functions of the imaging control device 100 by executing an imaging control application that runs on the operating system (OS), for example.

[0029] RAM 102 is used to load programs executed by CPU 101, and to temporarily store data processed by CPU 101, data being processed, etc. A portion of RAM 102 may also be used as video memory for the display unit 108.

[0030] ROM103 is a rewritable, non-volatile memory that stores programs executed by CPU101 (OS and applications), user data, and other similar information.

[0031] The inference unit 104 performs object region detection processing using a machine learning model on the video feed from the overhead camera 300. The inference unit 104 can be implemented using hardware circuits capable of high-speed execution of machine learning model calculations, such as a GPU (Graphics Processing Unit) or an NPU (Neural Network Processing Unit). Alternatively, the inference unit 104 may be implemented using reconfigurable logic circuits such as an FPGA (Field-Programmable Gate Array). The CPU 101 may execute a program to implement the functions of the inference unit 104.

[0032] The machine learning model may be a convolutional neural network (CNN) trained according to the type of subject to be detected. Here, the inference unit 104 detects human body regions or human face regions as subject regions from the input image. The inference unit 104 also outputs the position and size of the rectangular region inscribed in the subject region, as well as the detection confidence level, for each detected subject region. Multiple machine learning models may be used to perform detection processing of different types of subject regions on the same input image. The inference unit 104 may also perform subject region detection processing using known methods that do not use machine learning models. For example, the inference unit 104 can detect subject regions using methods that utilize local features such as SIFT or SURF, or methods that utilize pattern matching.

[0033] The network interface 105 is an interface for connecting the image capture control device 100 to the communication network 200. The image capture control device 100 (CPU 101) can communicate with external devices on the communication network 200, such as the overhead camera 300, the role control device 400, the main camera 500, and the sub-camera 600, via the network interface 105. The image capture control device 100 may also communicate with external devices via other communication interfaces (not shown, such as USB or Bluetooth®).

[0034] The CPU 101 communicates with each device on the communication network 200 (overhead camera 300, sub-camera 600, main camera 500, role control device 400) by acquiring the network address of each device at any time and storing it in RAM 102. The CPU 101 also acquires information about each device (device type, model name, etc.) at any time (for example, during the initial communication) and stores it in RAM 102. Thus, the CPU 101 is assumed to know at least the identification information and device type of the overhead camera 300, sub-camera 600, main camera 500, and role control device 400. The user may be allowed to assign arbitrary names to each device.

[0035] The user input unit 106 is an input device (not shown) such as a mouse, keyboard, or touch panel. The shooting control device 100 receives user instructions through the user input unit 106.

[0036] The display unit 108 is a display device such as a liquid crystal display (LCD). The display unit 108 displays a GUI screen provided by the OS or shooting control application.

[0037] (Overhead camera 300) Next, we will explain an example of the functional configuration of the overhead camera 300. The CPU 301 is a microprocessor capable of executing programmed instructions. For example, the CPU 301 controls the operation of each functional block by loading a program stored in the ROM 303 into the RAM 302 and executing it, thereby realizing the functions of the overhead camera 300 described later.

[0038] RAM302 is used to load programs executed by CPU301, and to temporarily store data processed by CPU301, data being processed, etc. RAM302 may also be used as a buffer for video signals obtained during capture.

[0039] ROM303 is a rewritable, non-volatile memory. ROM303 stores programs executed by the CPU301, settings for the overhead camera 300, user data, and other similar information. ROM303 can also be used as a recording destination for video signals. ROM303 may include both internal memory and a removable memory card.

[0040] The image sensor 307 comprises an imaging optical system and an image sensor. The image sensor may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The image sensor has a pixel array in which multiple pixels are arranged in two dimensions, and peripheral circuits for reading signals from each pixel. Each pixel accumulates charge according to the amount of incident light by photoelectric conversion. By reading signals with a voltage corresponding to the amount of charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface is obtained.

[0041] The image processing unit 306 applies predetermined signal processing and image processing to the analog image signal output by the image sensor 307 to generate signals and image data according to the application, and to acquire and / or generate various types of information.

[0042] The processing applied by the image processing unit 306 may include, for example, preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effects processing. Preprocessing may include A / D conversion, signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is performed when a color filter is provided on the image sensor 307, and it is a process that interpolates the values ​​of color components that are not included in the individual pixel data that make up the image data. Color interpolation is also called demosaicing. Correction processing may include white balance adjustment, gradation correction, correction of image degradation caused by optical aberrations in the imaging optical system (image recovery), correction of the effects of vignetting in the imaging optical system, and color correction. Data processing may include processes such as region extraction (trimming), merging, scaling, encoding and decoding, and header information generation (data file generation). The generation of video signals to be output externally and video data to be recorded in ROM303 are also included in data processing. The evaluation value calculation process may include generating signals and evaluation values ​​used for autofocus detection (AF), and generating evaluation values ​​used for automatic exposure control (AE). AF and AE are executed by CPU 301. Special effects processing may include adding blur effects, changing color tones, and relighting. These are merely examples of processes that the image processing unit 306 can apply, and do not limit the processes that the image processing unit 306 can apply. The image processing unit 306 outputs the acquired or generated information and data to the CPU 301, RAM 302, etc., depending on the application.

[0043] Furthermore, the type and settings of processing applied by the image processing unit 306 can be controlled by sending commands from the shooting control device 100 to the overhead camera 300.

[0044] The network interface 305 is an interface for connecting the overhead camera 300 to the communication network 200. The overhead camera 300 (CPU 301) can communicate with external devices on the communication network 200, such as the shooting control device 100, sub-camera 600, main camera 500, and role control device 400, via the network interface 305. The overhead camera 300 may also communicate with external devices via other communication interfaces (not shown, such as USB or Bluetooth).

[0045] (Sub-camera 600) Next, we will explain an example of the functional configuration of sub-camera 600. Functional blocks with the same name in sub-camera 600 and overhead camera 300 are assumed to have the same function, and their explanations will be omitted. Also, sub-cameras 700, 800, and 900 are assumed to have the same functional configuration as sub-camera 600, and their individual explanations will be omitted.

[0046] As described above, the sub-camera 600 is a PTZ camera, and its shooting direction and field of view can be controlled externally. Therefore, the sub-camera 600 has a drive unit 609 that can perform pan and tilt movements and zoom movements, and a drive I / F 608. The drive I / F 608 is a communication interface between the drive unit 609 and the CPU 601.

[0047] The drive unit 609 includes a pan / tilt mechanism that supports the sub-camera 600 so that it can be panned and tilted, a zoom mechanism that changes the field of view of the imaging optical system, and motors that drive these mechanisms. The zoom mechanism may use image enlargement and reduction by the image processing unit 606. The drive unit 609 drives the motors according to instructions received from the CPU 601 via the drive I / F 608 to adjust the optical axis direction and field of view of the imaging optical system.

[0048] The recording unit 612 is, for example, a non-volatile memory and is capable of recording image data generated by the image processing unit 606. The recording unit 612 may be built-in or detachable. The user input unit 613 is a general term for input devices used by the user to input instructions and data to the sub-camera 600. The user input unit 613 may include buttons, keys, dials, and touch panels (not shown). The CPU 601 performs operations in response to user input received through the user input unit 613.

[0049] (Main camera 500) Next, an example of the functional configuration of the main camera 500 will be described. Functional blocks with the same name in the main camera 500 and the sub-camera 600 are assumed to have the same function, and their explanation will be omitted. The main camera 500 is operated by the user. Here, it is assumed that the user remotely operates the main camera 500 by sending commands through the communication network 200. However, if the main camera 500 is not a PTZ camera, the user may operate the main camera 500 directly.

[0050] The shooting control device 100 (CPU 101) can acquire information on the shooting direction and field of view of the sub-cameras 600-900 and the main camera 500 from the sub-cameras 600-900 and the main camera 500 via the network I / F 505. The shooting direction may be the pan and tilt angles of the drive units 609 and 509, with a predetermined reference direction set at 0°. The reference direction may be the direction directly facing the shooting range.

[0051] (Role control device 400) Next, we will describe an example of the functional configuration of the role control device 400. The CPU 401 is a microprocessor capable of executing programmed instructions. For example, the CPU 401 controls the operation of each functional block and realizes the functions of the role control device 400 by reading a role setting program stored in the ROM 403 into the RAM 402 and executing it.

[0052] RAM 402 is used to load programs executed by CPU 401, and to temporarily store data processed by CPU 401, data being processed, etc. A portion of RAM 402 may also be used as video memory for the display unit 408.

[0053] ROM403 is a rewritable non-volatile memory that stores programs executed by the CPU401, settings of the role control unit 400, user data, and other similar information.

[0054] The user input unit 411 is an input device such as a button, dial, joystick, or touch panel. The role control device 400 receives user instructions regarding the setting of roles for sub-cameras 600 to 900 through the user input unit 411.

[0055] The network interface 405 is an interface for connecting the role control unit 400 to the communication network 200. The role control unit 400 (CPU 401) can communicate with external devices on the communication network 200, such as the overhead camera 300, main camera 500, sub-cameras 600-900, and shooting control device 100, via the network interface 405. The role control unit 400 may also communicate with external devices via other communication interfaces (not shown, such as USB or Bluetooth).

[0056] The display unit 408 is a display device such as a liquid crystal display (LCD). The display unit 408 displays GUI screens provided by the OS or role setting applications.

[0057] The role control device 400 stores role setting information, for example, in the ROM 403. The role setting information is information that associates the identification information of sub-cameras 600 to 900 with information indicating the role that has been set for them. The CPU 401 displays the role setting screen on the display unit 408 by executing the role setting application. The role setting screen displays, for example, the identification information of sub-cameras 600 to 900 (network address, user-set name, etc.) and the name of the currently set role, in association with each other. The initial value of the currently set role may be a pre-set default role. The user can change the current role displayed associated with the desired sub-camera 600 to 900 by operating the user input unit 411.

[0058] When the CPU 401 detects a user action indicating the completion of a setting operation, such as pressing the OK button on the role setting screen, it updates the role setting information stored in the ROM 403 according to the contents of the role setting screen.

[0059] When CPU401 receives a role acquisition command via network I / F405, it reads the role setting information stored in ROM403 and sends it to the source of the role acquisition command.

[0060] Although Figures 1 and 2 depict the role control device 400 as an independent device, for example, a shooting control application executed by the shooting control device 100 may provide the same functionality as the role control device 400. Alternatively, roles may be directly assigned to the sub-cameras 600-900, and the shooting control device 100 may acquire the roles assigned to the sub-cameras 600-900 from the sub-cameras 600-900.

[0061] The roles that can be assigned to sub-cameras 600-900 pre-determine how the information obtained from the main camera 500 will be used to control the operation of sub-cameras 600-900. Here, as an example, the information from the main camera will be used to control the tracking subject and zoom operation of sub-cameras 600-900.

[0062] Figure 3 shows examples of the types of roles that can be set for sub-cameras 600-900 and the control content associated with those roles. The control content for each role can be stored in the ROM 403 of the role control device 400 and the ROM 103 of the shooting control device 100 in a table format, for example, as shown in Figure 3. Here, it is assumed that one of the following roles can be set: "Main Follow," "Main Counter," "Assist Follow," or "Assist Counter." Roles can be set for each sub-camera.

[0063] For sub-cameras with the role "main follow," the shooting control device 100 (CPU 101) sets the same tracking subject as the main camera 500, and when the main camera 500 is zoomed, it also performs in-phase zoom control on the sub-camera. Here, in-phase means that the direction of zoom (telephoto or wide-angle) is the same, i.e., the direction of the angle of view change is the same. On the other hand, opposite phase means that the direction of zoom (telephoto or wide-angle) is in the opposite direction, i.e., the direction of the angle of view change is in the opposite direction. Note that even if the zoom direction is in-phase, the angle of view does not have to be the same as that of the main camera 500, and the degree of zoom change (such as the speed of change or rate of change) does not have to be the same as that of the main camera 500, regardless of whether it is in-phase or opposite phase.

[0064] For subcameras with the role of "main counter," the shooting control device 100 (CPU 101) sets the same tracking subject as the main camera 500, and when the main camera 500 is zoomed, it performs a zoom control in the opposite phase on the subcamera. Therefore, when the main camera 500 is zoomed in, the shooting control device 100 (CPU 101) controls the subcamera with this role to zoom out. Note that zooming in means changing the zoom in the telephoto direction (telephoto end direction), and zooming out means changing the zoom in the wide-angle direction (wide end direction). When zoom control is performed by the image processing unit 606, zooming in means making the area to be cut out from the image smaller and increasing the magnification of the cut-out area compared to before the area change. On the other hand, zooming down means making the area to be cut out from the image larger and decreasing the magnification of the cut-out area compared to before the area change.

[0065] For sub-cameras with the roles "Assist Follow L" and "Assist Follow R," the shooting control device 100 (CPU 101) sets a different tracking subject from the main camera 500. Furthermore, when the main camera 500 is zoomed, the shooting control device 100 also applies the same phase zoom control to the sub-cameras.

[0066] For sub-cameras with the role of "Assist Counter L" and "Assist Counter R", the shooting control device 100 (CPU 101) sets a different tracking subject from the main camera 500. In addition, when the main camera 500 is zoomed, the shooting control device 100 performs a zoom control in the opposite phase for the sub-cameras.

[0067] Here, for sub-cameras with the role set to "Assist Follow L" and "Assist Counter L," the subject located on the left (leftmost) side of the image, separate from the subject of focus of the main camera 500, is set as the subject to be tracked by the sub-camera. Similarly, for sub-cameras with the role set to "Assist Follow R" and "Assist Counter R," the subject located on the right (rightmost) side of the image, separate from the subject of focus of the main camera 500, is set as the subject to be tracked by the sub-camera.

[0068] Note that the subject tracked by the sub-camera may be set according to different conditions. For example, among the subjects in the image other than the subject of focus of the main camera 500, the subject located at the top (upper edge) or bottom (lower edge) may be set as the subject tracked by the sub-camera. Alternatively, among the subjects other than the subject of focus of the main camera 500, the subject located furthest in the foreground or background may be set as the subject tracked by the sub-camera.

[0069] For sub-cameras whose roles are "exposure shift" and "aperture counter," the shooting control device 100 (CPU 101) sets the same tracking subject as the main camera 500. Furthermore, when the main camera 500 is zoomed, the shooting control device 100 applies a corresponding zoom control to the sub-cameras. In addition, the shooting control device 100 determines the parameters that define the shooting conditions based on the shooting state of the main camera 500. Details will be described later.

[0070] For the sub-camera whose role is "wide-angle," the shooting control device 100 (CPU 101) sets the same tracking subject as the main camera 500. Furthermore, when the main camera 500 is zoomed, the shooting control device 100 applies a corresponding zoom control to the sub-camera. In addition, the shooting control device 100 determines the parameters that define the sub-camera's field of view and shooting conditions based on the shooting state of the main camera 500. Details will be described later.

[0071] Here, we have shown setting the tracked subject and zoom control as control elements associated with a role, but you may also perform only one of them, or add other control items.

[0072] The role setting information stored in the ROM 403 by the role control device 400 includes information indicating the role (such as the name of the type or the number assigned to the type) associated with the identification information of the sub-cameras 600 to 900. The CPU 101 of the shooting control device 100 obtains the role setting information from the role control device 400 and executes operation control of the sub-cameras 600 to 900 according to the type of role set for each sub-camera.

[0073] Furthermore, the role control device 400 may notify an external device (e.g., the shooting control device 100) if there is a change in the role settings for the sub-cameras 600 to 900. This allows the change in role settings to be immediately reflected in the operation control of the sub-cameras 600 to 900.

[0074] Figure 6 is a schematic diagram showing an example of the shooting scene of the imaging system 10, as well as the arrangement, shooting direction, and field of view of each camera. The orientation of each camera indicates the shooting direction in the horizontal plane, and the dashed line schematically shows the shooting field of view. The numbers 1 to 4 attached to sub-cameras 600 to 900 represent the set priority, with smaller numbers indicating higher priority.

[0075] The shooting scene contains three human subjects (Subject A to Subject C). Main camera 500 is only shooting subject B out of subjects A to C. Therefore, the subject of focus for main camera 500 is subject B.

[0076] The sub-camera 600 is assigned the role of "main counter". Therefore, the shooting control device 100 controls the shooting direction of the sub-camera 600 to track the subject of interest (subject B) of the main camera 500. In addition, the shooting control device 100 changes the field of view of the sub-camera 600 to the wide-angle side when the field of view of the main camera 500 changes to the telephoto side, and changes the field of view of the sub-camera 600 to the telephoto side when the field of view of the main camera 500 changes to the wide-angle side.

[0077] The sub-camera 700 is assigned the role "Assist Follow R". Therefore, the shooting control device 100 controls the shooting direction of the sub-camera 600 to track subject C, which is located to the right of the camera, among subjects A and C, which are different from the subject of interest (subject B) of the main camera 500. In addition, the shooting control device 100 changes the field of view of the sub-camera 600 to the telephoto side when the field of view of the main camera 500 changes to the telephoto side, and changes the field of view of the sub-camera 600 to the wide-angle side when the field of view of the main camera 500 changes to the wide-angle side.

[0078] The sub-camera 800 is set to the role "Assist Follow L". Therefore, the shooting control device 100 controls the shooting direction of the sub-camera 600 to track subject A, which is located to the left of the camera's perspective, among subjects A and C, which are different from the main camera 500's subject of interest (subject B). In addition, the shooting control device 100 changes the field of view of the sub-camera 600 to the telephoto side when the field of view of the main camera 500 changes to the telephoto side, and changes the field of view of the sub-camera 600 to the wide-angle side when the field of view of the main camera 500 changes to the wide-angle side.

[0079] The sub-camera 900 is assigned the role "Assist Counter R". Therefore, the shooting control device 100 controls the shooting direction of the sub-camera 600 to track subject C, which is located to the right of the camera, among subjects A and C, which are different from the subject of interest (subject B) of the main camera 500. In addition, the shooting control device 100 changes the field of view of the sub-camera 600 to the wide-angle side when the field of view of the main camera 500 changes to the telephoto side, and changes the field of view of the sub-camera 600 to the telephoto side when the field of view of the main camera 500 changes to the wide-angle side.

[0080] In this embodiment, the shooting control device 100 can control the sub-camera to substitute for the main camera 500. Therefore, even if the intended image is not obtained with the main camera 500, or if it is highly likely that the intended image will not be obtainable in the near future, it is possible to continuously obtain an image similar to or identical to the image that the main camera 500 should be capturing.

[0081] The operation of the imaging control device 100 according to this embodiment will be explained using the flowcharts shown in Figures 4A to 4D. The operation of the imaging control device 100 described below is realized by the CPU 101 executing a program stored in the ROM 103 or acquired through the network I / F 105, and controlling each part of the imaging control device 100.

[0082] Figure 4A is a flowchart showing the overall operation of the imaging control device 100 according to this embodiment. In S4101, the CPU 101 determines whether or not shooting has finished. If it determines that shooting has finished, it does not perform the following actions; otherwise, it executes S4102. For example, the CPU 101 determines that shooting has finished when it receives a notification from the main camera 500 via the network interface 105. The CPU 101 also determines that shooting has finished if it determines that the power to the main camera 500 is off. For example, the CPU 101 determines that the power to the main camera 500 is off if it does not receive a notification from the main camera 500 via the network interface 105 for a certain period of time or longer.

[0083] In S4102, CPU 101 determines (selects) which sub-cameras from sub-cameras 600 to 900 will be subordinate to main camera 500. Details of S4102 will be described later using Figure 4B. A sub-camera subordinate to main camera 500 is a sub-camera that will operate as a backup for main camera 500.

[0084] In S4103, the CPU 101 determines the appropriate role for the subcameras that are subordinate to the main camera 500 in order to back up the main camera 500. Details of S4103 will be described later using Figure 4C.

[0085] In S4104, CPU101 performs control over sub-cameras 600-900 based on their respective roles. After executing S4104, CPU101 executes S4101 again.

[0086] (Determining which sub-cameras should be subordinate to the main camera) Next, we will explain the details of the operation at S4102 in Figure 4A using the flowchart shown in Figure 4B. In S4201, the CPU 101 determines whether the conditions for temporarily releasing the dependency to the main camera 500 are met. The conditions are predetermined and stored in the ROM 103, such as receiving an operation notification from the main camera 500 to the user input unit 512, or receiving a release command from the role control device 400. If the CPU 101 determines that the conditions for temporarily releasing the dependency to the main camera 500 are met, it executes S4202; otherwise, it executes S4203.

[0087] In S4202, CPU101 turns on a temporary release flag set at a specific address in RAM102 (for example, by setting its value to 1). After that, CPU101 executes S4203.

[0088] In S4203, CPU101 refers to the temporary release flag in RAM102 to determine whether the dependency to the main camera is temporarily released or not. If the temporary release flag is ON, CPU101 determines that the dependency is temporarily released and executes S4204; if the temporary release flag is OFF, it determines that the dependency is not temporarily released and executes S4206.

[0089] In S4204, the CPU 101 determines whether the shooting state of the main camera 500 has changed. If it determines that it has changed, it executes S4205; otherwise, it terminates the process shown in the flowchart in Figure 4B. For example, the CPU 101 determines that the shooting state of the main camera 500 has changed when it receives notification from the main camera 500 of changes in settings, shooting direction (direction of the optical axis of the shooting optical system), vibration information, or other camera data that is not linked to the settings. The determination conditions used in S4204 are assumed to be pre-stored in the ROM 103.

[0090] In S4205, CPU101 turns OFF the temporary release flag in RAM102 (for example, setting its value to 0). After that, CPU101 executes S4206.

[0091] In S4206, the CPU 101 acquires the shooting status of the main camera 500 and stores it in the RAM 102. The shooting status includes the video (frame image) captured by the main camera 500, the results of subject detection processing on the video, the shooting direction (e.g., the pan and tilt angles of the drive unit 509), the angle of view, exposure conditions, white balance coefficient, and evaluation values ​​for AE and AF. The shooting conditions may be acquired from the main camera 500, or at least some of the information may be acquired by the shooting control device 100 based on the video received from the main camera 500.

[0092] In S4207, CPU101 determines whether or not it is necessary to determine which subcameras should be subordinate to the main camera 500. If it is determined that it is necessary, it executes S4209; otherwise, it executes S4208.

[0093] For example, CPU101 is • Brightness information detected from the video of the main camera 500 • The size and position of the subject area (or the rectangular area surrounding the subject area) detected from the video of the main camera 500. • Exposure conditions for the main camera 500 (sensitivity, ND filter value, shutter speed, one or more aperture values), focusing distance, focal length of the shooting optical system, • Main camera 500 movement A determination can be made based on one or more of the following conditions.

[0094] More specific examples include: (1) If the size of the area in the video (frame) that is overexposed or close to overexposed (for example, the number of pixels whose brightness value is above the upper threshold) remains above the threshold for a predetermined period of time, (2) If the size of the area in the video (frame) that is blacked out or close to blacked out (for example, the number of pixels whose brightness value is below the lower threshold) remains above the threshold for a predetermined period of time, (3) If the movement speed of the subject (area) of focus of the main camera 500 exceeds the threshold, (4) If, based on the position, direction of movement, and speed of the subject (area) of focus captured by the main camera 500, it is determined that there is a high probability that the subject of focus will move out of frame, (5) If the size of the subject of interest in the main camera 500 exceeds the upper threshold or falls below the lower threshold, or if it is determined from the change in size over time that there is a high probability that it will exceed the upper threshold or fall below the lower threshold, (6) When the depth of field of the subject in focus with the main camera 500 is below the threshold (the depth of field can be calculated based on the distance to the subject in focus (=focus distance) and the aperture value) (7) If the movement of the main camera 500 (e.g., angular velocity and / or acceleration) exceeds a threshold, The CPU 101 can determine that it is necessary to decide which subcamera to assign to the main camera 500 if one or more of the predetermined conditions are met. Note that the conditions exemplified here are not mandatory, and other conditions may be used.

[0095] In (1), the condition of continuing for a predetermined time is included to exclude temporary overexposure, such as flashes used in stage lighting. Note that (1) and (2) may also be expressed as a ratio to the total number of pixels in the frame or as a ratio to the number of pixels in the subject area. If one or more of (3) and (7) are satisfied, there is a high possibility that the subject of interest in the video is blurred significantly. In (4), for example, if the shortest distance from the position of the subject of interest area to any edge of the frame image is less than or equal to a threshold, the position of the subject of interest is too far from the center of the frame image. Also, if the value obtained by dividing the distance to the edge of the frame image in the direction of movement of the subject of interest area by the movement speed of the area is less than or equal to a threshold, there is a high possibility that the subject of interest will go out of frame. Note that an example of how the main camera 500 identifies the subject of interest will be explained in the third embodiment. If (6) is satisfied, the sharpness of the subject of interest tends to be unstable.

[0096] Thus, in S4207, CPU101 is, • If the image quality of the subject being photographed by the main camera 500 is low, or is expected to be low, • When it is not possible, or is anticipated that the subject of interest will not be photographed in the appropriate position and size within the frame. The system determines whether at least one of the following conditions applies, and if it determines that a condition applies, it determines that it is necessary to determine which sub-camera will be subordinate to the main camera 500.

[0097] In S4208, CPU 101 deletes the dependency settings for subcameras 600-900 that are configured to be subordinate to main camera 500. CPU 101 deletes the information about the subordinated subcameras stored, for example, in RAM 102. After that, CPU 101 terminates the processing of the flowchart in Figure 4B.

[0098] In S4209, CPU101 acquires information about each of the sub-cameras 600-900 and stores it in RAM102. The information obtained here is • Information on the photographic optics (focal length range, minimum and maximum aperture values), • Drive unit information (pan and tilt range of motion, current field of view), • The types and priorities of the roles that have been set. • Installation location (for example, 3D coordinates in a coordinate system with a predetermined origin), • Current shooting direction (this may be the pan and tilt angles of the drive unit) • Information about the image sensor (number of pixels, sensor size, type of color filter, etc.) This may include the following. Note that not all of these are mandatory, and other information may be included.

[0099] The information may be the most recently acquired and stored in RAM 102, or it may be acquired from the sub-camera each time. The roles and priorities set for the sub-cameras may be acquired from the role control device 400, or the most recently acquired and stored in RAM 102 may be used. Regarding the installation location of the sub-cameras, the information registered through the user input unit 106 when the sub-cameras were installed can be stored in ROM 103. The position of the sub-cameras measured by an external device using a beacon attached to the sub-camera may also be acquired.

[0100] In S4210, CPU101 determines whether any of the sub-cameras 600-900 is currently capturing main line video. If it determines that one is capturing video, it executes S4211; otherwise, it executes S4213. CPU101 can make this determination based on tally information obtained from the switcher via the network interface 105.

[0101] In S4211, the CPU 101 determines whether the sub-camera currently capturing the main video feed is the same as the sub-camera that has been set as a dependent. If it determines that they are the same, it executes S4213; otherwise, it executes S4212. The CPU 101 can make this determination by referring to the information stored in RAM 102.

[0102] In S4212, the CPU 101 excludes subcameras (subcameras configured as dependents) that are capturing the main line video from the subcameras to be searched. The CPU 101 can associate information indicating that a subcamera should not be included in the search with the information of the relevant subcamera, for example, from the subcamera information stored in RAM 102.

[0103] In S4213, the CPU 101 refers to the sub-camera information stored in RAM 102 and determines whether there is a sub-camera among the searched sub-cameras that has been pre-configured as a candidate to be subordinate to the main camera. If the CPU 101 determines that there is a sub-camera that has been configured as a candidate, it executes S4214; otherwise, it executes S4215. Note that it is optional whether or not to pre-configure candidate sub-cameras. For example, when installing sub-cameras 600 to 900, if it is possible to identify sub-cameras 600 to 900 that are suitable to be subordinate to the main camera 500, the candidate sub-cameras are pre-configured. This configuration can be performed through the user input unit 106.

[0104] In S4214, the CPU 101 assigns a subordinate status to the subcamera that has been set as a candidate. Specifically, the CPU 101 adds information indicating that it has been assigned a subordinate status to the information of the corresponding subcamera stored in the RAM 102. After that, the CPU 101 terminates the operation shown in Figure 4B.

[0105] In S4215, CPU101 determines whether there is a sub-camera capable of taking pictures with a similar composition to the main camera 500. If it determines that there is, it executes S4216; otherwise, it executes S4217.

[0106] Here, a sub-camera capable of shooting with a similar composition to the main camera 500 is: • The main camera 500 can capture an area that includes all the subjects it is photographing, and These subjects can be photographed at a size below a threshold where the difference in size from the image captured by the main camera 500 is below a certain threshold. This is a sub-camera. The CPU 101 can determine which sub-camera is capable of taking pictures with a similar composition to the main camera 500, taking into consideration the camera's installation position, the range of focal length of the imaging optical system, and the range of motion of the drive unit.

[0107] Or, as a simpler condition, • The installation location is closest to the main camera 500, and the distance from the main camera 500 is below a threshold. • The focal length range of the imaging optical system overlaps with the focal length range of the imaging optical system of the main camera 500 by a threshold percentage or more. The sub-camera may be determined as a sub-camera capable of shooting with a composition similar to that of the main camera 500.

[0108] In S4216, the CPU 101 assigns a sub-camera, determined in S4215 to be capable of capturing images with a similar composition to the main camera 500, to a subordinate position. After that, the CPU 101 terminates the process shown in Figure 4B. Note that there may be one or more sub-cameras to be assigned to a subordinate position.

[0109] In S4217, CPU101 searches for the sub-camera with the lowest priority among the sub-cameras being searched.

[0110] In S4218, CPU101 sets the subcamera found in S4217 as dependent. After that, CPU101 terminates the process shown in Figure 4B.

[0111] In addition, in S4216 and S4218, the dependency setting may be performed only after obtaining the user's consent. This is because automatic dependency setting may not be in line with the user's wishes. The CPU 101 will perform the dependency setting if the user consents, and will not perform the dependency setting if consent is not obtained. Furthermore, when obtaining consent, the user may be allowed to change (specify) the sub-camera to be set as a dependency. If the user requests a change, the CPU 101 will set the sub-camera to the one specified by the user as a dependency.

[0112] (Setting the role of the sub-camera, which is subordinate to the main camera) Next, we will explain the details of the operation at S4103 in Figure 4A using the flowchart shown in Figure 4C. In S4301, the CPU 101 refers to the subcamera information stored in RAM 102 to determine whether or not there is a subcamera that is configured as dependent. If the CPU 101 determines that there is a subcamera that is configured as dependent, it executes S4302; otherwise, it terminates the operation shown in Figure 4C.

[0113] In S4302, the CPU 101 determines whether the size of the subject of interest captured by the main camera 500 is less than or equal to a predetermined upper limit. If it is determined to be less than or equal to the upper limit, it executes S4304; otherwise, it executes S4303. Specifically, the CPU 101 determines the area of ​​the subject of interest from the subject area detected in the video captured by the main camera 500. Then, the CPU 101 determines whether the size of the area of ​​the subject of interest (for example, the size of the face area or the rectangular area circumscribing the face area in the case of a person) is less than or equal to the upper limit. The upper limit may be the number of pixels or a percentage of the entire image.

[0114] In S4303, CPU101 sets the role of the subordinate sub-camera to "wide angle" and terminates the operation shown in Figure 4C.

[0115] In S4304, the CPU 101 determines whether the brightness value of the area of ​​the subject of interest in the main camera 500 is within a specified range. If it is determined to be within the specified range, it executes S4306; otherwise, it executes S4305. Specifically, the CPU 101 determines whether the brightness evaluation value of the area of ​​the subject of interest in the main camera 500 is within a predetermined range. The brightness evaluation value may be, for example, the average brightness value or the maximum brightness value. It may also be possible to make this determination for areas of other subjects of the same type as well as the subject of interest. In this case, if there is even one subject area where the brightness evaluation value is determined to be outside the specified range, S4305 is executed.

[0116] In S4305, CPU101 sets the role of the subordinate sub-camera to "exposure shift" and terminates the operation shown in Figure 4C.

[0117] In S4306, the CPU 101 determines whether the depth of field of the area of ​​the subject of interest captured by the main camera 500 is within a specified range. If it is determined to be within the specified range, it executes S4308; otherwise, it executes S4307. Specifically, the CPU 101 calculates the depth of field of the area of ​​the subject of interest captured by the main camera 500 based on the subject distance (focusing distance), the focal length of the imaging optical system, and the aperture value. The CPU 101 then determines whether the depth of field is within a predetermined range. It is also possible to make this determination for areas of other subjects of the same type, not just the subject of interest. In this case, if there is even one subject area where the depth of field is determined to be outside the specified range, S4307 is executed.

[0118] In S4307, CPU101 sets the role of the subordinate sub-camera to "aperture counter" and terminates the operation shown in Figure 4C.

[0119] In S4308, CPU101 sets the role of the subordinate sub-camera to "main follow" and terminates the operation shown in Figure 4C.

[0120] (Control instructions based on the role of the sub-camera) Next, we will explain the details of the operation at S4104 in Figure 4A using the flowchart shown in Figure 4D. In S4401, the CPU 101 determines whether or not there is a sub-camera configured as a dependent. If it determines that there is, it executes S4402; otherwise, it executes S4411. The CPU 101 can make this determination by referring to the sub-camera information stored in RAM 102.

[0121] If there are multiple sub-cameras configured as dependents, CPU 101 executes the processing steps from S4402 onwards for each sub-camera. In S4402, the CPU 101 receives the role information of the subordinate sub-camera from the role control device 400 and stores it in RAM 102. Note that it is not necessary to receive the role information at this time. Role information that has been received in advance and stored in RAM 102 may be referenced.

[0122] In step S4403, the CPU 101 identifies the subject of interest in the main camera 500. If the subject of interest has already been identified in a previous process, that result may be used. In that case, this step does not need to be performed. As described in the third embodiment, the subject of interest can be identified based on the image and shooting direction of the main camera 500.

[0123] In S4404, CPU 101 calculates the amount of change instructions so that the subordinate sub-camera has the same settings and shooting range as the main camera 500. Specifically, CPU 101 obtains setting information from the main camera 500 and calculates the amount of change instructions based on the difference with the setting information of the sub-camera. CPU 101 also calculates the amount of change instructions so that the pan, tilt, and zoom values ​​of the sub-camera are such that subjects included in the shooting range of the main camera 500 are captured by the sub-camera at the same size as the image from the main camera 500. CPU 101 calculates the amount of change instructions for the pan, tilt, and zoom values ​​of the sub-camera based on the position and size of the subject area detected from the images of the main camera 500 and the sub-camera, as well as the position, shooting direction, and field of view of the sub-camera. The position of the subject in the image is adjusted by changing the pan and tilt values, and the size of the subject area is adjusted by changing the zoom value.

[0124] In S4405, CPU101 determines whether the role set in S4103 for the subordinate sub-camera is "exposure shift". If CPU101 determines that the set role is "exposure shift", it executes S4406; otherwise, it executes S4407.

[0125] In S4406, the CPU 101 calculates the exposure shift amount (change in exposure setting) to instruct the sub-camera to use, based on the current exposure setting of the main camera 500 and the brightness information of the subject area. Specifically, if the CPU 101 determines in S4207 that the brightness of the subject of interest in the main camera 500 is high, it calculates a shift amount that reduces the exposure amount relative to the exposure setting of the main camera 500 so that the brightness becomes a predetermined appropriate level. Conversely, if the CPU 101 determines in S4207 that the brightness of the subject of interest in the main camera 500 is low, it calculates a shift amount that increases the exposure amount relative to the exposure setting of the main camera 500 so that the brightness becomes a predetermined appropriate level. The unit of the shift amount can be predetermined, such as 1 / 2 stop or 1 / 3 stop.

[0126] Alternatively, the shift amount can be calculated using the automatic exposure control (AE) function of the sub-camera, or a fixed shift amount can be used. For example, in shooting scenes where the appropriate correction amount can be measured in advance, the shift amount can be determined beforehand. By setting the exposure amount of the dependent sub-camera to be different from that of the main camera 500, it becomes possible to capture properly exposed footage with the dependent sub-camera even if the main camera 500 cannot capture properly exposed footage.

[0127] Furthermore, if there are multiple subordinate sub-cameras, the shift amount may be set differently for each sub-camera. For example, by determining the shift amount for each sub-camera to shoot with either a higher or lower exposure than the main camera 500, it becomes possible to obtain appropriate images whether the image from the main camera 500 is too bright or too dark.

[0128] In S4407, CPU101 determines whether the role set in S4103 for the subordinate sub-camera is "aperture counter". If CPU101 determines that the set role is "aperture counter", it executes S4408; otherwise, it executes S4409.

[0129] In S4408, CPU101 calculates the amount to change the aperture value to instruct the sub-camera to change, based on the aperture value included in the current exposure setting of the main camera 500. Specifically, if the depth of field of the subject of interest is too shallow (narrower than the reference range), CPU101 calculates the amount to change the aperture value so that it is larger than the current aperture value of the main camera 500. Also, if the depth of field of the subject of interest is too deep (wider than the reference range), CPU101 calculates the amount to change the aperture value so that it is smaller than the current aperture value of the main camera 500. CPU101 also calculates the amount to change the shutter speed and / or shooting sensitivity so that the amount of exposure does not change before and after the aperture value change. CPU101 stores the calculated change amounts in RAM102.

[0130] In S4409, CPU101 determines whether the role set in S4103 for the subordinate sub-camera is "wide-angle". If CPU101 determines that the set role is "wide-angle", it executes S4410; otherwise, it executes S4411.

[0131] In S4410, the CPU 101 detects the same subject area from the video of the subordinate sub-camera as the subject area in the video of the main camera 500. The CPU 101 then calculates the zoom amount for the sub-camera so that the size of the detected subject area is smaller than the size in the video of the main camera 500. Specifically, the CPU 101 calculates a zoom amount such that the subject area is captured at a predetermined percentage (e.g., 10%, 20%) smaller than the size of the subject area in the main camera 500. The CPU 101 stores the calculated zoom amount in the RAM 102.

[0132] In S4411, the CPU 101 generates new setting information by reflecting the change instruction amount calculated in S4404 and the values ​​calculated in S4406, S4408, or S4410 onto the current setting information of the dependent sub-camera. The CPU 101 then sends a setting change command containing the new setting information to the dependent sub-camera via the network I / F 105. As shown in the role information in Figure 3, the role set for each dependent sub-camera is to track the subject of interest of the main camera 500. Therefore, the new setting information also reflects the pan and tilt values ​​necessary for changing the shooting direction for automatic tracking. The automatic tracking control of the sub-camera will be described in detail in the third embodiment.

[0133] Furthermore, the shooting direction of the dependent sub-camera may be controlled so that the position of the tracked subject in the image is the same as or close to the position of the subject of interest in the main camera 500's video. Alternatively, the shooting direction of the dependent sub-camera may be controlled so that the position of the tracked subject in the image is shifted by a specific amount from the position of the subject of interest in the main camera 500's video. The specific amount may be a value based on the distance between the position of the area of ​​the subject of interest in the main camera 500's video and the image center. Alternatively, the specific amount may be a value specified in the user input unit 411 of the role control device 400. Basically, the position is shifted horizontally, but the direction of the shift may be specified. In this way, by controlling the position of the tracked subject in the dependent sub-camera, it is possible to intentionally shoot video with the tracked subject shifted to the left or right, or to shoot video in which the tracked subject appears in the same position as in the main camera 500's video. Therefore, it is possible to obtain video that can be used as a substitute video for the main camera 500's video, or as video with a different composition from the main camera 500's video.

[0134] In S4412, the CPU 101 obtains the role information for each sub-camera other than the one that is assigned to a subordinate. The CPU 101 may refer to the role information stored in RAM 102, or it may obtain the role information from the role control device 400.

[0135] In S4413, the CPU 101 detects the position of the tracked subject according to its role from the video feed of one sub-camera other than the sub-camera configured to be subordinate. As will be described in detail in the third embodiment, the CPU 101 detects the subject and its position in the shooting scene from the video feed of the overhead camera 300. Then, the CPU 101 identifies the tracked subject of the target sub-camera from the detected subject, thereby determining the position of the tracked subject.

[0136] In S4414, the CPU 101 calculates the amount of change in pan and tilt values ​​(shooting direction) for the target sub-camera to track the subject according to its role. Furthermore, if the field of view of the main camera 500 changes, the CPU 101 calculates the amount of change in the zoom value according to the role of the target sub-camera. Further details will be explained in the third embodiment.

[0137] In S4415, CPU101 sends a control command to the target sub-camera that includes pan, tilt, and zoom values ​​that reflect the calculated change in the current value.

[0138] In S4416, CPU101 determines whether there are any subcameras that have not been instructed to be changed. If it determines that there are, it executes S4413 again. If it does not determine that there are any subcameras, it terminates the operation shown in Figure 4D.

[0139] According to this embodiment, in a multi-camera imaging system, the sub-cameras that automatically shoot in conjunction with the main camera can have their control methods, such as tracking subject, shooting direction, and field of view, changed according to their assigned roles. This makes it possible to acquire diverse images from the sub-cameras. Furthermore, if it is determined that the quality of the main camera's image is low or that a change in the shooting environment has occurred that may cause it to become low quality, the sub-cameras can be set to a role that backs up the main camera, allowing for continuous shooting of images that maintain a certain quality. This is particularly useful when shooting scenes that cannot be reshot, such as wedding scenes.

[0140] The roles that can be assigned to a sub-camera that backs up the main camera are not limited to the examples described above. Similar to how exposure levels can be varied, other shooting parameters that affect image quality (e.g., white balance) can also be varied. For example, if the main camera's white balance is set to manual, the sub-camera that will be used for backup shooting can be set to use auto white balance. This allows the backup footage from the sub-camera to be used even if color shift occurs in the main camera's image.

[0141] Alternatively, a secondary camera can be used to back up the main camera, recording with the same settings as the main camera. This allows for similar recording with both cameras.

[0142] Furthermore, when automatically determining which sub-camera will back up the main camera, the system now takes into account whether the sub-camera is currently recording video during live streaming and the priority set for the sub-camera. This prevents unintended sub-cameras from being selected as the backup for the main camera.

[0143] Furthermore, if candidates for a sub-camera to back up the main camera are pre-configured, the system will determine which sub-camera to back up the main camera according to the pre-configured settings. Therefore, users who do not wish for automatic selection can specify their preferred sub-cameras in advance.

[0144] Furthermore, by controlling the system to release the sub-camera's dependency setting when the main camera no longer meets certain conditions, it is possible to prevent the sub-camera's role from being changed in situations where dependency is unnecessary. This allows the sub-camera to spend more time shooting in its originally configured role.

[0145] In this embodiment, the shooting control device 100 and the main camera 500 have been described as separate devices, but the functions of the shooting control device 100 can also be incorporated into the main camera 500. In this case, the image from the overhead camera 300 is supplied to the main camera 500. This configuration reduces the amount of equipment required to realize a multi-camera imaging system.

[0146] ●<Second Embodiment> Next, a second embodiment of the present invention will be described. Since this embodiment can be implemented with the imaging system described in the first embodiment, a description of the configuration will be omitted. The roles that can be set for the sub-cameras are also the same as in the first embodiment. However, in this embodiment, the main camera 500 does not need to exist. Therefore, if the main camera 500 does not exist, the tracking subject and field of view control content in the role are determined independently for each sub-camera.

[0147] In a multi-camera imaging system that performs automatic shooting with multiple cameras, there may be cases where a change in the shooting environment occurs that prevents one camera from capturing the intended image due to some factor. In this embodiment, if a camera becomes unable to capture the intended image, the impact of the missing image that the camera should have captured is suppressed.

[0148] The operation of the imaging control device 100 according to this embodiment will be explained using the flowcharts shown in Figures 5A to 5D. The operation of the imaging control device 100 described below is realized by the CPU 101 executing a program stored in the ROM 103 or acquired through the network I / F 105, and controlling each part of the imaging control device 100.

[0149] Figure 5A is a flowchart showing the overall operation of the imaging control device 100 according to this embodiment. In S5101, CPU101 initializes the value of the variable n, which counts the sub-cameras, to 1. In S5102, CPU101 determines whether the variable n exceeds the total number of subcameras max. If it determines that it has exceeded the limit, it executes S5101; otherwise, it executes S5103. In the example shown in Figure 6, there are 4 subcameras, so max = 4.

[0150] In S5103, the CPU 101 obtains the role of subcamera n (1 ≤ n ≤ max) from the role control device 400. Note that it is not necessary to obtain the role for each subcamera; the role information for all subcameras can be obtained from the role control device 400 and stored in RAM 102, and then referred to from RAM 102 thereafter. Subcamera n corresponds to one of subcameras 600 to 900.

[0151] In S5104, CPU101 issues control instructions to subcamera n. Details will be described later using Figure 5B. In S5105, CPU101 determines the operating status of sub-camera n. Details will be described later using Figure 5C.

[0152] In S5106, the CPU 101 reads the operating status of subcamera n, which was determined in S5105, from RAM 102 and determines whether subcamera n is in a missing state or not. A missing state is a state in which the operation associated with the role acquired in S5103 is not being performed. If the CPU 101 determines that subcamera n is in a missing state, it executes S5107; otherwise, it executes S5111.

[0153] In S5107, CPU101 selects subcamera X to compensate for the missing subcamera n. Details will be described later using Figure 5D. Subcamera X corresponds to one of the subcameras 600-900 excluding subcamera n.

[0154] In S5108, CPU101 determines whether or not subcamera X was selected in S5107. If it is determined to be selected, it executes S5109; otherwise, it executes S5111.

[0155] In S5109, the CPU 101 sets the state in which subcamera X complements subcamera n. Specifically, the CPU 101 stores a value in the subcamera state management area located in RAM 102 that indicates that subcamera X is complementing subcamera n.

[0156] In S5110, CPU101 disables the setting if a subcamera X exists that complements subcamera n. Specifically, CPU101 deletes the value in the subcamera state management area in RAM102 that indicates that subcamera X is complementing subcamera n.

[0157] In S5111, the CPU 101 notifies the main camera 500 and the role control device 400 of the completion status via the network interface 105. The completion status here refers to information including the operating status of sub-camera n (whether or not it is in a missing state), and if sub-camera n is in a missing state, information about the sub-camera X to be used for completion. This allows the main camera 500 and the role control device 400 to display or cancel the display to notify the user of the completion status.

[0158] In S5112, CPU101 increments the variable n, which counts the subcameras. Then CPU101 executes S5102.

[0159] (Control instructions to the sub-camera) Next, we will explain the details of the operation at S5104 in Figure 5A using the flowchart shown in Figure 5B. In S5201, CPU101 determines whether subcamera n is set to complement another subcamera Y in S5109. If it is set, it executes S5203; otherwise, it executes S5202. Subcamera Y corresponds to one of the subcameras 600-900, excluding subcamera n.

[0160] In S5202, the CPU 101 calculates the pan, tilt, and zoom values ​​of subcamera n based on the role of subcamera n acquired in S5103. The calculation of the pan, tilt, and zoom values ​​can be performed, for example, by the method described in the third embodiment. After that, the CPU 101 executes S5210.

[0161] In S5203, CPU101 acquires the role of sub-camera Y in the same way as in S5103.

[0162] In S5204, the CPU 101 determines whether the direction of field of view control (follow or counter) matches among the control contents associated with the original role of subcamera n and the role of subcamera Y. If the main camera 500 does not exist, the CPU 101 determines whether the fields of view of subcamera n and subcamera Y are similar. For example, the CPU 101 determines that the fields of view are similar if the difference in fields of view is within a threshold. If the CPU 101 determines that the direction of field of view control matches or that the fields of view are similar, it executes S5205; otherwise, it executes S5209.

[0163] In S5205, the CPU 101 determines whether the amount of adjustment to the field of view necessary to include the subjects of both sub-camera n and sub-camera Y within the shooting range of sub-camera n is below a threshold. The adjustment amount and threshold may be values ​​in units of, for example, the zoom value. The CPU 101 can make this determination based on the position information of the subjects detected from the images of the overhead camera 300 and sub-camera Y, and the current shooting field of view of sub-camera n. If the CPU 101 determines that the required amount of adjustment to the field of view is below the threshold, it executes S5206; otherwise, it executes S5207.

[0164] In S5206, CPU101 calculates the pan, tilt, and zoom values ​​for subcamera n to include both subjects in the shooting range, based on the roles of both subcamera n and subcamera Y. From the images of subcamera n and subcamera Y, CPU101 calculates the center of gravity of the subject being filmed by each. Then, CPU101 calculates the pan and tilt values ​​for subcamera n to track the center of gravity. After that, CPU101 executes S5210.

[0165] In S5207, CPU 101 determines, for example, whether the distance between the subject in sub-camera n and the subject in sub-camera Y in the video from overhead camera 300 is less than a threshold. If it is determined to be less than the threshold, it executes S5208; otherwise, it executes S5209. The distance may be, for example, the distance between the centers of gravity.

[0166] In S5208, CPU101 calculates the pan and tilt values ​​for subcamera n so that it alternately photographs both subjects, based on the roles of both subcamera n and subcamera Y. Then CPU101 executes S5210.

[0167] In S5209, CPU101 calculates the pan, tilt, and zoom values ​​of subcamera n based on the role of subcamera Y acquired in S5203, similar to S5202.

[0168] In S5210, CPU101 sends a control command containing the calculated pan, tilt, and zoom values ​​to subcamera n via network I / F105. This completes the control instruction operation for subcamera n.

[0169] (Determining the operating status of sub-camera n) Next, using the flowchart shown in Figure 5C, we will explain the details of the operation at S5105 in Figure 5A.

[0170] In S5301, the CPU 101 determines whether the sub-camera n autonomously performed an image that differs from the role acquired in S5103. For example, this corresponds to a case where the sub-camera n independently determines the subject to be tracked and photographs a subject of interest different from its role, using a technology such as that disclosed in Japanese Patent Application Publication No. 2020-182075. If the CPU 101 determines that the sub-camera n performed an image that does not fulfill its role, it executes S5306; otherwise, it executes S5302.

[0171] In S5302, CPU101 determines whether subcamera n is being controlled in a way that differs from the control content associated with the role acquired in S5103. For example, the following cases apply. ·As described in the first embodiment, if subcamera n is configured to be dependent, When the shooting control device 100 controls sub-camera n using a subject included in the video captured by the overhead camera 300 or a sub-camera other than sub-camera n as the tracked subject, - When the shooting control device 100 controls the sub-camera n as a sub-camera to compensate for other cameras that are missing.

[0172] These are merely examples, and other cases may be included. The CPU 101 executes S5306 if it determines that the subcamera n is being controlled in a manner different from its intended role, and S5303 otherwise.

[0173] In S5303, CPU101 determines whether or not it has detected an abnormal condition in subcamera n. If it determines that an abnormal condition has been detected, it executes S5306; otherwise, it executes S5304. An abnormal condition refers to a state that interferes with shooting according to its role, such as a malfunction in any part of subcamera n, loss of power, communication interruption, or insufficient recording capacity.

[0174] An abnormal state of subcamera n can be detected by receiving a notification of the abnormality detected by the subcamera n's own self-check function at the shooting control device 100. Alternatively, the CPU 101 may detect an abnormality in subcamera n based on the images captured by the overhead camera 300 and the subcameras (including subcamera n). Examples of abnormalities include cases where video cannot be received from subcamera n, where the subject area cannot be detected from the video of subcamera n, or where subcamera n cannot capture the tracked subject at the correct position. Other examples include cases where communication with subcamera n is not possible, or where notifications that should be received from subcamera n are not received for a certain period of time.

[0175] In S5304, the CPU 101 determines whether the sub-camera n was operated by means other than those controlled by the shooting control device 100. If it is determined that it was operated by means other than those controlled, it executes S5306; otherwise, it executes S5305. For example, this applies when it is detected that the sub-camera n was operated through the user input unit 613.

[0176] In S5305, the CPU 101 determines that subcamera n is not in a missing state. The CPU 101 reflects the determination result in the subcamera information stored in RAM 102 and terminates the operation shown in Figure 5C. In S5306, the CPU 101 determines that subcamera n is in a missing state. The CPU 101 reflects the determination result in the subcamera information stored in RAM 102 and terminates the operation shown in Figure 5C.

[0177] (Select a sub-camera X to complement sub-camera n) Finally, the details of the operation at S5107 in Figure 5A will be explained using the flowchart shown in Figure 5D. In S5401, the CPU 101 retrieves the roles and priorities of all sub-cameras from the role control unit 400 and stores them in RAM 102. If this information is already stored in RAM 102, it does not need to be retrieved again.

[0178] In S5402, CPU101 acquires information about all sub-cameras and stores it in RAM102. The information acquired here is: • Information on the photographic optics (focal length range, minimum and maximum aperture values), • Drive unit information (pan and tilt range of motion, current field of view), • Installation location (for example, 3D coordinates in a coordinate system with a predetermined origin), • Current shooting direction (this may be the pan and tilt angles of the drive unit) • Information about the image sensor (number of pixels, sensor size, type of color filter, etc.) This may include the following. Note that not all of these are mandatory, and other information may be included.

[0179] The information can be taken from the most recently acquired data stored in RAM102, or it can be acquired from the sub-camera each time.

[0180] In S5403, the CPU 101 determines whether there is a subcamera A that is similar in position and capabilities to subcamera n. Subcamera A is selected from subcameras 600 to 900, excluding subcamera n. For example, the CPU 101 can select subcamera A if the distance to the installation location is less than a threshold, and the overlap of the pan / tilt drive range and zoom range is greater than or equal to a predetermined percentage. However, the conditions for subcamera A are not limited to these. If the CPU 101 determines that subcamera A exists, it executes S5404; otherwise, it executes S5405.

[0181] In S5404, CPU101 compares the priority of subcamera n with the priority of subcamera A. Then, if subcamera n has a higher priority (a lower priority value), CPU101 executes S5411; otherwise, it executes S5405.

[0182] In S5405, CPU101 determines whether there are multiple other subcameras with the same role as subcamera n. If it determines that there are, it executes S5406; otherwise, it executes S5408.

[0183] In S5406, CPU101 determines that the subcamera with the lowest priority among several other subcameras that have the same role as subcamera n is subcamera A.

[0184] In S5407, CPU101 compares the priority of subcamera n with the priority of subcamera A, similar to S5404. If subcamera n has a higher priority, CPU101 executes S5411; otherwise, it executes S5408.

[0185] In S5408, CPU101 determines that the sub-camera with the lowest priority among all sub-cameras is sub-camera A.

[0186] In S5409, CPU101 determines whether subcamera n and subcamera A are the same. If they are determined to be the same, it executes S5410; otherwise, it executes S5411.

[0187] In S5410, CPU101 determines that there is no sub-camera corresponding to sub-camera X and terminates the operation shown in Figure 5D. In step S5411, CPU101 selects sub-camera A as sub-camera X and terminates the operation shown in Figure 5D.

[0188] Using Figure 7, a schematic diagram similar to Figure 6, we will explain an example of the transition of the sub-camera's role according to the second embodiment.

[0189] In Figure 7(A), the role control device 400 has set the sub-camera 600 as "Assist Counter L," the sub-cameras 700 and 800 as "Main Follow," and the sub-camera 900 as "Assist Counter R." The shooting control device 100 then performs operation control for each of the sub-cameras 600 to 900 according to the assigned role.

[0190] Now, let's assume that sub-camera 900 becomes unavailable due to some factor. In this case, of the two sub-cameras 700 and 800, which have the same role assigned to them, sub-camera 700, which has a lower priority, is selected as sub-camera X to replace sub-camera 900. As a result, the role of sub-camera 700 switches to "assist counter R," and the control corresponding to that role is performed by the shooting control device 100 (Figure 7B). Sub-cameras 600 and 800 do not experience any change in their roles.

[0191] Let's look at another example. In Figure 7(C), sub-camera 600 is assigned the role of "main counter," sub-camera 700 the role of "assist follow R," sub-camera 800 the role of "assist follow L," and sub-camera 900 the role of "assist counter R." The shooting control device 100 then performs operation control for each of the sub-cameras 600 to 900 according to the role they have been assigned.

[0192] Now, let's assume that sub-camera 900 becomes unavailable due to some factor. In this case, sub-camera 700, which has similar performance and location to sub-camera 900 but a lower priority, is selected as sub-camera X to replace sub-camera 900. As a result, the role of sub-camera 700 switches to "assist counter R," and the control corresponding to that role is performed by the shooting control device 100. Furthermore, since sub-camera 700 is no longer performing its original role and is therefore unavailable, sub-camera 600, which has the lowest priority among the remaining sub-cameras, is selected as sub-camera X to replace sub-camera 700. As a result, the role of sub-camera 600 switches to "assist follow R," and the control corresponding to that role is performed by the shooting control device 100 (Figure 7D).

[0193] According to this embodiment, if a high-priority sub-camera among the multiple sub-cameras included in the multi-camera shooting system becomes unable to perform the intended shooting, another sub-camera will take over and perform the shooting. As a result, it becomes possible to continuously acquire images similar to the images that should be captured by the high-priority sub-camera, enabling highly reliable multi-camera shooting. It is also possible to have yet another sub-camera perform the substitute shooting of the sub-camera used to substitute shooting of another sub-camera.

[0194] Furthermore, instead of simply transferring the role previously assigned to the sub-camera with the missing sub-camera to the sub-camera used for replacement imaging, it is possible to control the operation so that the role previously assigned to the replacement sub-camera is maintained even after replacement imaging begins.

[0195] The failure status of a sub-camera can be notified to the operator of the main camera 500 or the operator of the role control device 400. This allows for prompt action, such as replacing the sub-camera in the failure state.

[0196] In this embodiment, the shooting control device 100 and the main camera 500 have been described as separate devices, but the functions of the shooting control device 100 can also be incorporated into the main camera 500. In this case, the image from the overhead camera 300 is supplied to the main camera 500. This configuration reduces the amount of equipment required to realize a multi-camera imaging system.

[0197] ●<Third Embodiment> Next, a third embodiment of the present invention will be described. Since this embodiment can be implemented with respect to the imaging system described with respect to Figures 1 and 2, a description of the configuration will be omitted. In this embodiment, the roles that can be set for sub-cameras 600 to 900 are the four types shown in Figure 9. Each role and its control content are the same as the roles with the same names in Figure 3. Note that "Assist Follow" in Figure 9 corresponds to "Assist Follow L" in Figure 3, and "Assist Counter" corresponds to "Assist Counter L" in Figure 3.

[0198] <Explanation of the operation of each device> Next, the operation of each device in the imaging system according to this embodiment will be described. Here, the shooting control device 100 automatically controls the shooting operation of the sub-cameras 600 to 900 based on the image from the overhead camera 300, the information obtained from the main camera 500, and the roles set for the sub-cameras 600 to 900.

[0199] Figure 8 is a diagram illustrating the series of processes performed by the shooting control device 100 when controlling the operation of sub-cameras 600 to 900, focusing on the main operations and signal flows. The functional blocks shown within the shooting control device 100 schematically represent the main operations and correspond to the main functions provided by the shooting control application. Each functional block in Figure 8 is realized by a combination of the CPU 101, which executes the shooting control application, and one or more functional blocks of the shooting control device 100 shown in Figure 2.

[0200] Figure 10 is a flowchart showing the operation of the CPU 101 as the role determination unit 120. Figures 11(a) to 11(d) are flowcharts showing the operation of the shooting control device 100, the overhead camera 300, the main camera 500, and the sub-camera 600, respectively. Since the operation of sub-cameras 700, 800, and 900 is the same as that of sub-camera 600, only sub-camera 600 will be described below. Therefore, in the following description, the description of sub-camera 600 also applies to sub-cameras 700, 800, and 900.

[0201] In the following description, it is assumed that the 3D coordinate values ​​of the viewpoint position and the shooting direction (optical axis direction) of the overhead camera 300 are known to the shooting control device 100. Furthermore, it is assumed that known positional information, such as the 3D coordinate values ​​of the viewpoint positions of the sub-camera 600 and the main camera 500, and the coordinate values ​​of markers placed within the shooting range, are pre-stored in the ROM 103 as default positional information REF_POSI. The coordinate system of the position is predetermined according to the type of position.

[0202] (Operation of the role determination unit 120) First, the operation of the CPU 101 as the role determination unit 120 in Figure 8 will be explained with reference to the flowchart shown in Figure 10. The operation described below is achieved by the CPU 101 executing the shooting control application.

[0203] There are no particular restrictions on the timing of initiating the operations shown in the flowchart of Figure 10, but they should be performed at least before starting control of the shooting operation of sub-cameras 600-900. They should also be performed when the role control device 400 receives notification via the network interface 105 that the role settings for sub-cameras 600-900 have been changed.

[0204] In S101, the CPU 101, acting as the role determination unit 120, acquires the roles (role setting information) corresponding to the sub-cameras 600 to 900 from the role control device 400. The CPU 101 can acquire the aforementioned role setting information from the role control device 400 by, for example, sending a role acquisition command to the role control device 400 via the network interface 105. The CPU 101 stores the acquired role setting information in the RAM 102.

[0205] In step S103, the CPU 101 references the role setting information stored in RAM 102 based on the identification information of sub-cameras 600-900 and obtains the operation control content for sub-cameras 600-900. Then, the CPU 101, acting as the role determination unit 120, transmits the acquired operation control content (CAMERA_ROLE) to the tracking subject determination unit 123. In practice, the CPU 101 stores the operation control content in a specific area of ​​RAM 102 and references it when functioning as the tracking subject determination unit 123.

[0206] In S104, the CPU 101, acting as the role determination unit 120, transmits the acquired operation control information (CAMERA_ROLE) to the zoom value calculation unit 125. In practice, the CPU 101 stores the operation control information in a specific area of ​​the RAM 102 and refers to it when functioning as the zoom value calculation unit 125.

[0207] (Operation of the shooting control device 100) Next, the operation of the shooting control device 100 in controlling shooting by the sub-cameras 600 to 900 will be explained with reference to Figures 8 and 11(a). The operation described below corresponds to the operation of the CPU 101 as the recognition unit 121, the focus subject determination unit 122, the tracking subject determination unit 123, the pan / tilt value calculation unit 124, and the zoom value calculation unit 125 in Figure 8. Note that the operation described below is realized by the CPU 101 executing the shooting control application.

[0208] In S201, the CPU 101 sends a shooting command to the overhead camera 300 via the network I / F 105 using a predetermined protocol. In response to this command, the overhead camera 300 begins supplying a video signal (video data) IMG to the video input unit 107. After the CPU 101 begins storing the video signal received by the video input unit 107 into the RAM 102, it executes S202.

[0209] In S202, the CPU 101 acquires information ANGLE indicating the shooting direction from the main camera 500. Specifically, the CPU 101 sends a shooting direction acquisition command to the main camera 500 via the network I / F 105 using a predetermined protocol. In response to the shooting direction acquisition command, the CPU 501 of the main camera 500 sends information ANGLE indicating the current shooting direction of the main camera 500 to the shooting control device 100. Information ANGLE may be, for example, the pan and tilt angles of the drive unit 509. The CPU 101 stores the acquired information ANGLE in the RAM 102.

[0210] In S203, the recognition unit 121 performs the following process. (1) Apply subject area detection processing to the input frame image and store the detection result. (2) For each detected subject area, the position information (image coordinates) is transformed. (3) Apply identification processing to each detected subject area to identify identification information (add information for identification processing in the case of a new subject). (4) Store the identification information ID[n] and location information POSITION[n] associated with each detected subject area.

[0211] The recognition unit 121 is mainly implemented by the CPU 101 and the inference unit 104. The CPU 101 reads one frame of video received from the overhead camera 300 from the RAM 102 and inputs it to the inference unit 104.

[0212] The operation of the recognition unit 121 will be explained step by step below. (1) First, the inference unit 104 inputs the frame image to the machine learning model and detects the subject area. The inference unit 104 stores the position and size of each detected subject area, and the detection confidence level, which are output by the machine learning model as detection results, in the RAM 102. The position and size of the subject area may be any information that can identify the position and size of the rectangular area inscribed in the subject area. Here, the coordinates of the center of the bottom edge of the rectangular area, as well as its width and height, are used as the position and size of the subject area.

[0213] Furthermore, the inference unit 104 stores the detection result for the first frame image in RAM 102 in association with the subject identification information ID[n]. Here, n is the subject number and is an integer that takes values ​​from 1 to the total number of detected subject regions. In addition, the inference unit 104 stores the subject regions detected from the first frame image in RAM 102 in association with the subject identification information ID[n] as a template for identifying individual subjects. If template matching is not used for subject identification, it is not necessary to store the template.

[0214] Figure 13(a) shows an example of the results of subject detection processing by the inference unit 104 on the image from the overhead camera 300 shown in Figure 12(a). Here, the areas of human subjects A to C that exist within the shooting range 20 are detected, and the coordinates of the center of the bottom edge of the rectangular area inscribed with the subject area (foot coordinates) are output as the position.

[0215] Furthermore, for coordinate transformations described later, if markers are placed at known positions within the shooting range 20, as shown in Figure 12(b), the CPU 101 detects the marker images included in the frame image (Figure 12(a)) and stores their positions in the RAM 102. The detection of marker images may also be performed by the inference unit 104. Marker image detection can be performed by any known method, such as pattern matching using a marker template. Marker images may also be detected using a pre-stored machine learning model for marker detection.

[0216] (2) Next, the coordinate transformation performed by the inference unit 104 will be explained. Figure 12(a) schematically shows the image from the overhead camera 300, and Figure 12(b) schematically shows the shooting range 20 as viewed from directly above its center. The inference unit 104 transforms the position of the subject area in the coordinate system of the overhead camera to the coordinate system (plane coordinate system) when the shooting range 20 is viewed from directly above its center.

[0217] The reason for converting the coordinates to a planar coordinate system here is that it is convenient for calculating the pan value (angle of movement in the horizontal plane) required to photograph a specific subject with sub-cameras 600-900. It is assumed here that sub-cameras 600-900 are positioned so that the drive unit 609 performs a panning motion within a horizontal plane parallel to the floor of the shooting range 20.

[0218] Coordinate transformation can be performed in various ways, but here, markers are placed at multiple known positions on the floor within the shooting range 20, and the coordinates are transformed from the overhead camera coordinate system to the planar coordinate system based on the marker positions in the image obtained from the overhead camera 300. Alternatively, the coordinate transformation may be performed without using markers, such as by using the viewpoint position and shooting direction of the overhead camera 300.

[0219] The coordinate transformation can be performed using the homography transformation matrix H, according to Equation 1 below.

number

[0220] The homography transformation matrix can be calculated by substituting the coordinates of the four markers detected from the video and the coordinates (known) of the four markers placed in the shooting range 20 into Equation 1 and solving the system of equations. If the positional relationship between the shooting range 20 and the overhead camera 300 is fixed, the homography transformation matrix H can be calculated in advance during test shooting and saved, for example, in ROM 103.

[0221] The CPU 101 sequentially reads the position of the subject area from the RAM 102 and transforms the coordinates to values ​​in a planar coordinate system. Figure 13(b) schematically shows the state after the foot coordinates (x, y) of each subject area detected in the overhead camera 300 video shown in Figure 13(a) have been transformed to coordinate values ​​(X, Y) in a planar coordinate system using Equation 1 and the homography transformation matrix H stored in ROM 103. The CPU 101 stores the transformed foot coordinates as POSITION[n] in the RAM 102.

[0222] (3) Next, the operation of the inference unit 104 in identifying the subject identification information ID[n] will be described. Here, the subject will be identified using template matching. Subject identification will be performed on the processing results of the second and subsequent subject detections. For the first processing result, a new identification information ID[n] will be assigned to the subject region.

[0223] The inference unit 104 identifies the identification information ID[n] of the detected subject area by template matching using templates stored in RAM 102. This identifies the subject within the shooting range. For example, the inference unit 104 calculates an evaluation value representing the correlation of individual templates for each detected subject area. Then, the inference unit 104 identifies the identification information ID[n] corresponding to the template with the highest correlation that has a certain level of correlation as the identification information ID[n] of the subject area. The evaluation value can be a known value, such as the sum of the absolute differences of pixel values.

[0224] Furthermore, the inference unit 104 assigns a new identification information ID[n] to any subject area that does not have a certain level of correlation with all templates, and adds the image of the subject area to the template.

[0225] Furthermore, the inference unit 104 may update existing templates using subject regions detected in the most recent frame image, or delete templates for which no subject regions with a certain level of correlation have been found for a certain period of time. In addition, the inference unit 104 may store templates corresponding to frequently appearing identification information ID[n] in the ROM 103.

[0226] Furthermore, subjects may be identified by methods other than template matching. For example, at least one of the previously detected position and size may be identified as having the same identification information ID [n] as the nearest subject region. Alternatively, the position in the current frame image may be predicted using a Kalman filter or the like based on the position changes in multiple past detection results associated with the same identification information, and the same identification information ID may be identified as the subject region closest to the predicted position. These methods may also be combined. By not using template matching, the accuracy of identifying different subjects that look similar can be improved.

[0227] (4) The inference unit 104 associates the identified identification information ID[n] with the corresponding position (plane coordinate system) POSITION[n] of the subject area and stores it in the RAM 102.

[0228] Note that, of the processes (1) to (4), the CPU 101 may execute the processes other than subject detection instead of the inference unit 104.

[0229] Here, the image from the overhead camera 300 was used to determine the identification information ID[n] and position[n] of a subject within the shooting range 20. However, images from sub-cameras 600 to 900 may also be used. The CPU 101 performs the operations shown in the flowchart in Figure 11(a) for each sub-camera. The position of the subject area is output as a value in the coordinate system of each sub-camera. Thus, although the overhead camera 300 is not essential, it is considered that using the overhead camera 300 improves the accuracy of subject detection.

[0230] Returning to the explanation of Figure 11(a), in S204, the CPU 101, which acts as the subject of interest determination unit 122 in Figure 8, determines the subject of interest to be tracked by the main camera 500. The CPU 101 can determine the subject of interest of the main camera 500 from among the subjects detected in S203, based on the shooting direction of the main camera 500 acquired in S202. The CPU 101 stores the identification information ID[n] corresponding to the subject area determined to be the subject of interest of the main camera 500 in the RAM 102 as the identification information of the subject of interest MAIN_SUBJECT.

[0231] For example, the CPU 101 can determine the subject closest to the shooting direction of the main camera 500 in a planar coordinate system as the subject of interest for the main camera 500. If there are multiple subjects whose distance from the shooting direction of the main camera 500 is below a threshold, the user may be allowed to select the subject of interest from among them.

[0232] When prompting the user to select a subject of interest, the CPU 101 displays the frame image to which subject detection processing has been applied in S202 on the display unit 108 or an external display device, along with an indicator showing the shooting direction and an indicator showing the subject area that is a candidate for the subject of interest. The subject area indicator may be a rectangular frame indicating the outer edge of the subject area, as shown in Figure 13(a), but other indicators may also be used. The CPU 101 may also display messages on the display unit 108 prompting the user to select a subject of interest in the image.

[0233] The user can select a subject area corresponding to a desired subject of interest by operating the user input unit 106 (input device). There are no particular restrictions on the selection method, but it may be an operation to specify the desired subject area by using a mouse or keyboard.

[0234] When the CPU 101 detects a user operation specifying a subject area, it stores the identification information ID[n] corresponding to the specified subject area in the RAM 102 as the identification information MAIN_SUBJECT of the subject of interest.

[0235] Next, in S205, the CPU 101, acting as the tracking subject determination unit 123 in Figure 8, obtains the control content CAMERA_ROLE corresponding to the role set for the sub-camera 600. Specifically, the CPU 101 reads the control content CAMERA_ROLE obtained in the role determination process explained using Figure 10 and stored in RAM 102. Note that the CPU 101 executes the processes in S205 to S207 for each sub-camera.

[0236] In S206, the CPU 101, acting as the tracking subject determination unit 123, determines the subject to be tracked and photographed by the sub-camera 600 according to the control content CAMERA_ROLE. The CPU 101 determines the subject to be tracked by the sub-camera 600 according to the definition of the tracking subject included in the control content CAMERA_ROLE (Figure 9).

[0237] If the subject being tracked by sub-camera 600 is to be the same as the subject of interest of main camera 500, CPU 101 sets the identification information of the subject of interest determined in S203, MAIN_SUBJECT, as the identification information of the subject being tracked by sub-camera 600, SUBJECT_ID.

[0238] If the subject to be tracked by the sub-camera 600 is to be a subject located to the left of the subject of interest of the main camera 500, the CPU 101 detects the subject region located at the leftmost edge of the subject region other than the subject of interest among the subject regions detected in S203. Then, the CPU 101 sets the identification information ID[n] corresponding to the detected subject region as the identification information SUBJECT_ID of the subject to be tracked by the sub-camera 600.

[0239] The CPU 101 writes the identification information SUBJECT_ID of the determined tracked subject to RAM 102. If the tracked subject may differ depending on the sub-camera, the CPU 101 stores the identification information SUBJECT_ID of the tracked subject in association with the identification information of the sub-camera. If the tracked subject changes, the CPU 101 retains the information of the previous tracked subject in RAM 102 without erasing it.

[0240] Here, we will explain the operation when the role set for the sub-camera 600 is "main follow," using Figure 14. When the role "main follow" is set for the sub-camera 600, the shooting control device 100 controls it to track the subject of interest of the main camera 500.

[0241] Therefore, when the subject of interest of the main camera 500 is determined to be subject B as shown in FIG. 14(a), the CPU 101 determines subject B as the tracking subject of the sub-camera 600. Thereafter, when it is determined that the subject of interest of the main camera 500 has changed to subject A as shown in FIG. 14(b), the CPU 101 changes the tracking subject of the sub-camera 600 to subject A. Similarly, when it is determined that the subject of interest of the main camera 500 has changed to subject C as shown in FIG. 14(c), the CPU 101 changes the tracking subject of the sub-camera 600 to subject C.

[0242] The operation when the role set for the sub-camera 600 is "assist follow" will be described with reference to FIG. 15. For the sub-camera 600 for which the role "assist follow" is set, the imaging control device 100 controls to track the subject located on the left among the subjects different from the subject of interest of the main camera 500.

[0243] Therefore, when the subject of interest of the main camera 500 is determined to be subject B as shown in FIG. 15(a), the CPU 101 determines the subject A on the left among subjects A and C as the tracking subject of the sub-camera 600. Thereafter, when it is determined that the subject of interest of the main camera 500 has changed to subject A as shown in FIG. 15(b), the CPU 101 changes the tracking subject of the sub-camera 600 to the subject B on the left among subjects B and C. Also, when it is determined that the subject of interest of the main camera 500 has changed to subject C as shown in FIG. 15(c), the CPU 101 changes the tracking subject of the sub-camera 600 to the subject A on the left among subjects A and B.

[0244] By dynamically changing the roles set for the sub-cameras 600 to 900 by the role control device 400, the tracking subjects of the sub-cameras 600 to 900 can be changed, enabling flexible automatic imaging.

[0245] Returning to Fig. 11(a), in S207, the CPU 101 as the pan-tilt value calculation unit 124 calculates the amount of change in the pan angle and tilt angle necessary for the sub-camera 600 to perform tracking shooting of the tracking subject determined in S206. Also, the CPU 101 as the zoom value calculation unit 125 calculates the zoom value of the sub-camera 600 according to the change in the angle of view of the main camera 500. Although the sub-camera 600 will be described below, the calculation of the amount of change in the pan angle and tilt angle and the calculation of the zoom value are similarly performed for the other sub-cameras 700 to 900.

[0246] First, the operation of the CPU 101 as the pan-tilt value calculation unit 124 will be described. Here, it is assumed that the following information is stored in advance in the ROM 103 as the default position information REF_POSI for each sub-camera. · Three-dimensional coordinates of the installation position (values in the plane coordinate system) · Shooting direction corresponding to the initial values of the pan angle and tilt angle of the drive unit · Controllable range of the pan and tilt angles

[0247] The CPU 101 reads out the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking subject of the sub-camera 600 from the RAM 102. Then, the CPU 101 first determines the pan angle from the position information POSITION_OH and the installation position of the sub-camera 600.

[0248] Fig. 16 is a diagram showing an example of the positional relationship between the sub-camera 600 and the tracking subject in the plane coordinate system. Here, it is assumed that the pan angle θ in the direction of the optical axis of the sub-camera 600 towards the subject position is determined. The CPU 101 calculates the pan angle θ using the following equation (2).

Equation

[0249] In Equation 2, px and py are the horizontal and vertical coordinates of the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracked subject. Also, subx and suby are the horizontal and vertical coordinates of the installation position of the sub-camera. Here, it is assumed that the current pan angle is the initial value of 0°, and the optical axis direction is the vertical direction (Y-axis direction). If the current optical axis direction is not the vertical direction, the angle difference between the current optical axis direction and the vertical direction may be reflected in the angle obtained by Equation 2. Also, the direction of pan is counterclockwise if subx > px, and clockwise if subx < px.

[0250] Next, the method for determining the tilt angle will be described using FIG. 17. FIG. 17 shows a state of the sub-camera and the tracked subject viewed from the side. It is assumed that the current optical axis of the sub-camera 600 is in the horizontal direction and the height is h1, and the height of the face of the tracked subject facing the optical axis is h2. Let ρ be the angle difference in the height direction (tilt angle) between the current optical axis direction and the target optical axis direction. The CPU 101 calculates the tilt angle ρ using the following Equations 3 and 4.

Equation

[0251] The coordinate values used in Equation 4 are the same as those used in Equation 2. It is assumed that h1 and h2 are input in advance to the shooting control application and stored in the RAM 102. In this case, the identification number associated with h2 for each subject is made equal to the identification number assigned in the subject detection process. Alternatively, h2 may be a value measured in real time using a sensor not shown.

[0252] Here, it is assumed that the current tilt angle is the initial value of 0°, and the optical axis direction is the horizontal direction (constant height). If the current optical axis direction is not the horizontal direction, the angle difference between the current optical axis direction and the horizontal direction may be reflected in the angle obtained by Equation 4. Also, the direction of tilt is downward if h1 > h2, and upward if h1 < h2.

[0253] The CPU 101 periodically communicates with the sub-camera 600 via the communication network 200 to obtain the current optical axis direction (pan angle and tilt angle of the drive unit) and stores it in the RAM 102. The communication period can be, for example, less than or equal to the reciprocal of the frame rate. Alternatively, the CPU 101 may store the sum of the pan angle and tilt angle controlled for the sub-camera 600 from its initial state in the RAM 102 and use this as the current optical axis direction.

[0254] The CPU 101 calculates the pan and tilt angle changes for sub-camera 600 in this manner and stores them in RAM 102. The CPU 101 also calculates the pan and tilt angle changes for the other sub-cameras 700 to 900.

[0255] The changes in pan and tilt angles may be the angular velocity required for the sub-camera 600 to rotate towards the tracked subject. For example, the CPU 101 obtains the current pan and tilt angles from the sub-camera 600 via the communication network 200. The CPU 101 then calculates the angular velocity of pan proportional to the difference between the pan angle θ read from RAM 102 and the current pan angle. The CPU 101 also calculates the angular velocity of tilt proportional to the difference between the tilt angle ρ read from RAM 102 and the current tilt angle. The CPU 101 stores these calculated angular velocities in RAM 102.

[0256] Alternatively, the change in pan angle and tilt angle may be calculated using the image from the sub-camera 600 instead of the image from the overhead camera 300. In this case, the CPU 101 may calculate the change in pan angle from the horizontal difference between the current optical axis direction and the direction of the tracked subject in the coordinate system of the sub-camera 600, and the change in tilt angle from the vertical difference. Furthermore, the imaging system may change the shooting direction for tracking the tracked subject by only changing either the pan direction or the tilt direction, and in such an imaging system, only the change in either the pan angle or the tilt angle may be calculated.

[0257] Next, the operation of the CPU 101 as the zoom value calculation unit 125 will be explained. The CPU 101 as the zoom value calculation unit 125 periodically acquires information MAIN_ZOOM indicating the field of view of the main camera 500 and stores it in RAM 102. When the information MAIN_ZOOM changes, the CPU 101 calculates the zoom value Z_VALUE for the sub-camera 600 according to the control content CAMERA_ROLE corresponding to the role set for the sub-camera 600.

[0258] The CPU 101 can determine the zoom operation of the main camera 500 and its phase, for example, by detecting changes in the field of view of the image from the main camera 500. For example, changes in the field of view may be detected from changes in the size and spacing of the subject area over time.

[0259] Figure 18 shows an example of mapping the zoom values ​​of the main camera and sub-cameras. Here, it is assumed that the main camera 500 and sub-cameras 600-900 optically change their field of view (the imaging optical system has a zoom function). However, a similar function may be achieved with digital zoom using the image processing units 506 and 606.

[0260] The zoom value is a parameter whose value corresponds to the angle of view. In this embodiment, the smaller (narrower) the angle of view, the smaller the zoom value, and the zoom value on the telephoto side is smaller than the zoom value on the wide-angle side. Sub-cameras 600-900 and the main camera 500 can control the imaging optical system to the angle of view corresponding to the zoom value by sending a command that specifies the zoom value. In other words, the zoom value is information about the angle of view and represents the zoom state. The zoom value may also be, for example, the focal length (mm) of the imaging optical system corresponding to a 35mm full-frame image sensor, in which case the zoom value on the telephoto side will be larger than the zoom value on the wide-angle side.

[0261] In FIG. 18, the range of the zoom value MAIN_ZOOM of the main camera 500 is main_min to main_max. Also, the zoom ranges of the sub-cameras 600 to 900 are sub_min to sub_max. main_min and sub_min are the telephoto ends of the main camera 500 and the sub-cameras 600 to 900 respectively, and main_max and sub_max are the zoom values corresponding to the wide-angle ends of the main camera 500 and the sub-cameras 600 to 900 respectively. FIG. 18 shows an example where the range of the zoom value of the main camera 500 is wider than the range of the zoom values of the sub-cameras 600 to 900 for both the telephoto end and the wide-angle end.

[0262] When controlling the zoom value SUB_ZOOM of the sub-camera 600 to be in the same phase as the zoom value MAIN_ZOOM of the main camera 500, the CPU 101 calculates the SUB_ZOOM corresponding to the current MAIN_ZOOM using the following equation 5.

Equation

[0263] On the other hand, when controlling the zoom value SUB_ZOOM of the sub-camera 600 to be in the opposite phase to the zoom value MAIN_ZOOM of the main camera 500, the SUB_ZOOM corresponding to the current MAIN_ZOOM is calculated using the following equation 6. Specifically, the CPU 101 substitutes the SUB_ZOOM calculated by equation 5 into the right side of the following equation 6 to calculate the SUB_ZOOM corresponding to the current MAIN_ZOOM. SUB_ZOOM = sub_max - (SUB_ZOOM - sub_min) ···(Equation 6)

[0264] When the main camera 500 performs digital zoom and controls the field of view by cropping, the CPU 101 can determine the zoom value SUB_ZOOM of the sub-camera 600 according to the size of the area to be cropped by the main camera 500. Specifically, the CPU 101 sets the zoom value SUB_ZOOM to a smaller size (higher magnification) when the area to be cropped by the main camera 500 is small, and to a larger size (lower magnification) when the area to be cropped is large.

[0265] Furthermore, the zoom control associated with the roles of sub-cameras 600-900 is not limited to in-phase or out-of-phase control with the main camera 500. For example, a zoom operation independent of the angle of view change of the main camera 500 may be associated with the role. For example, an auto-zoom operation that maintains a constant size of the tracked subject may be associated with the role. Alternatively, the angle of view of sub-cameras 600-900 may be fixed to a specific angle of view. By adding roles associated with these zoom controls to the control content for each role shown in Figure 9, or by changing the zoom control content of the roles shown in Figure 9, various zoom controls for sub-cameras 600-900 become possible.

[0266] Returning to Figure 11(a), in S207, the CPU 101 reads the pan and tilt angle changes and the zoom value calculated in S206 from the RAM 102. The CPU 101 then generates a control command PT_VALUE that instructs the sub-camera 600 to change the pan and tilt angles by the amounts of these changes. The CPU 101 also generates a control command Z_VALUE that instructs the sub-camera 600 to change the field of view by the zoom value. The format of the control commands is predetermined. The CPU 101 stores the generated control commands PT_VALUE and Z_VALUE in the RAM 102. Note that if the tracked subject is stationary or the field of view of the main camera 500 does not change, and there is no need to generate control commands, S207 may be skipped.

[0267] The CPU 101 then reads the control commands PT_VALUE and Z_VALUE from RAM 102 and sends them to the communication network 200 via network I / F 105. The sub-camera 600 receives the control commands PT_VALUE and Z_VALUE via network I / F 605.

[0268] CPU 101 executes the processing from S201 for the next frame image of the video from the overhead camera 300. Note that the processing shown in Figure 11(a) does not necessarily have to be executed every frame.

[0269] (Operation of overhead camera 300) Next, the operation of the overhead camera 300 will be explained with reference to Figure 11(b). The operation described below is achieved by the CPU 301 executing a program.

[0270] When the overhead camera 300 is powered on, the CPU 301 initializes each functional block, and then the camera enters a shooting standby state. In the shooting standby state, the CPU 301 may start video recording processing for live view display and output the display image data generated by the image processing unit 306 to the shooting control device 100 via the network I / F 305.

[0271] In the shooting standby state, the CPU 301 waits for the reception of control commands via the network I / F 305. When the CPU 301 receives a control command, it executes an action corresponding to the control command. This section describes the operation when a shooting command is received as a control command from the shooting control device 100.

[0272] In S301, CPU301 receives shooting commands from the shooting control device 100 via the network I / F305.

[0273] The shooting command may also specify shooting parameters such as frame rate and resolution. Furthermore, it may include settings related to the processing to be applied by the image processing unit 306.

[0274] In S302, the CPU 301 responds to the reception of a shooting command and starts video recording processing to be supplied to the shooting control device 100. This video recording processing captures video with higher image quality than the video recording processing for live view display. For example, at least one of the video resolution and the shooting frame rate is higher than that of the video for live view display. The image processing unit 306 applies processing to the image based on the settings for the video to be supplied to the shooting control device 100. The image processing unit 306 sequentially stores the generated video data in the RAM 302.

[0275] In step S303, the CPU 101 reads the video data from the RAM 302 and transmits it to the shooting control device 100 via the network interface 305. From this point onward, the processing from shooting to the supply of video data continues until a control command to stop shooting is received.

[0276] (Operation of Main Camera 500) Next, the operation of the main 500 will be explained with reference to Figure 11(c). The operations described below are achieved by the CPU 501 executing a program.

[0277] When the main camera 500 is powered on, the CPU 501 initializes each functional block, and then starts video recording processing to supply to the shooting control device 100. The image processing unit 506 applies processing to the analog image signal obtained from the image sensor 507 based on the settings for video to be supplied to the shooting control device 100. The image processing unit 506 sequentially stores the generated video data in the RAM 502. The CPU 501 reads the video data from the RAM 502 and supplies it to the shooting control device 100 via the network I / F 505.

[0278] The CPU 501 supplies video data to the shooting control device 100 while waiting for control commands to be received via the network interface 305. When the CPU 501 receives a control command, it executes an action corresponding to the control command. This section describes the operation when a shooting direction acquisition command is received. Note that when the pan / tilt control command PT_VALUE or the zoom control command Z_VALUE is received, the CPU 501 drives the drive unit 509 according to the command.

[0279] In S501, CPU501 receives a shooting direction acquisition command via network I / F505. CPU501 stores the received shooting direction acquisition command in RAM502.

[0280] In S502, the CPU 501, in response to receiving a command to acquire the shooting direction, acquires the current pan angle and tilt angle from the drive unit 509 via the drive I / F 508 and stores them in the RAM 502.

[0281] In S503, the CPU 501 reads the current pan angle and tilt angle from RAM 502 and transmits the shooting direction information ANGLE to the shooting control device 100 via the network I / F 305.

[0282] (Operation of sub-camera 600) Next, the operation of sub-camera 600 will be explained with reference to Figure 11(d). The operation described below is achieved by the CPU 601 executing a program. Sub-cameras 700 to 900 perform similar operations.

[0283] When the sub-camera 600 is powered on, the CPU 601 initializes each functional block, and then starts video recording processing to supply to the shooting control device 100. The image processing unit 606 applies processing to the analog image signal obtained from the image sensor 607 based on the settings for video to be supplied to the shooting control device 100. The image processing unit 606 sequentially stores the generated video data in the RAM 602. The CPU 601 reads the video data from the RAM 602 and supplies it to the shooting control device 100 via the network I / F 605.

[0284] The CPU 601 supplies video data to the shooting control device 100 while waiting for control commands to be received via the network interface 305. When the CPU 601 receives a control command, it executes an action corresponding to the control command. This section describes the operation when the CPU 601 receives the pan / tilt control command PT_VALUE and the zoom control command Z_VALUE from the shooting control device 100.

[0285] In S401, the CPU 601 receives at least one of the pan / tilt control command PT_VALUE and the zoom control command Z_VALUE from the imaging control device 100 via the network I / F 605. The CPU 601 stores the received control commands in the RAM 602.

[0286] In S402, CPU601 reads the control command stored in RAM602 and the corresponding control variable, and stores them in RAM602. Here, in the case of the pan / tilt control command PT_VALUE, the control direction is the direction of pan and / or tilt, and the control variable is the target angle. In the case of the zoom control command Z_VALUE, the control variable is the zoom value, and since the control direction can be determined from the zoom value, reading and storing the control direction is unnecessary.

[0287] In S403, the CPU 601 generates drive parameters for the drive unit 609 based on the operating direction and operating amount read in S403. The CPU 601 may, for example, obtain drive parameters corresponding to a combination of operating direction and operating amount using a table previously stored in ROM 603. If the operating amount is given as a target value (target angle or zoom value), the CPU 410 obtains the drive parameters from the difference between that value and the current value.

[0288] In S404, the CPU 601 controls the drive unit 609 via the drive I / F 608 based on the drive parameters acquired in S404. This causes the drive unit 609 to change the shooting direction of the sub-camera 600 to the operating direction and angle specified by the pan / tilt control command PT_VALUE. The drive unit 609 also changes the field of view of the shooting optical system to the zoom value specified by the zoom control command Z_VALUE.

[0289] Next, using the flowchart shown in Figure 19, we will explain in more detail how the shooting control device 100 controls the shooting direction (pan and tilt) and field of view (zoom value) of the sub-camera 600 according to the role set for the sub-camera. The operations shown in the flowchart of Figure 19 are performed as part of the operations S205 to S207 in Figure 11(a). The shooting control device 100 performs similar operations for sub-cameras 700 to 900.

[0290] S601 corresponds to S205, and CPU101 reads the control content CAMERA_ROLE stored in RAM102 in S103 of Figure 10.

[0291] S602-S607 are implemented, for example, in S206. In S602, the CPU 101 determines whether the tracking subject specification for sub-camera 600, included in the control content CAMERA_ROLE, indicates the tracking subject (subject of interest) of main camera 500. For example, if the tracking subject specification for sub-camera 600 has a value indicating "same as main", the CPU 101 determines that the tracking subject specification for sub-camera 600 indicates the tracking subject of main camera 500 and executes S603. On the other hand, if the tracking subject specification for sub-camera 600 has a value indicating "different from main (left side)", the CPU 101 determines that the tracking subject specification for sub-camera 600 does not indicate the tracking subject of main camera 500 and executes S604.

[0292] In S603, CPU101 decides to control the shooting direction of sub-camera 600 so that it tracks the subject being tracked (the subject of interest) by main camera 500. In S604, CPU101 decides to control the shooting direction of sub-camera 600 to track a subject located to the left of the main camera 500's subject of interest.

[0293] In S605, the CPU 101 determines whether the zoom control specification for the sub-camera 600, included in the control content CAMERA_ROLE, indicates control in the same phase as the main camera 500. For example, if the zoom control specification for the sub-camera 600 has a value indicating "same phase as the main," the CPU 101 determines that the zoom control specification for the sub-camera 600 indicates control in the same phase as the main camera 500 and executes S606. On the other hand, if the zoom control specification for the sub-camera 600 has a value indicating "opposite phase to the main," the CPU 101 determines that the zoom control specification for the sub-camera 600 does not indicate control in the same phase as the main camera 500 and executes S607.

[0294] In S606, CPU101 decides to control the zoom value (angle of view) of sub-camera 600 in phase with the change in the zoom value of main camera 500. In S607, CPU101 decides to control the zoom value (angle of view) of sub-camera 600 in the opposite phase to the change in the zoom value of main camera 500.

[0295] Figure 20 illustrates an example of sub-camera control when the sub-camera 600 is set to "main follow". Figure 20 schematically shows how the shooting control device 100 controls the shooting direction and field of view of the sub-camera 600 when the subject of focus and field of view of the main camera 500 change over time during shooting. In the figure, time progresses from left to right. Note that in Figure 20, the zoom state is shown in three stages: "telephoto end", "intermediate", and "wide-angle end". This is because, as shown in Figure 18, the zoom range of the sub-camera and the main camera may differ. "Telephoto end" corresponds to the state when zoomed up to the telephoto end, "wide-angle end" corresponds to the state when zoomed down to the wide-angle end, and "intermediate" corresponds to a zoom state between the telephoto end and the wide-angle end, but the actual zoom values ​​may differ between the sub-camera and the main camera. For example, when the zoom state is "telephoto end", the zoom value of the main camera is main_min, and the zoom value of the sub-camera is sub_min.

[0296] Initially, the main camera 500's focus subject (tracking subject) is subject B, and the zoom level is set to "intermediate." Therefore, the CPU 101 determines that the sub-camera 600's tracking subject is subject B and controls the shooting direction so that the sub-camera 600 tracks subject B. The CPU 101 also controls the zoom level of the sub-camera 600 to "intermediate."

[0297] Subsequently, the main camera 500's focus subject changes from subject B to subject A, and its zoom state is changed to "telephoto end." In response, the CPU 101 changes the subject being tracked by the sub-camera 600 from subject B to subject A and controls the shooting direction so that the sub-camera 600 tracks subject A. The CPU 101 also controls the zoom state of the sub-camera 600 to "telephoto end."

[0298] Subsequently, the main camera 500's focus subject changes from subject A to subject C, and its zoom state changes to "wide-angle end." In response, the CPU 101 changes the subject being tracked by the sub-camera 600 from subject A to subject C and controls the shooting direction so that the sub-camera 600 tracks subject C. The CPU 101 also controls the zoom state of the sub-camera 600 to "wide-angle end."

[0299] Thus, when the sub-camera 600's role is "main follow," the CPU 101 automatically changes the tracked subject and zoom value of the sub-camera 600 to follow the changes in the subject of interest and field of view (zoom value) of the main camera 500. In the example shown in Figure 20, the sub-camera 600 is controlled to change in zoom state to be the same as that of the main camera 500, but the actual zoom values ​​may differ as long as the direction of change in zoom value is in phase. For example, the sub-camera 600 does not need to be at the telephoto end when the main camera 500 is at its telephoto end. Whether or not to match the zoom value of the sub-camera 600 to the zoom value of the main camera 500 may be configurable using role setting information.

[0300] Next, an example of controlling the sub-camera 600 when its assigned role is "assist counter" will be explained using Figure 21, which is similar to Figure 20.

[0301] Initially, the main camera 500's focus subject (tracking subject) is subject B, and the zoom state is at the "telephoto end". Therefore, the CPU 101 determines that the sub-camera 600's tracking subject is subject A, the leftmost of subjects A and C other than subject B, and controls the shooting direction so that the sub-camera 600 tracks subject A. The CPU 101 also controls the zoom state of the sub-camera 600 to the "wide-angle end", which is in the opposite phase to that of the main camera 500.

[0302] Subsequently, the main camera 500's focus subject changes from subject B to subject A, and its zoom level changes to "intermediate." In response, the CPU 101 changes the subject being tracked by the sub-camera 600 to subject B, which is on the left of subjects B and C other than subject A, and controls the shooting direction so that the sub-camera 600 tracks subject B. Also, because the zoom level of the main camera 500 has changed from "telephoto end" to "intermediate," the CPU 101 controls the zoom level of the sub-camera 600 from "wide-angle end" to "intermediate" (in opposite phase).

[0303] Subsequently, the main camera 500's focus subject changes from subject A to subject C, and its zoom state changes to "wide-angle end." In response, the CPU 101 changes the subject being tracked by the sub-camera 600 to subject A, which is the leftmost of subjects A and B (excluding subject C), and controls the shooting direction so that the sub-camera 600 tracks subject A. Also, because the main camera 500's zoom state has changed from "intermediate" to "wide-angle end," the CPU 101 controls the sub-camera 600's zoom state from "intermediate" to "telephoto end" (in opposite phase).

[0304] Thus, when the sub-camera 600's role is that of an "assist counter," the CPU 101 automatically changes the subject tracked by the sub-camera 600 to a different subject from the main camera 500's subject of focus, in response to changes in the main camera 500's subject of focus. In addition, the CPU 101 automatically changes the zoom value of the sub-camera 600 in the opposite direction to the change in the main camera 500's field of view (zoom value).

[0305] In the example shown in Figure 21, the zoom state of the sub-camera 600 is controlled to the same degree of change as the zoom state of the main camera 500. However, the amount of change in the zoom value may differ if the direction of change in the zoom value is in opposite phase. For example, the zoom state of the sub-camera 600 does not need to be at the wide-angle end when the zoom state of the main camera 500 is at the telephoto end. The degree of change in the zoom state of the sub-camera 600 relative to the degree of change in the zoom state of the main camera 500 may be configurable using role setting information.

[0306] (modified version) Up to this point, we have described an example of automatically controlling the tracking subject and zoom value of the sub-camera 600 based on the focus subject and zoom value of the main camera 500. In the example above, the sub-camera was automatically controlled to track a single subject, but it is also possible to automatically control it to track multiple subjects within the shooting range.

[0307] Figure 22 illustrates the control methods for tracking a single subject and multiple subjects using the sub-camera 600 when its role is set to "Assist Follow". For ease of understanding and explanation, this section describes the case where the field of view of the main camera 500 does not change and only the control of the tracked subject is performed. It is also assumed that the field of view of the sub-camera 600 is always capable of capturing all subjects within the shooting range 20, regardless of the shooting direction. Note that the shooting direction of the sub-camera 600 shown at the top of Figure 22 indicates the shooting direction when tracking a single subject.

[0308] Similar to Figure 21, initially, the subject of interest (tracking subject) for the main camera 500 is subject B. Therefore, the CPU 101 determines that the tracking subject for the sub-camera 600 is subject A, the leftmost of subjects A and C other than subject B, and controls the shooting direction so that the sub-camera 600 tracks subject A. If the shooting direction is controlled so that the tracking subject is in the center of the screen, the image from the sub-camera 600 will be unbalanced, with subjects A-C shifted to the right, as shown in the second row from the bottom. Therefore, if the image from the sub-camera 600 contains multiple subjects, including the tracking subject, the shooting direction can be controlled to track all of them. For example, the CPU 101 can control the shooting direction to track the center of gravity of the multiple subjects A-C included in the image from the sub-camera 600. As a result, the sub-camera 600 captures an image like the one shown in the bottom row.

[0309] In the example shown in Figure 22, regardless of which of subjects A to C the sub-camera 600 tracks, all subjects A to C are captured. Therefore, even if the subject of focus for the main camera 500 changes, the shooting direction of the sub-camera 600 remains almost constant.

[0310] (Modification 2) Furthermore, after determining the subject to be tracked by the sub-camera 600, the CPU 101 may control the sub-camera 600 to focus on the subject. Basically, the CPU 601 continuously controls the focus distance of the sub-camera 600 to focus on the specified subject, but the shooting control device 100 can set the AF frame of the sub-camera 600 to the position of the subject. This allows the sub-camera 600 to focus on the subject quickly and reliably. Note that if the AF frame is set when the panning speed is slow (below a threshold), the subject to be tracked is likely to be in the center of the screen, which may shorten the time required for focusing.

[0311] (Variation 3) Furthermore, the sub-camera 600 can be controlled even without using the overhead camera 300. In this case, the shooting direction of the sub-camera 600 can be determined from the installation positions of the main camera 500 and the sub-camera 600, and the shooting direction of the main camera 500 (the orientation of the main camera 500 relative to the subject being tracked). The sub-camera 600 can be controlled by having the main camera 500 perform subject detection processing, and the CPU 101 acquire and use the results. For example, the CPU 101 acquires an image of the subject area from the main camera 500 as a result of the subject detection processing. Then, the CPU 101 can control the sub-camera 600 to perform subject tracking processing using the acquired image as a template. Alternatively, the CPU 101 may use the acquired image as a template and control the sub-camera 600 to track a subject area with a low correlation in the template.

[0312] (Modification 4) Although the shooting control device 100, the role control device 400, and the main camera 500 have been described as separate devices, the functions of the shooting control device 100 and the role control device 400 can also be integrated into the main camera 500. In this case, the image from the overhead camera 300 is supplied to the main camera 500. This configuration reduces the amount of equipment required to realize a multi-camera imaging system.

[0313] As described above, according to this embodiment, when automatically controlling the operation of the sub-camera based on the state and image of the main camera, automatic control is performed according to the role set for the sub-camera. Therefore, the shooting control device of this embodiment can achieve a higher degree of freedom in automatic shooting control while saving labor.

[0314] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0315] The disclosure of this embodiment includes the following imaging control device, imaging control method, imaging system, and program. (Item 1) A control means for controlling the shooting direction and field of view of a sub-camera, among a plurality of cameras including a main camera and two or more sub-cameras, based on the role assigned to the sub-camera and the subject of interest and field of view of the main camera. A determination means for determining whether the main camera satisfies predetermined conditions, The control means is characterized in that, when it is determined that the main camera satisfies the conditions, it selects one or more of the two or more sub-cameras and changes the control content of the selected sub-camera to track and photograph the subject of interest of the main camera with settings different from those of the main camera. (Item 2) The shooting control device according to item 1, characterized in that the control means controls the angle of view of the selected sub-camera in the same phase as the angle of view of the main camera when it is determined that the main camera satisfies the conditions. (Item 3) The determination means performs the determination based on multiple conditions, The aforementioned multiple conditions, One or more conditions relating to the brightness of the image from the main camera, One or more conditions relating to one or more of the brightness, position, movement speed, size, and depth of field of the subject area in the image from the main camera, One or more conditions relating to the movement of the main camera, One or more conditions relating to the white balance of the main camera, A shooting control device according to item 1 or 2, characterized by including one or more of the above. (Item 4) The shooting control device according to item 3, characterized in that the control means makes the type of setting to be different in the selected sub-camera from that of the main camera according to the conditions that the main camera satisfies among the plurality of conditions. (Item 5) The shooting control device according to item 4, characterized in that the control means causes the exposure amount setting of the selected sub-camera to differ from that of the main camera when the main camera satisfies conditions relating to the brightness of the image of the main camera or the brightness of the subject area. (Item 6) The shooting control device according to item 4 or 5, characterized in that the control means causes the angle of view setting of the selected sub-camera to be different from that of the main camera when the main camera satisfies one or more conditions relating to the position, movement speed, and size of the subject area. (Item 7) The shooting control device according to any one of items 4 to 6, characterized in that the control means causes the aperture value setting of the selected sub-camera to be different from that of the main camera when the main camera satisfies the conditions relating to the depth of field. (Item 8) The shooting control device according to any one of items 4 to 7, characterized in that the control means causes the shutter speed and / or shooting sensitivity setting of the selected sub-camera to be different from that of the main camera when the main camera satisfies the conditions relating to the movement of the main camera. (Item 9) The shooting control device according to any one of items 4 to 8, characterized in that the control means causes the white balance setting of the selected sub-camera to be different from that of the main camera when the main camera satisfies the conditions relating to the white balance. (Item 10) The shooting control device according to any one of items 1 to 9, characterized in that the control means selects a pre-set sub-camera from among the two or more sub-cameras. (Item 11) The shooting control device according to any one of items 1 to 9, characterized in that the control means selects from the two or more sub-cameras a sub-camera other than the one currently shooting the image selected by the external device. (Item 12) The shooting control device according to any one of items 1 to 9, characterized in that the control means selects a sub-camera from among the two or more sub-cameras that is capable of shooting with a composition similar to that of the main camera. (Item 13) A sub-camera capable of shooting with a composition similar to that of the main camera is provided. The shooting control device according to item 12, characterized in that it is a sub-camera capable of shooting an area that includes all the subjects to be photographed by the main camera, and capable of photographing the subjects at a size where the difference in size from the image of the main camera is below a threshold. (Item 14) A sub-camera capable of shooting with a composition similar to that of the main camera is provided. The shooting control device according to item 12, characterized in that the installation position is closest to the main camera, the distance from the main camera is below a threshold, and the range of the focal length of the shooting optical system overlaps with the range of the focal length of the shooting optical system of the main camera by a threshold ratio or more. (Item 15) The shooting control device according to any one of items 1 to 9, characterized in that, if there is no sub-camera capable of shooting with a composition similar to that of the main camera among the two or more sub-cameras, the control means selects the sub-camera with the lowest predetermined priority among the two or more sub-cameras. (Item 16) The shooting control device according to any one of items 1 to 15, characterized in that when a plurality of sub-cameras are selected, the control means provides different control content for each of the selected sub-cameras. (Item 17) A shooting control method performed by a shooting control device, Among a main camera and a plurality of cameras including two or more sub-cameras, the shooting direction and field of view of the sub-cameras are controlled based on the role set for the sub-camera and the subject of interest and field of view of the main camera. The main camera has the ability to determine whether or not predetermined conditions are met, The aforementioned control means If the main camera is determined to satisfy the above conditions, then one or more of the two or more sub-cameras are selected. This includes changing the control settings of the selected sub-camera to track and photograph the subject of interest of the main camera with settings different from those of the main camera, A method for controlling photography, characterized by the following features. (Item 18) A control means for controlling the shooting direction and field of view of each of the multiple cameras based on at least the role set for each of the multiple cameras, A determination means for determining whether or not there is a camera in a first state that is not performing an operation associated with a set role among the plurality of cameras, The camera capture control device is characterized in that, when the control means determines that a camera in the first state exists, it selects a camera other than the camera in the first state from among the plurality of cameras, and changes the control content of the selected sub-camera to perform shooting that complements the image to be captured by the camera in the first state. (Item 19) The shooting control device according to item 18, characterized in that, if the control means does not match the control content of the field of view associated with the role set for the camera in the first state and the selected camera, the control content of the selected camera is changed to the control content according to the role set for the camera in the first state. (Item 20) The control means is If the angle of view control content associated with the role set for the camera in the first state and the selected camera matches, and the amount of angle of view adjustment required for the selected camera to capture the subject to be photographed by both the selected camera and the camera in the first state is less than or equal to a threshold, The shooting control device according to item 18, characterized in that the selected camera changes the control content of the selected camera so that it shoots an image that includes the subject to be photographed within its shooting range, in accordance with the roles set for the selected camera and the camera in the first state. (Item 21) The control means is If the angle of view control content associated with the role set for the camera in the first state and the selected camera matches, and the amount of angle of view adjustment required for the selected camera to capture the subject to be photographed by both the selected camera and the camera in the first state is not below a threshold, If the distance between the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state is less than a threshold, The shooting control device according to item 18, characterized in that the selected camera changes the control content of the selected camera so that it alternately photographs the subject that the selected camera is to photograph and the subject that the camera in the first state is to photograph, according to the roles set for the selected camera and the camera in the first state. (Item 22) The control means is If the angle of view control content associated with the role set for the camera in the first state and the selected camera matches, and the amount of angle of view adjustment required for the selected camera to capture the subject to be photographed by both the selected camera and the camera in the first state is not below a threshold, If the distance between the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state is not less than the threshold, The shooting control device according to item 18, characterized in that it changes the control content of the selected camera to a control content that conforms to the role set for the camera in the first state. (Item 23) The shooting control device according to any one of items 18 to 22, characterized in that the determination means determines that a camera autonomously performing shooting in a role different from the set role is a camera in the first state. (Item 24) The camera control device according to any one of items 18 to 23, characterized in that the determination means determines a camera in which an abnormality has been detected as a camera in the first state. (Item 25) The shooting control device according to any one of items 18 to 24, characterized in that the determination means determines a camera that has been detected to have been operated by means outside the control of the control means as a camera in the first state. (Item 26) The shooting control device according to any one of items 18 to 25, characterized in that the control means selects a camera from among the plurality of cameras other than the camera in the first state that is similar in position and capability to the camera in the first state and has a predetermined priority lower than the camera in the first state. (Item 27) The shooting control device according to item 26, characterized in that a camera similar in position and capability to the camera in the first state is a camera whose distance from the camera in the first state to its installation position is less than a threshold, and whose overlap in pan-tilt drive range and zoom range is each greater than or equal to a predetermined percentage. (Item 28) The control means is If, among the multiple cameras, there are no cameras other than the camera in the first state that are similar in position and capabilities to the camera in the first state, nor are there any cameras that are assigned the same role as the camera in the first state, The shooting control device according to any one of items 18 to 25, characterized in that it selects a camera from the plurality of cameras that has the lowest predetermined priority and is not a camera in the first state. (Item 29) The control means is If, among the plurality of cameras, there is no camera other than the camera in the first state that has a similar position and capability to the camera in the first state, The shooting control device according to any one of items 18 to 25, characterized in that, among cameras that have the same role as the camera in the first state, the camera has the lowest predetermined priority among the multiple cameras and the priority is lower than that of the camera in the first state. (Item 30) The shooting control device according to any one of items 18 to 29, characterized in that, when a main camera different from the plurality of cameras is present, the control means controls the shooting direction and field of view of the plurality of cameras based on the role set for each of the plurality of cameras and the subject of interest and field of view of the main camera. (Item 31) The shooting control device according to any one of items 18 to 30, characterized in that the control means notifies the user when it is determined that a camera in the first state exists. (Item 32) A shooting control method performed by a shooting control device, Controlling the shooting direction and field of view of each of the multiple cameras based on at least the role assigned to each of the multiple cameras, The method includes determining whether or not there is a camera in a first state that is not performing an operation associated with a set role among the plurality of cameras, The aforementioned control means If it is determined that a camera in the first state exists, one or more cameras other than the camera in the first state are selected from the plurality of cameras, This includes changing the control content of the selected sub-camera to perform shooting that complements the image to be captured by the camera in the first state, A method for controlling photography, characterized by the following features. (Item 33) A photographic control device as described in any one of items 1 to 16 and 18 to 31, The plurality of cameras, which are communicably connected to the aforementioned shooting control device, An imaging system characterized by having the following features. (Item 34) A program for causing a computer to function as one of the means of a photographic control device according to any one of claims 1 to 16 and 18 to 31.

[0316] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0317] 100...Shooting control device, 300...Overhead camera, 400...Role control device, 500...Main camera, 600, 700, 800, 900...Sub-camera, 101...CPU, 102...RAM, 103...ROM, 104...Inference unit, 105...Network I / F

Claims

1. A main camera whose shooting direction and field of view are controlled by the operator, Two or more sub-cameras whose shooting direction and field of view are controlled based on their assigned roles, A shooting control device that is communicatively connected to multiple cameras, including, A main camera information acquisition means for acquiring information about the main camera, including information about the subject of interest that the main camera is photographing, A control means for controlling the shooting direction and field of view of the sub-camera based on the role set for the sub-camera, The system includes a determination means for determining whether the main camera satisfies predetermined conditions, When the control means determines that the main camera satisfies the conditions, Select one or more of the two or more sub-cameras mentioned above, The selected sub-camera is assigned the role of simultaneously photographing the subject of interest being photographed by the main camera. A photographic control device characterized by the following:

2. The shooting control device according to claim 1, characterized in that the control means controls the selected sub-camera to shoot the changed subject when the subject of interest being photographed by the main camera is changed.

3. The shooting control device according to claim 1, characterized in that the control means is configured to set the selected sub-camera to shoot the subject of interest to be photographed by the main camera with different shooting settings than the main camera.

4. The shooting control device according to claim 3, characterized in that the shooting setting is at least one of the following: setting the amount of exposure, setting the angle of view, setting the aperture value, and setting the white balance.

5. The information of the main camera includes the field of view information of the main camera. The shooting control device according to claim 1, characterized in that the control means controls the angle of view of the selected sub-camera in the same phase as the angle of view of the main camera when it is determined that the main camera satisfies the conditions.

6. The determination means performs the determination based on multiple conditions, The aforementioned multiple conditions, One or more conditions relating to the brightness of the image from the main camera, One or more conditions relating to one or more of the brightness, position, movement speed, size, and depth of field of the subject area in the image from the main camera, One or more conditions relating to the movement of the main camera, One or more conditions relating to the white balance of the main camera, The imaging control device according to claim 1, characterized by including one or more of the above.

7. The shooting control device according to claim 6, characterized in that the control means sets the role of the selected sub-camera to shoot the subject of interest to be photographed by the main camera with settings different from those of the main camera, and sets the type of setting to be different for the selected sub-camera from that of the main camera according to the conditions that the main camera satisfies among the plurality of conditions.

8. The shooting control device according to claim 7, characterized in that the control means causes the exposure amount setting of the selected sub-camera to differ from that of the main camera when the main camera satisfies conditions relating to the brightness of the image of the main camera or the brightness of the subject area.

9. The shooting control device according to claim 7, characterized in that the control means causes the angle of view setting of the selected sub-camera to be different from that of the main camera when the main camera satisfies one or more conditions relating to the position, movement speed, and size of the subject area.

10. The shooting control device according to claim 7, characterized in that the control means causes the aperture value setting of the selected sub-camera to be different from that of the main camera when the main camera satisfies the conditions relating to the depth of field.

11. The shooting control device according to claim 7, characterized in that the control means causes the shutter speed and / or shooting sensitivity settings of the selected sub-camera to differ from those of the main camera when the main camera satisfies conditions relating to the movement of the main camera.

12. The shooting control device according to claim 7, characterized in that the control means causes the white balance setting of the selected sub-camera to be different from that of the main camera when the main camera satisfies the conditions relating to the white balance.

13. The shooting control device according to claim 1, characterized in that the control means selects a pre-set sub-camera from among the two or more sub-cameras.

14. The shooting control device according to claim 1, characterized in that the control means selects from the two or more sub-cameras a sub-camera other than the one currently shooting the image selected by the external device.

15. The shooting control device according to claim 1, characterized in that the control means selects a sub-camera from among the two or more sub-cameras that is capable of shooting with a composition similar to that of the main camera.

16. A sub-camera capable of shooting with a composition similar to that of the main camera is provided. The shooting control device according to claim 15, characterized in that it is a sub-camera capable of shooting an area that includes all the subjects to be photographed by the main camera, and capable of photographing the subjects at a size where the difference in size from the image of the main camera is below a threshold.

17. A sub-camera capable of shooting with a composition similar to that of the main camera is provided. The shooting control device according to claim 15, characterized in that the sub-camera is installed closest to the main camera, is at a distance of less than or equal to a threshold, and the range of the focal length of the shooting optical system overlaps with the range of the focal length of the shooting optical system of the main camera by a threshold ratio or more.

18. The shooting control device according to claim 1, characterized in that, if there is no sub-camera capable of shooting with a composition similar to that of the main camera among the two or more sub-cameras, the control means selects the sub-camera with the lowest predetermined priority among the two or more sub-cameras.

19. When a plurality of sub-cameras are selected, the control means controls each of the selected sub-cameras. The photographic control device according to claim 1, characterized in that the control content is different.

20. A main camera whose shooting direction and field of view are controlled by the operator, Two or more sub-cameras whose shooting direction and field of view are controlled according to their assigned roles, A shooting control method performed by a shooting control device that is communicatively connected to multiple cameras, including, A main camera information acquisition step, which includes acquiring information about the main camera, including information about the subject of interest that the main camera is photographing; A control step that controls the shooting direction and field of view of the sub-camera based on the role set for the sub-camera, The main camera has a determination step of determining whether or not it satisfies predetermined conditions, If it is determined that the main camera satisfies the above conditions, the control step is performed. The step of selecting one or more of the two or more sub-cameras, The steps include assigning the selected sub-camera the role of photographing the subject of interest being photographed by the main camera in parallel with the main camera's photography, A method for controlling photography, characterized by including the following:

21. A control means for controlling the shooting direction and field of view of each of the multiple cameras based on at least the role set for each of the multiple cameras, The system includes a determination means for determining whether or not there is a camera in a first state that is not performing an operation associated with a set role among the plurality of cameras. If it is determined that a camera in the first state exists, the control means will While continuing to take pictures with the camera in the first state described above, Select a camera other than the camera in the first state from among the multiple cameras mentioned above. The selected camera is assigned the role of performing a complementary shot to the image that should be captured by the camera in the first state. The aforementioned role is to instruct the subject to be photographed and the zoom operation. A photographic control device characterized by the following:

22. The shooting control device according to claim 21, characterized in that, if the zoom operation instructed by the role set for the camera in the first state does not match the zoom operation instructed by the role set for the selected camera, the control means changes the zoom operation of the selected camera to the zoom operation instructed by the role set for the camera in the first state.

23. The control means is If the zoom operation instructed by the role set for the camera in the first state matches the zoom operation instructed by the role set for the selected camera, and the amount of angle of view adjustment required for the selected camera to capture both the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state is less than or equal to a threshold, The shooting control device according to claim 21, characterized in that the selected camera changes the control content of the selected camera so that it shoots an image that includes both the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state, in accordance with the roles set for the selected camera and the camera in the first state.

24. The control means is If the zoom operation instructed by the role set for the camera in the first state matches the zoom operation instructed by the role set for the selected camera, and the amount of angle of view adjustment required for the selected camera to capture both the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state is not less than or equal to a threshold, If the distance between the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state is less than a threshold, The shooting control device according to claim 21, characterized in that the selected camera modifies the control content of the selected camera so that it alternately photographs the subject that the selected camera is to photograph and the subject that the camera in the first state is to photograph, according to the roles set for the selected camera and the camera in the first state.

25. The control means is If the zoom operation instructed by the role set for the camera in the first state matches the zoom operation instructed by the role set for the selected camera, and the amount of angle of view adjustment required for the selected camera to capture both the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state is not less than or equal to a threshold, If the distance between the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state is not less than the threshold, The shooting control device according to claim 21, characterized in that it changes the control content of the selected camera to a control content in accordance with the role set for the camera in the first state.

26. The shooting control device according to claim 21, characterized in that the determination means determines that a camera autonomously performing shooting in a role different from the set role is a camera in the first state.

27. The shooting control device according to claim 21, characterized in that the determination means determines that the camera in which an abnormality has been detected is a camera in the first state.

28. The shooting control device according to claim 21, characterized in that the determination means determines a camera that has been detected to have been operated by means outside the control of the control means to be a camera in the first state.

29. The camera control device according to claim 21, characterized in that, when it is determined that a camera in the first state exists, the control means selects a camera from among the plurality of cameras other than the camera in the first state, which has a similar position and capability to the camera in the first state, and which has a predetermined priority lower than the camera in the first state.

30. The shooting control device according to claim 29, characterized in that a camera similar in position and capability to the camera in the first state is a camera whose distance from the camera in the first state to its installation position is less than a threshold, and whose overlap in pan-tilt drive range and zoom range is each greater than or equal to a predetermined percentage.

31. If it is determined that a camera in the first state exists, the control means will If, among the plurality of cameras, there are no cameras other than the camera in the first state that are similar in position and capabilities to the camera in the first state, and no cameras that are assigned the same role as the camera in the first state, The shooting control device according to claim 21, characterized in that it selects a camera from the plurality of cameras that has the lowest predetermined priority and is not a camera in the first state.

32. If it is determined that a camera in the first state exists, the control means will If, among the plurality of cameras, there is no camera other than the camera in the first state that has a similar position and capability to the camera in the first state, The shooting control device according to claim 21, characterized in that, among cameras that have the same role as the camera in the first state, the camera has the lowest predetermined priority among the plurality of cameras and the priority is lower than that of the camera in the first state.

33. The shooting control device according to claim 21, wherein, when a main camera different from the plurality of cameras is present, the control means controls the shooting direction and field of view of the plurality of cameras based on the role set for each of the plurality of cameras, the field of view of the main camera and the subject of interest.

34. The shooting control device according to claim 21, characterized in that the control means notifies the user that a camera in the first state exists when it is determined that such a camera exists.

35. A shooting control method performed by a shooting control device, A control step that controls the shooting direction and field of view of each of the multiple cameras based on at least the role set for each of the multiple cameras, The method includes a determination step of determining whether or not there is a camera in a first state that is not performing an operation associated with a set role among the plurality of cameras, If it is determined that a camera in the first state exists, the control step is performed. While continuing to take pictures with the camera in the first state described above, A step of selecting one or more cameras from the aforementioned plurality of cameras other than the camera in the first state, The steps include assigning the selected camera the role of performing shooting that complements the image to be captured by the camera in the first state, The aforementioned role is to instruct the subject to be photographed and the zoom operation. A method for controlling photography, characterized by the following features.

36. A photographic control device according to any one of claims 1 to 19 and 21 to 34, The plurality of cameras, which are communicably connected to the aforementioned shooting control device, An imaging system characterized by having the following features.

37. A program for causing a computer to function as one of the means of the imaging control device described in any one of claims 1 to 19 and 21 to 34.

Images