Imaging control device and imaging control method, and imaging system

Abstract

To implement an automatic response in a case where a camera which cannot capture an estimated video image is generated in an imaging control device and an imaging control method for implementing automatic imaging cooperating a plurality of cameras.SOLUTION: An imaging control device controls an imaging direction and an angle of view of a sub camera in a plurality of cameras including a main camera and two or more sub cameras on the basis of a role, which is set to the sub cameras, and a subject of interest and an angle of view of the main camera. In a case where it is determined that the main camera fulfills a predetermined condition, the imaging control device selects one or more sub cameras from the two or more sub cameras and changes control contents of the selected sub cameras so as to track and photograph the subject of interest of the main camera with settings different from the main camera.SELECTED DRAWING: Figure 4A

Description

[Technical field]

[0001] The present invention relates to an imaging control device, an imaging control method, and an imaging system, and more particularly to a technique for controlling a plurality of imaging devices. [Background technology]

[0002] Patent Document 1 describes an imaging system in which a plurality of cameras are divided into main cameras and sub-cameras, and the sub-cameras are controlled so as to capture the same subject as that of the main camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-25248 A Summary of the Invention [Problem to be solved by the invention]

[0004] The imaging system described in Patent Document 1 is capable of automatic imaging by linking multiple cameras, which allows for labor saving. However, there is a risk that an appropriate response cannot be taken when an unexpected change in the imaging environment occurs, such as when a camera is unable to capture the expected image due to a malfunction or when an accident occurs to the subject.

[0005] In view of the problems with the conventional technology, the present invention provides an automatic response when an unexpected change in the shooting environment occurs in an shooting control device and a shooting control method that realizes automatic shooting by coordinating multiple cameras. [Means for solving the problem]

[0006] In one aspect, the present invention provides a shooting control device comprising: a control means for controlling the shooting direction and angle of view of the sub-cameras among a plurality of cameras including a main camera and two or more sub-cameras based on the role set for the sub-cameras and the subject of interest and angle of view of the main camera; a determination means for determining whether the main camera satisfies predetermined conditions; and when it is determined that the main camera satisfies the conditions, the control means selects one or more of the two or more sub-cameras and changes the control content of the selected sub-camera so that it tracks and shoots the subject of interest of the main camera with settings different from those of the main camera. Effect of the Invention

[0007] According to the present invention, in a shooting control device and a shooting control method that realize automatic shooting by coordinating multiple cameras, it is possible to automatically respond when an unexpected change in the shooting environment occurs. [Brief description of the drawings]

[0008] [Figure 1] Schematic diagram of an imaging system according to an embodiment. [Diagram 2] FIG. 1 is a block diagram showing an example of the functional configuration of each device in an imaging system according to an embodiment. [Diagram 3] FIG. 13 is a diagram showing examples of roles that can be set to a sub-camera in an embodiment; [Figure 4A] Flowchart regarding the operation of the imaging control device 100 according to the first embodiment [Figure 4B] Flowchart regarding the operation of the imaging control device 100 according to the first embodiment [Figure 4C] Flowchart regarding the operation of the imaging control device 100 according to the first embodiment [Figure 4D] Flowchart regarding the operation of the imaging control device 100 according to the first embodiment [Figure 5A] A flowchart showing the operation of the imaging control device 100 according to the second embodiment. [Figure 5B] A flowchart showing the operation of the imaging control device 100 according to the second embodiment. [Figure 5C] A flowchart showing the operation of the imaging control device 100 according to the second embodiment. [Figure 5D] A flowchart showing the operation of the imaging control device 100 according to the second embodiment. [Figure 6] FIG. 1 is a diagram showing an example of a state of shooting using an imaging system according to an embodiment; [Figure 7] FIG. 13 is a diagram showing an example of the transition of the role of the sub-camera in the second embodiment; [Figure 8] FIG. 1 is a diagram illustrating the main operations and signal flows of an imaging control device 100 according to an embodiment. [Figure 9] FIG. 13 is a diagram showing an example of roles and control contents that can be set for a sub-camera in the third embodiment. [Figure 10] Flowchart of role determination process in the third embodiment [Figure 11] 11 is a flowchart showing the operation of each device in the imaging system according to the third embodiment. [Figure 12] FIG. 1 is a diagram for explaining coordinate conversion in an embodiment. [Figure 13] FIG. 1 is a diagram relating to subject detection and coordinate transformation in an embodiment. [Figure 14] Schematic diagram of operational control of a sub-camera in the third embodiment. [Figure 15] FIG. 13 is a schematic diagram of another operational control of the sub camera in the third embodiment; [Figure 16] FIG. 1 is a diagram for explaining pan value calculation in an embodiment; [Figure 17] FIG. 1 is a diagram for explaining tilt value calculation in an embodiment; [Figure 18] FIG. 13 is a diagram showing an example of mapping of zoom values ​​between a main camera and a sub camera in an embodiment; [Figure 19] Flowchart for determining the control content according to the role of the sub-camera in the third embodiment [Figure 20] Schematic diagram of control according to the role of the sub-camera in the third embodiment. [Figure 21] Schematic diagram of control according to the role of the sub-camera in the third embodiment. [Figure 22] FIG. 13 is a schematic diagram of a tracking operation of a plurality of subjects in a modified example of the third embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] <First embodiment> (Overview of the multi-camera imaging system) 1 is a schematic diagram showing an example of the configuration of a multi-camera imaging system 10 (hereinafter simply referred to as an imaging system) according to this embodiment. The imaging system 10 has multiple cameras 300, 500, 600, 700, 800, 900, an imaging control device 100, and a role control device 400. The multiple cameras 300, 500, 600, 700, 800, 900, the imaging control device 100, and the role control device 600 are communicatively connected via a communication network 200.

[0011] The communication network 200 complies with known wired or wireless communication standards such as the IEEE802.3 series, IEEE802.11 series, etc. Each of the cameras 300, 500, 600, 700, 800, 900, the shooting control device 100, and the role control device 600 has a communication interface that complies with the standard of the communication network 700.

[0012] Camera 300 captures the entirety of a predetermined shooting range. The shooting range is set as a range in which a subject to be shot may exist in a studio, for example. Therefore, the video of camera 300 captures all of the subjects within the shooting range.

[0013] The purpose of the camera 300 is to capture an image for detecting a subject of the shooting target existing within the shooting range. Therefore, the shooting direction and angle of view of the camera 300 are determined according to the position of the camera 300 and the shooting range, and are basically fixed during shooting. Furthermore, the focusing distance may also be basically fixed in order to obtain a pan-focus image of the shooting range. Moreover, it is preferable that the camera 300 captures the entire shooting range so that it is not hidden by objects outside the shooting range. Therefore, the camera 300 is installed in a position where it overlooks the entire shooting range. In order to distinguish the camera 300 from the other cameras 400 and 500 whose shooting direction and angle of view during shooting are not basically fixed, the camera 300 will be referred to as an overhead camera below. However, the installation position of the camera 300 is not limited to a position where it overlooks the shooting range. The operation of the overhead camera 300 can be controlled by the shooting control device 100.

[0014] The cameras 500, 600, 700, 800, and 900 (hereinafter referred to as cameras 500 to 900) are, for example, PTZ cameras, and their operations including the shooting direction (pan and tilt angles) and angle of view (zoom) can be controlled from an external device. The overhead camera 300 may also be a PTZ camera. Here, it is assumed that the operation of the camera 500 is controlled by a user of the imaging system, and the operations of the remaining cameras 600 to 900 are controlled by the imaging control device 100. In the following, since the imaging control device 100 controls the operations of the cameras 600 to 900 based on the state of the camera 500, the camera 500 is referred to as the main camera, and the cameras 600 to 900 are referred to as the sub-cameras.

[0015] In the second embodiment described later, the main camera 500 is not essential. In the absence of the main camera 500, the shooting direction and angle of view of each of the sub-cameras 600 to 900 are controlled by the shooting control device 100 according to control content that is associated with the role set by the role control device 400 and does not depend on the main camera.

[0016] The number of sub-cameras is arbitrary and may be more or less than four, but is set to two or more. The main camera 500 may be directly operated by the user. The cameras 500 to 900 may be configured so that the shooting direction (pan and tilt angles) can be controlled by attaching the camera body to a camera platform. The cameras 500 to 900 may be configured so that a zoomable interchangeable lens is attached to the camera body.

[0017] Here, in this embodiment, it is assumed that the role control device 400 has an operator. The shooting control device 100 may also have an operator (user), but this is not essential. The operator of the role control device 400 may be the user of the shooting control device 100. Since the shooting of the overhead camera 300 and the sub camera 400 is controlled by the shooting control device 100, a photographer is not required. It is assumed that the main camera 500 has an operator or photographer. In this way, a configuration that does not require operators or photographers for some devices can achieve labor saving.

[0018] 1 shows that all signals are communicated through the communication network 200, but for example, the video signal and the control signal may be communicated in a different manner. For example, each of the multiple cameras 300, 500 to 900 may directly supply the video signal to the shooting control device 100 via a cable. The cameras 300, 500 to 900 and the shooting control device 100 have a communication circuit according to the standard of the video signal. The standard of the video signal is, for example, the SDI (Serial Digital Interface) standard or the HDMI (High-Definition Multimedia Interface) (registered trademark), but is not limited to these.

[0019] The photography control device 100 detects a subject from a video signal received from the overhead camera 300. The photography control device 100 determines the photography direction and angle of view of the sub-camera based on the subject detection result, the state of the main camera 500, and the role set for the sub-camera. The photography control device 100 transmits a control command including the determined photography direction and angle of view to the sub-cameras 600-900. By changing the role setting, the method of determining the photography direction and angle of view of the sub-cameras 600-900 can be changed, and the degree of freedom in controlling the operation of the sub-cameras 600-900 can be increased.

[0020] The role control device 400 sets a role and a priority order for each of the sub-cameras 600 to 900. As described later, each role is associated with a different control content, and has a name that represents the control content. The priority order for each of the sub-cameras 600 to 900 is set in advance in consideration of the shooting scene, the role to be set for the sub-camera, the shooting position, etc., but can be changed by the user at any time. The priority order can also be automatically changed depending on the shooting situation, etc.

[0021] When imaging system 10 is used for live distribution or the like, a switcher for selecting and outputting one of the images captured by main camera 500 and sub-cameras 600 to 900 is further connected to network 200.

[0022] The switcher is a selection device that receives video signals from the main camera 500 and each of the sub-cameras 600 to 900, selects a video from one sub-camera designated by the user, and outputs the video as a main line video.

[0023] The switcher also transmits a control signal (tally information) for performing a tally display on the camera capturing the main line video, via the network 200. By acquiring the tally information via the network 200, the shooting control device 100 can know which video of the main camera 500 or the sub cameras 600 to 900 has been selected as the main line video.

[0024] The tally information is, for example, information in which each of the main camera 500 and the sub cameras 600 to 900 is associated with one of "streaming," "previewing," and "waiting." "Streaming" and "previewing" are each associated with one camera, and the remaining cameras are associated with "waiting." "Streaming" is associated with the camera capturing the main line video. "Previewing" is associated with the camera capturing the video that is to be selected next as the main line video. Note that it is sufficient for the tally information to at least identify which camera "streaming" is associated with.

[0025] The CPU (described below) of a camera associated with "Streaming" in the tally information can display a tally, for example by turning on a red tally lamp. Note that a tally lamp is not turned on in a camera associated with "Streaming" in the tally information. Also, the CPU of a camera associated with "Previewing" in the tally information can display a tally, for example by turning on a tally lamp of a color different from "Streaming" (for example, green).

[0026] (Example of functional configuration of each device) Fig. 2 is a block diagram showing an example of the functional configuration of each device constituting the multi-camera imaging system shown in Fig. 1. The configuration expressed as a functional block in the drawing can be realized by an integrated circuit such as an ASIC or an FPGA, by a discrete circuit, or by a combination of a memory and a processor that executes a program stored in the memory. Also, one functional block may be realized by multiple integrated circuit packages, or multiple functional blocks may be realized by one integrated circuit package. Also, the same functional block may be implemented in different configurations depending on the operating environment, required capabilities, etc.

[0027] (Photography control device 100) First, a functional configuration example of the imaging control device 100 will be described. The imaging control device 100 may be a general-purpose computer device such as a personal computer or a workstation. The imaging control device 100 has a configuration in which a CPU 101, a RAM 102, a ROM 103, an inference unit 104, a network interface (I / F) 105, a user input unit 106, and a display unit 108 are mutually connected via an internal bus 110.

[0028] The CPU 101 is a microprocessor capable of executing programmed instructions. The CPU 101, for example, loads a program stored in a ROM 103 into a RAM 102 and executes the program to realize the functions of the imaging control device 100, which will be described later. The CPU 101, for example, can realize the functions of the imaging control device 100 by executing an imaging control application that runs on an operating system (OS).

[0029] The RAM 102 is used to load programs executed by the CPU 101 and to temporarily store data to be processed by the CPU 101, data currently being processed, etc. Furthermore, a part of the RAM 102 may be used as a video memory for the display unit 108.

[0030] The ROM 103 is a rewritable non-volatile memory, and stores the programs (OS and applications) executed by the CPU 101, user data, and the like.

[0031] The inference unit 104 executes a detection process of a subject area using a machine learning model for the image of the overhead camera 300. The inference unit 104 can be implemented using a hardware circuit capable of executing the calculation of the machine learning model at high speed, such as a GPU (Graphics Processing Unit) or an NPU (Neural network Processing Unit). Alternatively, the inference unit 104 may be implemented using a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array). The CPU 101 may execute a program to realize the function of the inference unit 104.

[0032] The machine learning model may be a convolutional neural network (CNN) trained according to the type of subject to be detected. Here, the inference unit 104 detects a human body region or a human face region as a subject region from an input image. The inference unit 104 outputs the position and size of a rectangular region inscribed in the subject region and the detection reliability for each detected subject region. Note that, a plurality of types of machine learning models may be used to execute detection processing of different types of subject regions for the same input image. Note that the inference unit 104 may execute detection processing of the subject region by a known method that does not use a machine learning model. The inference unit 104 can detect the subject region by using, for example, a method using local features such as SIFT or SURF, or a method using pattern matching.

[0033] The network I / F 105 is an interface for connecting the photography control device 100 to the communication network 200. The photography control device 100 (CPU 101) can communicate with external devices on the communication network 200, such as the overhead camera 300, the role control device 400, the main camera 500, and the sub-camera 600, through the network I / F 105. The photography control device 100 may communicate with external devices through other communication interfaces (USB, Bluetooth (registered trademark), etc.) not shown.

[0034] To communicate with each device (overhead camera 300, sub camera 600, main camera 500, role control device 400) on communication network 200, CPU 101 acquires the network address of each device at any timing and stores it in RAM 102. CPU 101 also acquires information about each device (device type, model name, etc.) at any timing (for example, at the time of the first communication) and stores it in RAM 102. In this way, CPU 101 is assumed to know at least the identification information and device type of overhead camera 300, sub camera 600, main camera 500, and role control device 400. Note that the user may be allowed to give any name to each device.

[0035] The user input unit 106 is an input device (not shown) such as a mouse, a keyboard, a touch panel, etc. The imaging control device 100 receives instructions from a user through the user input unit 106.

[0036] The display unit 108 is a display device such as a liquid crystal display (LCD), etc. The display unit 108 displays a GUI screen provided by the OS, a shooting control application, etc.

[0037] (Overhead camera 300) Next, an example of the functional configuration of the overhead camera 300 will be described. The CPU 301 is a microprocessor capable of executing programmed instructions. For example, the CPU 301 loads a program stored in the ROM 303 into the RAM 302 and executes the program to control the operation of each functional block and realize the function of the overhead camera 300, which will be described later.

[0038] The RAM 302 is used to load programs executed by the CPU 301 and to temporarily store data to be processed by the CPU 301, data being processed, etc. The RAM 302 may also be used as a buffer for video signals obtained by shooting.

[0039] The ROM 303 is a rewritable non-volatile memory. The ROM 303 stores the programs executed by the CPU 301, the setting values ​​of the overhead camera 300, user data, etc. The ROM 303 can also be used as a recording destination for video signals. The ROM 303 may include an internal memory and a removable memory card.

[0040] The image sensor 307 has a photographing optical system and an image sensor. The image sensor may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The image sensor has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from each pixel. Each pixel accumulates electric charge according to the amount of incident light by photoelectric conversion. A pixel signal group (analog image signal) representing the subject image formed on the imaging surface is obtained by reading out from each pixel a signal having a voltage according to the amount of electric charge accumulated during the exposure period.

[0041] The image processing unit 306 applies predetermined signal processing and image processing to the analog image signal output by the image sensor 307, generates signals and image data according to the application, and acquires and / or generates various types of information.

[0042] The processes applied by the image processing unit 306 can include, for example, pre-processing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effect processing. Pre-processing may include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on the image sensor 307, and is processing for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging optical system (image restoration), correction of the effect of peripheral light falloff of the imaging optical system, color correction, and the like. The data processing may include processes such as cutting out (trimming), compositing, scaling, encoding and decoding, generating header information (generating a data file), etc. The data processing also includes the generation of a video signal to be output to the outside and video data to be recorded in the ROM 303. The evaluation value calculation process may include processes such as generation of a signal and evaluation value used in automatic focus detection (AF), generation of an evaluation value used in automatic exposure control (AE), etc. AF and AE are executed by the CPU 301. The special effects processing may include adding a blur effect, changing color tones, relighting, and the like. Note that these are merely examples of processes that the image processing unit 306 can apply, and do not limit the processes that the image processing unit 306 can apply. The image processing unit 306 outputs the acquired or generated information and data to the CPU 301, RAM 302, etc. depending on the purpose.

[0043] The type and settings of the processing applied by the image processing unit 306 can be controlled by sending commands from the photography control device 100 to the overhead camera 300.

[0044] The network I / F 305 is an interface for connecting the overhead camera 300 to the communication network 200. The overhead camera 300 (CPU 301) can communicate with external devices on the communication network 200, such as the shooting control device 100, the sub camera 600, the main camera 500, and the role control device 400, via the network I / F 305. The overhead camera 300 may communicate with external devices via other communication interfaces (USB, Bluetooth, etc.) not shown.

[0045] (Sub camera 600) Next, an example of the functional configuration of the sub-camera 600 will be described. It is assumed that functional blocks with the same names in the sub-camera 600 and the overhead camera 300 have the same functions, and their descriptions will be omitted. It is also assumed that the sub-cameras 700, 800, and 900 have the same functional configuration as the sub-camera 600, and their individual descriptions will be omitted.

[0046] As described above, the sub-camera 600 is a PTZ camera, and the shooting direction and angle of view can be controlled externally. Therefore, the sub-camera 600 has a driving unit 609 capable of panning, tilting, and zooming, and a driving I / F 608. The driving I / F 608 is a communication interface between the driving unit 609 and the CPU 601.

[0047] The driving unit 609 has a pan / tilt mechanism that supports the sub-camera 600 so that it can pan and tilt, a zoom mechanism that changes the angle of view of the photographing optical system, and a motor that drives these mechanisms. The zoom mechanism may use the image enlargement / reduction by the image processing unit 606. The driving unit 609 drives the motor according to instructions received from the CPU 601 via the driving I / F 608, and adjusts the optical axis direction and the angle of view of the photographing optical system.

[0048] The recording unit 612 is, for example, a non-volatile memory, and is capable of recording image data generated by the image processing unit 606. The recording unit 612 may be built-in or may be detachable. The user input unit 613 is a general term for an input device for a user to input instructions and data to the sub camera 600. The user input unit 613 may include a button, a key, a dial, and a touch panel (not shown). The CPU 601 executes an operation according to the user input through the user input unit 613.

[0049] (Main camera 500) Next, an example of the functional configuration of the main camera 500 will be described. It is assumed that functional blocks with the same names in the main camera 500 and the sub camera 600 have the same functions, and their description will be omitted. The main camera 500 is operated by a user. Here, it is assumed that the user remotely controls the main camera 500 by transmitting commands through the communication network 200. However, when the main camera 500 is not a PTZ camera, the user may directly operate the main camera 500.

[0050] The photography control device 100 (CPU 101) can obtain information on the photography direction and angle of view of the sub-cameras 600-900 and the main camera 500 from the sub-cameras 600-900 and the main camera 500 via the network I / F 505. The photography direction may be the pan and tilt angles of the drive units 609 and 509 with a predetermined reference direction set to 0°. The reference direction may be the direction directly facing the photography range.

[0051] (Role control device 400) Next, an example of the functional configuration of the role control device 400 will be described. The CPU 401 is a microprocessor capable of executing programmed instructions. For example, the CPU 401 loads a role setting program stored in the ROM 403 into the RAM 402 and executes the program to control the operation of each functional block and realize the functions of the role control device 400.

[0052] The RAM 402 is used to load programs executed by the CPU 401 and to temporarily store data to be processed by the CPU 401, data currently being processed, etc. In addition, a part of the RAM 402 may be used as a video memory for the display unit 408.

[0053] The ROM 403 is a rewritable non-volatile memory, and stores the programs executed by the CPU 401, the setting values ​​of the role control device 400, user data, and the like.

[0054] The user input unit 411 is an input device such as a button, a dial, a joystick, a touch panel, etc. The role control device 400 receives, via the user input unit 411, a user's instruction regarding the setting of the roles of the sub-cameras 600-900.

[0055] The network I / F 405 is an interface for connecting the role control device 400 to the communication network 200. The role control device 400 (CPU 401) can communicate with external devices on the communication network 200, such as the overhead camera 300, the main camera 500, the sub-cameras 600-900, and the shooting control device 100, through the network I / F 405. The role control device 400 may communicate with external devices through other communication interfaces (USB, Bluetooth, etc.) not shown.

[0056] The display unit 408 is a display device such as a liquid crystal display (LCD), etc. The display unit 408 displays a GUI screen provided by the OS, a role setting application, etc.

[0057] The role control device 400 stores role setting information in, for example, the ROM 403. The role setting information is information in which identification information of the sub-cameras 600 to 900 is associated with information indicating the role that has been set. The CPU 401 executes a role setting application to display a role setting screen on the display unit 408. For example, the identification information of the sub-cameras 600 to 900 (network address, name set by the user, etc.) and the name of the currently set role are displayed in association with each other on the role setting screen. The initial value of the currently set role may be a default role that has been set in advance. The user can change the current role that is displayed in association with the desired sub-camera 600 to 900 by operating the user input unit 411.

[0058] When a user operation indicating the end of the setting operation, such as an operation of an OK button included in the role setting screen, is detected, the CPU 401 updates the role setting information stored in the ROM 403 in accordance with the content of the role setting screen.

[0059] When the CPU 401 receives a role acquisition command via the network I / F 405, it reads out the role setting information stored in the ROM 403 and transmits it to the sender of the role acquisition command.

[0060] 1 and 2, the role control device 400 is described as an independent device, but for example, a shooting control application executed by the shooting control device 100 may provide the same function as the role control device 400. Furthermore, roles may be directly set to the sub-cameras 600-900, and the shooting control device 100 may acquire the roles assigned to the sub-cameras 600-900 from the sub-cameras 600-900.

[0061] The role that can be set for the sub-cameras 600 to 900 is a predetermined way of using information obtained from the main camera 500 to control the operation of the sub-cameras 600 to 900. As an example, it is assumed here that information from the main camera is used to control the tracking subject and zoom operation of the sub-cameras 600 to 900.

[0062] Fig. 3 shows an example of types of roles that can be set for the sub-cameras 600 to 900 and control contents associated with the roles. The control contents for each role can be stored in the ROM 403 of the role control device 400 and the ROM 103 of the shooting control device 100 in the table format shown in Fig. 3, for example. Here, it is assumed that any of "main follow", "main counter", "assist follow", and "assist counter" can be set as the role ROLE. A role can be set for each sub-camera.

[0063] For a sub-camera with the role "main follow", the photography control device 100 (CPU 101) sets the same tracking subject as the main camera 500, and when the main camera 500 is zoomed, the photography control device 100 (CPU 101) performs zoom control of the sub-camera in the same phase. Here, "in phase" means that the zoom direction (telephoto direction or wide-angle direction) is the same, that is, the direction of the angle of view change is the same. On the other hand, "opposite phase" means that the zoom direction (telephoto direction or wide-angle direction) is the opposite direction, that is, the direction of the angle of view change is the opposite direction. Note that even if the zoom direction is in the same phase, the angle of view does not have to be the same as the main camera 500, and whether in the same phase or opposite phase, the degree of zoom change (such as the speed or rate of change) does not have to be the same as the main camera 500.

[0064] For the sub-camera with the role of "main counter", the photography control device 100 (CPU 101) sets the same tracking subject as the main camera 500, and when the main camera 500 is zoomed, the photography control device 100 (CPU 101) controls the sub-camera with this role to zoom down when the main camera 500 is zoomed up. Note that zooming up refers to changing the zoom in the telephoto direction (telephoto end direction), and zooming down refers to changing the zoom in the wide-angle direction (wide end direction). When zoom control is performed by the image processing unit 606, zooming up refers to making the area cut out from the image smaller and the magnification ratio of the cut out area larger than before the area change. On the other hand, zooming down refers to making the area cut out from the image larger and the magnification ratio of the cut out area smaller than before the area change.

[0065] For the sub-cameras with the ROLE of "assist follow L" and "assist follow R", the photography control device 100 (CPU 101) sets a tracking subject different from that of the main camera 500. In addition, when the main camera 500 is zoomed, the photography control device 100 also performs zoom control of the sub-cameras in the same phase.

[0066] For the sub-cameras whose ROLEs are "assist counter L" and "assist counter R", the photography control device 100 (CPU 101) sets a tracking subject different from that of the main camera 500. In addition, when the main camera 500 is zoomed, the photography control device 100 performs zoom control in the opposite phase on the sub-cameras.

[0067] Here, for the sub-cameras with ROLES of "Assist Follow L" and "Assist Counter L", a subject on the left side (left edge) of the image other than the subject of interest of the main camera 500 is set as the tracking subject of the sub-camera. Also, for the sub-cameras with ROLES of "Assist Follow R" and "Assist Counter R", a subject on the right side (right edge) of the image other than the subject of interest of the main camera 500 is set as the tracking subject of the sub-camera.

[0068] The tracking subject of the sub camera may be set according to other conditions. For example, among the subjects other than the target subject of the main camera 500 in the image, the subject existing on the upper side (upper end) or the lower side (lower end) may be set as the tracking subject of the sub camera. Alternatively, among the subjects other than the target subject of the main camera 500, the subject existing on the front or back may be set as the tracking subject of the sub camera.

[0069] For the sub-cameras with the roles of "exposure shift" and "aperture counter", the photography control device 100 (CPU 101) sets the same tracking subject as the main camera 500. Furthermore, when the main camera 500 is zoomed, the photography control device 100 performs zoom control of the sub-cameras in the same phase. Furthermore, the photography control device 100 determines parameters that define the photography conditions based on the photography state of the main camera 500. Details will be described later.

[0070] For the sub-camera with the role "wide angle", the photography control device 100 (CPU 101) sets the same tracking subject as the main camera 500. When the main camera 500 is zoomed, the photography control device 100 performs zoom control of the sub-camera in the same phase. Furthermore, the photography control device 100 determines parameters that determine the angle of view and photography conditions of the sub-camera based on the photography state of the main camera 500. Details will be described later.

[0071] Here, the control contents associated with the roles are shown as setting of the subject to be tracked and zoom control, but it is also possible to execute only one of them, or to add another control item.

[0072] In the role setting information stored in the ROM 403 by the role control device 400, information indicating the role ROLE (such as the above-mentioned type name or the number assigned to the type) is associated with the identification information of the sub-cameras 600-900. The CPU 101 of the photography control device 100 acquires the role setting information from the role control device 400, and executes operation control of the sub-cameras 600-900 according to the type of role ROLE set to the sub-cameras 600-900.

[0073] When the role settings of the sub-cameras 600 to 900 are changed, the role control device 400 may notify an external device (for example, the shooting control device 100). This allows the change in the role settings to be immediately reflected in the operation control of the sub-cameras 600 to 900.

[0074] 6 is a schematic diagram showing an example of a scene captured by the imaging system 10, and the arrangement, shooting direction, and angle of view of each camera. The direction of each camera indicates the shooting direction in a horizontal plane, and the dashed lines show the shooting angle of view. The numbers 1 to 4 attached to the sub-cameras 600 to 900 indicate the priority order, and the smaller the number, the higher the priority order.

[0075] There are three human subjects (subjects A to C) in the shooting scene. Furthermore, main camera 500 shoots only subject B among subjects A to C. Therefore, the target subject of main camera 500 is subject B.

[0076] The role of "main counter" is set to the sub camera 600. Therefore, the photography control device 100 controls the photography direction of the sub camera 600 so that it tracks the subject of interest (subject B) of the main camera 500. Furthermore, when the angle of view of the main camera 500 changes to the telephoto side, the photography control device 100 changes the angle of view of the sub camera 600 to the wide-angle side, and when the angle of view of the main camera 500 changes to the wide-angle side, the photography control device 100 changes the angle of view of the sub camera 600 to the telephoto side.

[0077] The role of "assist follow R" is set for the sub camera 700. Therefore, the photography control device 100 controls the photography direction of the sub camera 600 so that it tracks subject C, which is located on the right side as seen from the camera, out of subjects A and C other than the subject of interest (subject B) of the main camera 500. In addition, when the angle of view of the main camera 500 changes to the telephoto side, the photography control device 100 also changes the angle of view of the sub camera 600 to the telephoto side, and when the angle of view of the main camera 500 changes to the wide-angle side, the photography control device 100 also changes the angle of view of the sub camera 600 to the wide-angle side.

[0078] The role of "assist follow L" is set for the sub camera 800. Therefore, the photography control device 100 controls the photography direction of the sub camera 600 so that it tracks the subject A located on the left side as seen from the camera, out of subjects A and C other than the subject of interest (subject B) of the main camera 500. In addition, the photography control device 100 changes the angle of view of the sub camera 600 to the telephoto side when the angle of view of the main camera 500 changes to the telephoto side, and changes the angle of view of the sub camera 600 to the wide angle side when the angle of view of the main camera 500 changes to the wide angle side.

[0079] The role of "assist counter R" is set for the sub camera 900. Therefore, the photography control device 100 controls the photography direction of the sub camera 600 so that it tracks the subject C located on the right side as seen from the camera, out of subjects A and C other than the subject of interest (subject B) of the main camera 500. In addition, the photography control device 100 changes the angle of view of the sub camera 600 to the wide angle side when the angle of view of the main camera 500 changes to the telephoto side, and changes the angle of view of the sub camera 600 to the telephoto side when the angle of view of the main camera 500 changes to the wide angle side.

[0080] In this embodiment, the shooting control device 100 can control the sub-camera to replace the main camera 500. Therefore, even if the intended image is not obtained by the main camera 500 or it is considered highly likely that the intended image will not be obtained in the near future, it is possible to continuously obtain an image similar or similar to the image that the main camera 500 should shoot.

[0081] The operation of the imaging control device 100 according to this embodiment will be described with reference to the flowcharts shown in Figures 4A to 4D. The operation of the imaging control device 100 described below is realized by the CPU 101 executing a program stored in the ROM 103 or acquired via the network I / F 105 and controlling each unit of the imaging control device 100.

[0082] FIG. 4A is a flowchart showing the overall operation of the imaging control device 100 according to this embodiment. In S4101, CPU 101 determines whether or not shooting has ended, and if it is determined that shooting has ended, the following operation is not executed, and if not, S4102 is executed. CPU 101 determines that shooting has ended, for example, when it receives a notification of the end of shooting from main camera 500 via network I / F 105. CPU 101 also determines that shooting has ended when it is determined that the power of main camera 500 is off. CPU 101 determines that the power of main camera 500 is off, for example, when no notification from main camera 500 is received via network I / F 105 for a certain period of time or more.

[0083] In S4102, CPU 101 determines (selects) a sub-camera to be subordinated to main camera 500 from sub-cameras 600 to 900. Details of S4102 will be described later with reference to Fig. 4B. A sub-camera to be subordinated to main camera 500 is a sub-camera that is operated as a backup for main camera 500.

[0084] In S4103, CPU 101 determines an appropriate role for the sub-cameras to be subordinate to main camera 500 in order to back up main camera 500. Details of S4103 will be described later with reference to FIG. 4C.

[0085] In S4104, the CPU 101 executes control based on the roles of the sub-cameras 600 to 900. After executing S4104, the CPU 101 executes S4101 again.

[0086] (Determining which sub-camera to subordinate to the main camera) Next, the operation in S4102 in FIG. 4A will be described in detail with reference to the flowchart shown in FIG. 4B. In S4201, CPU 101 determines whether or not a condition for temporarily releasing the subordination to main camera 500 is met. The condition is assumed to be predetermined and stored in ROM 103, such as receiving a notification of an operation on user input unit 512 from main camera 500 or receiving a release command from role instruction device 400. If CPU 101 determines that the condition for temporarily releasing the subordination to main camera 500 is met, it executes S4202, and if not, it executes S4203.

[0087] In S4202, the CPU 101 turns ON (for example, sets the value to 1) a temporary release flag that is set in a specific address of the RAM 102, for example. Then, the CPU 101 executes S4203.

[0088] In S4203, CPU 101 refers to the temporary release flag in RAM 102 and determines whether or not the subordination to the main camera is being temporarily released. If the temporary release flag is ON, CPU 101 determines that the subordination is being temporarily released and executes S4204, and if the temporary release flag is OFF, it determines that the subordination is not being temporarily released and executes S4206.

[0089] In S4204, CPU 101 determines whether the shooting state of main camera 500 has changed, and if it is determined that the shooting state has changed, executes S4205, and if not, ends the processing of the flowchart in Fig. 4B. For example, CPU 101 determines that the shooting state of main camera 500 has changed when notified from main camera 500 of a change in the body settings or shooting direction (direction of the optical axis of the shooting optical system), or a change in camera data not linked to settings such as vibration information. It is assumed that the determination conditions used in S4204 are stored in ROM 103 in advance.

[0090] In S4205, the CPU 101 sets the temporary release flag in the RAM 102 to OFF (for example, the value is 0). Then, the CPU 101 executes S4206.

[0091] In S4206, CPU 101 acquires the shooting state of main camera 500 and stores it in RAM 102. The shooting state includes the video (frame image) captured by main camera 500, the result of subject detection processing on the video, the shooting direction (for example, pan and tilt angles of drive unit 509), the angle of view, exposure conditions, white balance coefficient, AE and AF evaluation values, etc. The shooting conditions may be acquired from main camera 500, or at least a part of the information may be acquired by shooting control device 100 based on the video received from main camera 500.

[0092] In S4207, CPU 101 determines whether or not it is necessary to determine a sub-camera to be subordinated to main camera 500, and if it is determined that it is necessary, executes S4209, and if not, executes S4208.

[0093] For example, the CPU 101 Luminance information detected from the image of the main camera 500 The size and position of the subject area (or a rectangular area circumscribing the subject area) detected from the image of the main camera 500 Exposure conditions of the main camera 500 (one or more of sensitivity, ND filter value, shutter speed, and aperture value), focusing distance, and focal length of the photographing optical system, · Main camera 500 movement The determination may be made based on a condition based on one or more of the following:

[0094] More specifically, (1) When the size of a region of a video (frame) that is overexposed or close to being overexposed (for example, the number of pixels whose brightness value is equal to or exceeds an upper threshold) is equal to or exceeds a threshold for a predetermined period of time, (2) When the size of a region of a video (frame) that is blacked out or close to being blacked out (for example, the number of pixels whose brightness value is equal to or less than a lower threshold) continues to be equal to or greater than a threshold for a predetermined period of time, (3) When the moving speed of the target subject (area) of the main camera 500 exceeds a threshold value, (4) When it is determined that there is a high possibility that the target subject (area) of the main camera 500 will go out of frame based on the position, movement direction, and speed of the target subject, (5) When the size of the target subject of the main camera 500 exceeds the upper threshold or falls below the lower threshold, or when it is determined that there is a high possibility that the size will exceed the upper threshold or fall below the lower threshold based on a change in size over time, (6) When the depth of field of the target subject of the main camera 500 is less than the threshold value (the depth of field can be calculated based on the distance of the target subject (=focus distance) and the aperture value) (7) If the movement (e.g., angular velocity and / or acceleration) of the main camera 500 exceeds a threshold, When one or more of the above predetermined conditions are satisfied, CPU 101 can determine that it is necessary to determine a sub-camera to be subordinated to main camera 500. Note that the conditions exemplified here are not essential, and other conditions may be used.

[0095] In (1), the condition of continuing for a predetermined time is included in order to exclude temporary whiteout caused by flash used in stage lighting, etc. Note that (1) and (2) may be a ratio to the number of pixels in the entire frame or a ratio to the number of pixels in the subject area. If one or more of (3) and (7) are satisfied, there is a high possibility that the blur of the target subject in the video is large. In (4), for example, if the shortest distance from the position of the target subject area to any end of the frame image is equal to or less than a threshold, the position of the target subject is too far from the center of the frame image. Also, if the value obtained by dividing the distance to the end of the frame image in the moving direction of the target subject area by the moving speed of the area is equal to or less than a threshold, there is a high possibility that the target subject will go out of frame. Note that an example of a method for identifying the target subject of the main camera 500 will be described in the third embodiment. If (6) is satisfied, the sharpness of the target subject is likely to become unstable.

[0096] Thus, in S4207, CPU101: When the image quality of the target subject photographed by the main camera 500 is low or is expected to become low, When the subject of interest cannot be captured in the correct position and size on the screen, or when it is expected that this will not be possible If it is determined that at least one of the above applies, it is determined that it is necessary to determine a sub-camera to be subordinated to the main camera 500.

[0097] In S4208, CPU 101 deletes the slave setting for the sub-cameras among sub-cameras 600 to 900 that are set to be slaved to main camera 500 (slave setting). CPU 101 deletes information on the slaved sub-cameras stored in, for example, RAM 102. After that, CPU 101 ends the process of the flowchart in FIG. 4B.

[0098] In S4209, the CPU 101 obtains information regarding each of the sub cameras 600 to 900, and stores it in the RAM . The information obtained here is - Information about the optical system (focal length range, minimum and maximum aperture), Drive unit information (pan and tilt range, current angle of view), - the type and priority of roles that have been set; Installation position (e.g. three-dimensional coordinates in a coordinate system having a predefined origin), The current shooting direction (which can be the pan and tilt angles of the drive), Image sensor information (number of pixels, sensor size, color filter type, etc.), It should be noted that not all of these are required, and other information may be included.

[0099] The information may be the one most recently acquired and stored in RAM 102, or may be acquired from the sub-camera each time. The role and priority set for the sub-camera may be acquired from the role control device 400, or may be the one most recently acquired and stored in RAM 102. Information regarding the installation position of the sub-camera, which is registered via the user input unit 106 when the sub-camera is installed, may be stored in ROM 103. The position of the sub-camera measured by an external device using a beacon or the like attached to the sub-camera may be acquired.

[0100] In S4210, CPU 101 determines whether any of sub cameras 600 to 900 is capturing main line video or not, and if it is determined that any of sub cameras 600 to 900 is capturing main line video, executes S4211, and if not, executes S4213. CPU 101 can make this determination based on tally information obtained from the switcher via network I / F 105.

[0101] In S4211, CPU 101 determines whether or not the sub camera capturing the main line video is the same as the sub camera set as a slave, and if it is determined that they are the same, executes S4213, and if not, executes S4212. CPU 101 can make the determination by referring to information stored in RAM 102.

[0102] In S4212, CPU 101 excludes the sub-camera capturing the main line video (sub-camera set as subordinate) from the sub-cameras to be searched. CPU 101 can associate information indicating that the sub-camera is not to be searched with the information of the corresponding sub-camera among the information of the sub-cameras stored in RAM 102, for example.

[0103] In S4213, CPU 101 refers to the information of the sub-cameras stored in RAM 102 and judges whether or not there is a sub-camera that has been set as a candidate to be subordinated to the main camera among the sub-cameras to be searched. If CPU 101 judges that there is a sub-camera that has been set as a candidate, CPU 101 executes S4214, otherwise executes S4215. Note that it is optional whether or not to set the sub-cameras as candidates in advance. For example, when installing the sub-cameras 600 to 900, if the sub-cameras 600 to 900 suitable for being subordinated to the main camera 500 can be specified, the sub-cameras as candidates are set in advance. The setting can be executed through the user input I / F unit 106.

[0104] In S4214, CPU 101 sets the sub camera that has already been set as a candidate as a slave. Specifically, CPU 101 adds information indicating that the sub camera has been set as a slave to the information of the corresponding sub camera among the information of the sub cameras stored in RAM 102. Then, CPU 101 ends the operation of FIG. 4B.

[0105] In S4215, CPU 101 determines whether or not there is a sub-camera capable of shooting with a composition similar to that of main camera 500, and if it is determined that there is a sub-camera, executes S4216, and if not, executes S4217.

[0106] Here, the sub-camera capable of shooting with a composition similar to that of the main camera 500 is The main camera 500 is capable of capturing an image of a range including all of the subjects to be captured, and These subjects can be photographed at a size that is less than a threshold value, compared to the size in the image captured by the main camera 500. The CPU 101 can determine a sub-camera that can capture an image with a similar composition to that of the main camera 500, taking into consideration the installation position of the camera, the focal length range of the photographing optical system, the movable range of the drive unit, and the like.

[0107] Or, more simply, The installation position is closest to the main camera 500, and the distance from the main camera 500 is equal to or less than a threshold value, The focal length range of the photographing optical system overlaps with the focal length range of the photographing optical system of the main camera 500 by a threshold percentage or more. The sub-camera may be determined as a sub-camera capable of shooting with a composition similar to that of the main camera 500.

[0108] In S4216, CPU 101 sets the sub-camera determined in S4215 to be capable of shooting with a composition similar to that of main camera 500 as a slave. CPU 101 then ends the processing in Fig. 4B. Note that the number of sub-cameras to be set as a slave may be one or more, and there may be more than one.

[0109] In S4217, CPU 101 searches for the sub-camera with the lowest priority among the sub-cameras to be searched.

[0110] In S4218, CPU 101 sets the sub camera found in S4217 as a slave. Then, CPU 101 ends the process in FIG. 4B.

[0111] Note that in S4216 and S4218, the sub-camera may be set as subordinate after obtaining the user's consent. This is because automatic subordinate setting may not be in line with the user's intention. CPU 101 sets the sub-camera as subordinate if the user consents, and does not set the sub-camera as subordinate if the user consents. Furthermore, when obtaining consent, the user may be allowed to change (specify) the sub-camera to be set as subordinate. When a change instruction is given by the user, CPU 101 sets the sub-camera as subordinate to the sub-camera specified by the user.

[0112] (Role setting of sub camera subordinate to main camera) Next, the operation in S4103 in FIG. 4A will be described in detail with reference to the flowchart shown in FIG. 4C. In S4301, CPU 101 determines whether or not there is a sub-camera that is set as a slave by referring to the information on the sub-cameras stored in RAM 102. If it is determined that there is a sub-camera that is set as a slave, CPU 101 executes S4302, and if not, ends the operation in FIG. 4C.

[0113] In S4302, CPU 101 determines whether the size of the subject of interest in main camera 500 is equal to or less than a predetermined upper limit, and executes S4304 if it is determined to be equal to or less than the upper limit, and executes S4303 if it is not. Specifically, CPU 101 determines the area of ​​the subject of interest from the subject area detected in the video captured by main camera 500. Then, CPU 101 determines whether the size of the area of ​​the subject of interest (for example, in the case of a human subject, the size of the face area or the size of a rectangular area circumscribing the face area) is equal to or less than the upper limit. The upper limit may be the number of pixels, or may be a ratio to the entire image.

[0114] In S4303, CPU 101 sets the role of the sub-camera set as a subordinate camera to "wide angle," and ends the operation shown in FIG. 4C.

[0115] In S4304, CPU 101 determines whether the brightness value of the area of ​​the target object of main camera 500 is within a specified range, and if it is determined to be within the specified range, executes S4306, and if it is not, executes S4305. Specifically, CPU 101 determines whether the brightness evaluation value of the area of ​​the target object of main camera 500 is within a predetermined specified range. The brightness evaluation value may be, for example, an average brightness value or a maximum brightness value. In addition, the determination may be made not only for the target object but also for the areas of other objects of the same type. In this case, if there is even one object area whose brightness evaluation value is determined to be outside the specified range, S4305 is executed.

[0116] In S4305, CPU 101 sets the role of the sub camera set as a slave to "exposure shift", and ends the operation shown in FIG. 4C.

[0117] In S4306, CPU 101 judges whether the depth of field of the area of ​​the target object of main camera 500 is within a specified range, and if it is judged to be within the specified range, executes S4308, and if it is not, executes S4307. Specifically, CPU 101 calculates the depth of field of the area of ​​the target object of main camera 500 based on the object distance (focus distance) and the focal length and aperture value of the photographing optical system. Then, CPU 101 judges whether the depth of field is within a predetermined specified range. In addition, it may be judged not only for the target object but also for the areas of other objects of the same type. In this case, if there is at least one object area whose depth of field is judged to be outside the specified range, S4307 is executed.

[0118] In S4307, CPU 101 sets the role of the sub camera set as a slave to "aperture counter," and ends the operation shown in FIG. 4C.

[0119] In S4308, CPU 101 sets the role of the sub-camera that is set as a slave to "main follow", and ends the operation shown in FIG. 4C.

[0120] (Control instructions based on the role of the sub-camera) Next, the operation in S4104 in FIG. 4A will be described in detail with reference to the flowchart shown in FIG. 4D. In S4401, CPU 101 determines whether or not there is a sub-camera set as a slave, and if it is determined that there is a sub-camera, executes S4402, and if not, executes S4410. CPU 101 can make the determination by referring to the information on the sub-camera stored in RAM 102.

[0121] When there are multiple sub-cameras set as slaves, the CPU 101 executes the process from S4402 onwards for each sub-camera. In S4402, the CPU 101 receives role information of the sub-camera that has been set as a subordinate camera from the role control device 400, and stores it in the RAM 102. Note that it is not necessary to receive the role information at this timing. Role information that has been received in advance and stored in the RAM 102 may be referenced.

[0122] In S4403, CPU 101 identifies a subject of interest of main camera 500. If a subject of interest has already been identified in a previous process, that result may be used. In that case, this step does not need to be executed. As described in the third embodiment, a subject of interest can be identified based on the image and shooting direction of main camera 500.

[0123] In S4404, CPU 101 calculates the change instruction amount so that the sub-camera set as a slave has the same setting and shooting range as main camera 500. Specifically, CPU 101 acquires setting information from main camera 500 and calculates the change instruction amount based on the difference with the setting information of the sub-camera. CPU 101 also calculates the change instruction amount so that the pan, tilt, and zoom values ​​are such that the subject included in the shooting range of main camera 500 is shot by the sub-camera at the same size as the image of main camera 500. CPU 101 calculates the change instruction amount of the pan, tilt, and zoom values ​​of the sub-camera based on the position and size of the subject area detected from each image of main camera 500 and sub-camera, as well as the position, shooting direction, and angle of view of the sub-camera. The subject position in the image is adjusted by changing the pan value and tilt value, and the size of the subject area is adjusted by changing the zoom value.

[0124] In S4405, CPU 101 determines whether the role set in S4103 for the sub camera set as a slave is "exposure shift." If it is determined that the set role is "exposure shift," CPU 101 executes S4406, and if not, executes S4407.

[0125] In S4406, CPU 101 calculates the amount of exposure shift (amount of change in exposure setting) to be instructed to the sub-camera based on the current exposure setting of main camera 500 and information on the brightness of the subject area. Specifically, if CPU 101 determines in S4207 that the brightness of the subject of interest of main camera 500 is high, it calculates the amount of shift to reduce the amount of exposure for the exposure setting of main camera 500 so that the brightness becomes a predetermined appropriate level. Also, if CPU 101 determines in S4207 that the brightness of the subject of interest of main camera 500 is low, it calculates the amount of shift to increase the amount of exposure for the exposure setting of main camera 500 so that the brightness becomes a predetermined appropriate level. The unit of the shift amount can be preset, such as 1 / 2 step or 1 / 3 step.

[0126] The shift amount may be calculated using the automatic exposure control (AE) function of the sub-camera, or a fixed shift amount may be used. For example, if the shooting scene is one in which the appropriate correction amount can be measured in advance, the shift amount can be determined in advance. By making the exposure amount of the sub-camera set as a slave different from that of the main camera 500, even if the main camera 500 cannot shoot an image with an appropriate exposure, the sub-camera set as a slave can shoot an image with an appropriate exposure.

[0127] In addition, when there are multiple sub-cameras set as sub-subordinates, the shift amount may be different for each sub-camera. For example, by determining the shift amount of each sub-camera so that the sub-camera shoots with a higher or lower exposure amount than the main camera 500, it becomes possible to obtain an appropriate image even when the image of the main camera 500 is too bright or too dark.

[0128] In S4407, CPU 101 determines whether the role set in S4103 for the sub camera set as a slave is "aperture counter." If it is determined that the set role is "aperture counter," CPU 101 executes S4408, and if not, executes S4409.

[0129] In S4408, CPU 101 calculates the amount of change in the aperture value to be instructed to the sub-camera based on the aperture value included in the current exposure setting of main camera 500. Specifically, if the depth of field of the target subject is too shallow (narrower than the reference range), CPU 101 calculates the amount of change so that the aperture value is larger than the current aperture value of main camera 500. Also, if the depth of field of the target subject is too deep (wider than the reference range), CPU 101 calculates the amount of change so that the aperture value is smaller than the current aperture value of main camera 500. CPU 101 also calculates the amount of change in the shutter speed and / or the shooting sensitivity so that the amount of exposure does not change before and after the change in the aperture value. CPU 101 stores the calculated amount of change in RAM 102.

[0130] In S4409, CPU 101 determines whether the role set in S4103 for the sub camera set as a slave is "wide angle." If it is determined that the role set is "wide angle," CPU 101 executes S4410, and if not, executes S4411.

[0131] In S4410, CPU 101 detects an area of ​​the same subject as the subject included in the video of main camera 500 from the video of the sub-camera set as a slave. Then, CPU 101 calculates the zoom amount of the sub-camera so that the size of the detected subject area is smaller than the size in the video of main camera 500. Specifically, CPU 101 calculates the zoom amount so that the size of the subject area captured by main camera 500 is smaller by a predetermined percentage (for example, 10%, 20%, etc.). CPU 101 stores the calculated zoom amount in RAM 102.

[0132] In S4411, CPU 101 reflects the change instruction amount calculated in S4404 and the value calculated in S4406, S4408, or S4410 in the current setting information of the sub-camera set as a slave, and generates new setting information. Then, CPU 101 transmits a setting change command including the new setting information to the sub-camera set as a slave via network I / F 105. Note that, as shown in the role information in FIG. 3, the role set for each of the sub-cameras set as a slave is to make the target subject of main camera 500 the tracking subject. Therefore, the new setting information also reflects pan and tilt values ​​for changing the shooting direction required for automatic tracking. The automatic tracking control of the sub-camera will be described in detail in the third embodiment.

[0133] Also, the shooting direction of the subordinated sub-camera may be controlled so that the position of the tracking subject in the image is the same as or close to the position of the target subject in the image of the main camera 500. Alternatively, the shooting direction of the subordinated sub-camera may be controlled so that the position of the tracking subject in the image is shifted by a specific amount from the position of the target subject in the image of the main camera 500. The specific amount may be a value based on the distance between the position of the area of ​​the target subject in the image of the main camera 500 and the center of the image. Alternatively, the specific amount may be a value specified by the user input unit 411 of the role control device 400. The position is basically shifted in the horizontal direction, but the shift direction may be made specifiable. In this way, by controlling the position of the tracking subject in the subordinated sub-camera, it is possible to intentionally shoot an image in which the tracking subject is shifted to the left or right, or shoot an image in which the tracking subject appears in the same position as the image of the main camera 500. Therefore, an image that can be used as an alternative image to the image of the main camera 500, or as an image with a different composition from the image of the main camera 500 can be obtained.

[0134] In S4412, the CPU 101 acquires role information of each sub-camera other than the sub-camera set as a slave. The CPU 101 may refer to the role information stored in the RAM 102, or may acquire the role information from the role control device 400.

[0135] In S4413, CPU 101 detects the position of the tracking subject according to the role from the video of one sub-camera other than the sub-camera set as a slave. As will be described in detail in the third embodiment, CPU 101 detects the subject and its position in the shooting scene from the video of overhead camera 300. Then, CPU 101 identifies the tracking subject of the target sub-camera from the detected subjects, thereby identifying the position of the tracking subject.

[0136] In S4414, CPU 101 calculates the amount of change in pan and tilt values ​​(shooting direction) for the target sub-camera to track the tracking subject according to its role. Also, when the angle of view of main camera 500 has changed, CPU 101 calculates the amount of change in zoom value according to the role of the target sub-camera. Details will be described in the third embodiment.

[0137] In S4415, the CPU 101 transmits a control command including pan / tilt / zoom values ​​in which the calculated change amount is reflected in the current value to the target sub-camera.

[0138] In S4416, CPU 101 determines whether or not there is a sub camera for which a change instruction has not been given, and if it is determined that there is, executes S4413 again, and if not, ends the operation of FIG. 4D.

[0139] According to this embodiment, in a multi-camera imaging system, the sub-camera that automatically shoots in cooperation with the main camera can be controlled in different ways, such as by changing the tracking subject, shooting direction, and angle of view, depending on the role that has been set. Therefore, it is possible to obtain a variety of images from the sub-camera. In addition, when it is determined that the quality of the image of the main camera is low or that a change in the shooting environment that may cause it to be low has occurred, the sub-camera can be set to play a role of backing up the main camera, so that images that maintain a certain quality can be continuously shot. This is particularly useful in shooting scenes that cannot be re-shot, such as wedding scenes.

[0140] The role that can be set for the sub-camera that backs up the main camera is not limited to the above example. In the same way as in the case of changing the exposure amount, other shooting parameters that affect the image quality (e.g., white balance) may be changed. For example, when the white balance of the main camera is manually set, the role of shooting using auto white balance is set for the sub-camera that performs backup shooting. This allows the image taken by the sub-camera for backup shooting to be used even if color shift occurs in the image of the main camera.

[0141] In addition, a sub-camera that backs up the main camera may be set to shoot with the same settings as the main camera, allowing two cameras to shoot in the same way.

[0142] In addition, when automatically determining which sub camera will back up the main camera, the system takes into consideration whether the video being streamed is being shot and the priority level set for the sub camera, so that an unintended sub camera is not selected as the sub camera to back up the main camera.

[0143] In addition, if candidates for the sub-camera that backs up the main camera are set in advance, the sub-camera that backs up the main camera is determined according to the settings in advance. Therefore, a user who does not want the automatic determination can specify a candidate sub-camera in advance.

[0144] In addition, when the main camera no longer satisfies a predetermined condition, the sub-camera's subordinate setting is cancelled, so that the role of the sub-camera is not changed in a situation where subordinate setting is not necessary. This makes it possible to extend the time that the sub-camera takes pictures in the role originally set.

[0145] In this embodiment, the shooting control device 100 and the main camera 500 have been described as independent devices, but the functions of the shooting control device 100 can also be incorporated into the main camera 500. In this case, the image from the overhead camera 300 is supplied to the main camera 500. With this configuration, it is possible to reduce the number of devices required to realize a multi-camera imaging system.

[0146] <Second embodiment> Next, a second embodiment of the present invention will be described. This embodiment can be implemented in the imaging system described in the first embodiment, so a description of the configuration will be omitted. In addition, the roles that can be set for the sub-cameras are the same as those in the first embodiment. However, in this embodiment, the main camera 500 does not have to exist. Therefore, when the main camera 500 does not exist, the control contents of the tracking subject and the angle of view in the role are determined independently for each sub-camera.

[0147] In a multi-camera imaging system that automatically captures images using multiple cameras, there may be cases where a change in the imaging environment occurs such that a camera is unable to capture the intended image due to some factor. In this embodiment, when a camera is unable to capture the intended image, the effect of the camera missing the image that should have been captured is suppressed.

[0148] 5A to 5D, the operation of the imaging control device 100 according to this embodiment will be described. The operation of the imaging control device 100 described below is realized by the CPU 101 executing a program stored in the ROM 103 or acquired via the network I / F 105 and controlling each unit of the imaging control device 100.

[0149] FIG. 5A is a flowchart showing the overall operation of the imaging control device 100 according to this embodiment. In S5101, the CPU 101 initializes the value of a variable n, which counts the sub cameras, to 1. In S5102, the CPU 101 determines whether or not the variable n has exceeded the total number max of sub-cameras, and if it is determined that the variable n has exceeded the total number max of sub-cameras, the CPU 101 executes S5101, and if it is determined that the variable n has exceeded the total number max of sub-cameras, the CPU 101 executes S5103. In the example shown in FIG. 6, the number of sub-cameras is four, so max=4.

[0150] In S5103, CPU 101 acquires the role of sub camera n (1≦n≦max) from the role control device 400. It is not necessary to acquire a role for each sub camera, and role information for all sub cameras may be acquired from the role control device 400 and stored in RAM 102, and RAM 102 may be referred to thereafter. Sub camera n corresponds to any of sub cameras 600 to 900.

[0151] In S5104, the CPU 101 issues a control instruction to the sub camera n. Details will be described later with reference to FIG. In S5105, the CPU 101 determines the operating state of the sub camera n, which will be described in detail later with reference to FIG.

[0152] In S5106, CPU 101 reads the operation state of sub camera n determined in S5105 from RAM 102, and determines whether or not sub camera n is in a missing state. A missing state is a state in which the operation associated with the role acquired in S5103 is not being performed. If CPU 101 determines that sub camera n is in a missing state, it executes S5107, and if not, it executes S5111.

[0153] In S5107, the CPU 101 selects a sub-camera X to supplement the missing sub-camera n. Details will be described later with reference to Fig. 5D. The sub-camera X corresponds to one of the sub-cameras 600 to 900 excluding the sub-camera n.

[0154] In S5108, the CPU 101 determines whether or not the sub-camera X was selected in S5107, and if it is determined that the sub-camera X was selected, the CPU 101 executes S5109, and if it is not determined that the sub-camera X was selected, the CPU 101 executes S5111.

[0155] In S5109, the CPU 101 sets that the sub camera X is in a state that complements the sub camera n. Specifically, the CPU 101 stores, in a sub camera state management area provided in the RAM 102, a value indicating that the sub camera X is in a state that complements the sub camera n.

[0156] In S5110, the CPU 101 cancels the setting if there is a sub camera X that complements the sub camera n. Specifically, the CPU 101 deletes the value indicating that the sub camera X is in a state that complements the sub camera n, from the sub camera state management area in the RAM 102.

[0157] In S5111, CPU 101 notifies main camera 500 and role control device 400 of the complementation state via network I / F 105. The complementation state here refers to information including the operating state of sub camera n (whether or not it is in a missing state), and information on sub camera X that complements sub camera n when sub camera n is in a missing state. This makes it possible for main camera 500 and role control device 400 to perform or cancel a display to notify the user of the complementation state.

[0158] In S5112, the CPU 101 increments the variable n that counts the sub cameras. Then, the CPU 101 executes S5102.

[0159] (Control instructions to the sub camera) Next, the operation in S5104 in FIG. 5A will be described in detail with reference to the flowchart shown in FIG. 5B. In S5201, CPU 101 determines whether or not sub-camera n is set in a state to complement another sub-camera Y in S5109, and executes S5203 if it is determined that it is set, and executes S5202 if it is not determined that it is set. Sub-camera Y corresponds to one of sub-cameras 600 to 900 excluding sub-camera n.

[0160] In S5202, CPU 101 calculates the pan, tilt, and zoom values ​​of sub camera n based on the role of sub camera n acquired in S5103. The calculation of the pan, tilt, and zoom values ​​can be performed, for example, by the method described in the third embodiment. After that, CPU 101 executes S5210.

[0161] In S5203, the CPU 101 acquires the role of the sub camera Y in the same manner as in S5103.

[0162] In S5204, CPU 101 determines whether the direction of the angle of view control (follow or counter) of the control contents associated with the original role of sub camera n and the role of sub camera Y matches. If main camera 500 is not present, CPU 101 determines whether the angles of view of sub camera n and sub camera Y are similar. CPU 101 determines that the angles of view are similar if the difference in the angles of view is within a threshold, for example. If CPU 101 determines that the directions of the angle of view control match or that the angles of view are similar, it executes S5205, and if not, it executes S5209.

[0163] In S5205, CPU 101 determines whether the adjustment amount of the angle of view required to include the subjects of both sub-camera n and sub-camera Y in the shooting range of sub-camera n is equal to or less than a threshold. The adjustment amount and threshold may be values ​​in units of zoom value, for example. CPU 101 can make this determination based on position information of the subject detected from the video of overhead camera 300 and sub-camera Y, and the current shooting angle of view of sub-camera n. If CPU 101 determines that the necessary adjustment amount of the angle of view is equal to or less than the threshold, it executes S5206, and if not, it executes S5207.

[0164] In S5206, CPU 101 calculates the pan / tilt / zoom values ​​of sub camera n so as to include both subjects in the shooting range based on the roles of both sub camera n and sub camera Y. CPU 101 calculates the center of gravity positions of the subjects shot by each of the images of sub camera n and sub camera Y. CPU 101 then calculates the pan / tilt values ​​of sub camera n so as to track the center of gravity positions. After that, CPU 101 executes S5210.

[0165] In S5207, CPU 101 determines whether or not the distance between the subject of sub camera n and the subject of sub camera Y in the video of overhead camera 300 is less than a threshold, and executes S5208 if it is determined that the distance is less than the threshold, and executes S5209 if it is not determined that the distance is less than the threshold. The distance may be, for example, the distance between the center of gravity positions.

[0166] In S5208, the CPU 101 calculates the pan and tilt values ​​of the sub camera n so as to alternately photograph both subjects based on the roles of both the sub camera n and the sub camera Y. After that, the CPU 101 executes S5210.

[0167] In S5209, the CPU 101 calculates the pan / tilt / zoom values ​​of the sub camera n based on the role of the sub camera Y acquired in S5203, similarly to S5202.

[0168] In S5210, the CPU 101 transmits a control command including the calculated pan, tilt, and zoom values ​​to the sub camera n via the network I / F 105. This completes the control instruction operation to the sub camera n.

[0169] (Sub camera n operation status determination) Next, the operation in S5105 in FIG. 5A will be described in detail with reference to the flowchart shown in FIG. 5C.

[0170] In S5301, CPU 101 determines whether sub camera n autonomously took a photograph different from the role acquired in S5103. For example, this corresponds to a case where sub camera n determines a tracking subject by itself and takes a photograph of a target subject different from the role using a technology such as that disclosed in JP2020-182075A. If CPU 101 determines that sub camera n took a photograph that does not satisfy the role, it executes S5306, and if not, it executes S5302.

[0171] In S5302, the CPU 101 determines whether or not the sub camera n is being controlled with a control content different from the control content associated with the role acquired in S5103. For example, the following cases apply. As described in the first embodiment, when the sub camera n is set as a slave, When the shooting control device 100 controls the sub camera n to track a subject included in a video captured by the overhead camera 300 or a sub camera other than the sub camera n, The shooting control device 100 controls the sub-camera n as a sub-camera to compensate for another camera that is missing.

[0172] These are merely examples, and other cases may be included. If it is determined that the sub-camera n is being controlled with a role different from its original role, the CPU 101 executes S5306, and if not, executes S5303.

[0173] In S5303, CPU 101 determines whether an abnormal state of sub camera n has been detected, and if it is determined that an abnormal state has been detected, executes S5306, and if not, executes S5304. An abnormal state means a state that causes an impediment to shooting based on the role, such as a breakdown of any part of sub camera n, a power loss, a communication interruption, or a lack of recording capacity.

[0174] The abnormal state of the sub camera n can be detected by receiving a notification of an abnormality detected by the self-check function of the sub camera n itself at the shooting control device 100. Also, the CPU 101 may detect an abnormality of the sub camera n based on the video captured by the overhead camera 300 and the sub cameras (including the sub camera n). For example, this applies when a video cannot be received from the sub camera n, when a subject area cannot be detected from the video of the sub camera n, when the sub camera n cannot capture the tracking subject at an appropriate position, etc. Also applicable may be when communication with the sub camera n is not possible, or when a notification that should be received from the sub camera n is not received for a certain period of time.

[0175] In S5304, the CPU 101 determines whether or not the sub-camera n has been operated by a means outside the control of the shooting control device 100. If it is determined that the sub-camera n has been operated by a means outside the control, the CPU 101 executes S5306, and if not, the CPU 101 executes S5305. For example, this corresponds to a case where it is detected that the sub-camera n has been operated via the user input unit 613.

[0176] In S5305, the CPU 101 determines that the sub camera n is not missing. The CPU 101 reflects the determination result in the sub camera information stored in the RAM 102, and ends the operation shown in FIG. In S5306, the CPU 101 determines that the sub camera n is in a missing state. The CPU 101 reflects the determination result in the sub camera information stored in the RAM 102, and ends the operation shown in FIG.

[0177] (Select sub-camera X to complement sub-camera n) Finally, the operation in S5107 in FIG. 5A will be described in detail with reference to the flowchart shown in FIG. 5D. In S5401, the CPU 101 acquires the roles and priorities of all sub cameras from the role control device 400, and stores them in the RAM 102. Note that if this information has already been stored in the RAM 102, there is no need to acquire it again.

[0178] In S5402, the CPU 101 acquires information about all the sub cameras and stores it in the RAM 102. The information acquired here is - Information about the optical system (focal length range, minimum and maximum aperture), Drive unit information (pan and tilt range, current angle of view), Installation position (e.g. three-dimensional coordinates in a coordinate system having a predefined origin), The current shooting direction (which can be the pan and tilt angles of the drive), Image sensor information (number of pixels, sensor size, color filter type, etc.), It should be noted that not all of these are required, and other information may be included.

[0179] The information may be the most recently acquired information stored in the RAM 102, or may be acquired from the sub-camera each time.

[0180] In S5403, CPU 101 determines whether sub camera A similar in position and capability to sub camera n exists. Sub camera A is selected from among sub cameras 600 to 900 other than sub camera n. For example, CPU 101 can select as sub camera A a sub camera whose installation position distance is less than a threshold and whose pan / tilt driving ranges and zoom ranges overlap at a predetermined rate or more. Note that the conditions for sub camera A are not limited to these. If CPU 101 determines that sub camera A exists, it executes S5404, and if not, it executes S5405.

[0181] In S5404, CPU 101 compares the priority of sub camera n with the priority of sub camera A. Then, if the priority of sub camera n is higher (if the priority number is smaller), CPU 101 executes S5411, and if not, executes S5405.

[0182] In S5405, CPU 101 determines whether or not there are a plurality of other sub-cameras to which the same role as sub-camera n is set, and if it is determined that there are, executes S5406, and if not, executes S5408.

[0183] In S5406, the CPU 101 determines, as the sub camera A, the sub camera with the lowest priority among a plurality of other sub cameras to which the same role as the sub camera n is set.

[0184] In S5407, similarly to S5404, CPU 101 compares the priority order of sub camera n with the priority order of sub camera A. If the priority order of sub camera n is higher, CPU 101 executes S5411, and if not, executes S5408.

[0185] In S5408, the CPU 101 determines the sub-camera with the lowest priority among all the sub-cameras as the sub-camera A.

[0186] In S5409, the CPU 101 determines whether or not the sub camera n and the sub camera A are the same, and if it is determined that they are the same, the CPU 101 executes S5410, and if not, the CPU 101 executes S5411.

[0187] In S5410, CPU 101 determines that there is no sub-camera that corresponds to sub-camera X, and ends the operation shown in FIG. 5D. In S5411, CPU 101 selects sub camera A as sub camera X, and ends the operation shown in FIG. 5D.

[0188] An example of transition of roles of the sub-cameras according to the second embodiment will be described with reference to FIG. 7, which is a schematic diagram similar to FIG.

[0189] 7(A), the role control device 400 assigns the sub camera 600 the role of "assist counter L," the sub cameras 700 and 800 the role of "main follow," and the sub camera 900 the role of "assist counter R." Then, the photography control device 100 executes operation control for each of the sub cameras 600 to 900 according to the assigned role.

[0190] Now, suppose that the sub camera 900 becomes missing due to some factor. In this case, of the sub cameras 700 and 800 that are set to the same role, the sub camera 700 with the lower priority is selected as the sub camera X to replace the sub camera 900. As a result, the role of the sub camera 700 switches to "assist counter R", and the shooting control device 100 performs control corresponding to that role (FIG. 7B). The roles of the sub cameras 600 and 800 do not change.

[0191] Another example will be described. In Fig. 7(C), the roles of "main counter" are set to sub camera 600, "assist follow R" to sub camera 700, "assist follow L" to sub camera 800, and "assist counter R" to sub camera 900. Then, the photography control device 100 executes operation control according to the set role for each of the sub cameras 600 to 900.

[0192] Here, suppose that the sub camera 900 becomes missing due to some factor. In this case, the sub camera 700, which has similar performance and position to the sub camera 900 and has a lower priority than the sub camera 900, is selected as the sub camera X to replace the sub camera 900. As a result, the role of the sub camera 700 is switched to "assist counter R", and the control corresponding to that role is performed by the shooting control device 100. Furthermore, since the sub camera 700 is missing because it is no longer performing its original role, the sub camera 600, which has the lowest priority among the remaining sub cameras, is selected as the sub camera X to replace the sub camera 700. As a result, the role of the sub camera 600 is switched to "assist follow R", and the control corresponding to that role is performed by the shooting control device 100 (FIG. 7D).

[0193] According to this embodiment, when a sub-camera with a higher priority among the multiple sub-cameras included in the multi-camera shooting system is unable to shoot as intended, another sub-camera takes over and shoots instead. Therefore, it is possible to continuously obtain images similar to those to be shot by the sub-camera with a higher priority, and highly reliable multi-camera shooting can be realized. It is also possible to perform alternative shooting of the sub-camera used for alternative shooting of the other sub-camera with yet another sub-camera.

[0194] Furthermore, instead of simply setting the role set to the missing sub-camera for alternative shooting to the sub-camera for alternative shooting, it is also possible to control the operation so that the role set to the sub-camera for alternative shooting can be maintained even after alternative shooting begins.

[0195] The missing sub-camera state can be notified to the photographer of the main camera 500 or the operator of the role control device 400. This makes it possible to support a quick response, such as replacement of the missing sub-camera.

[0196] In this embodiment, the shooting control device 100 and the main camera 500 have been described as independent devices, but the functions of the shooting control device 100 can also be incorporated into the main camera 500. In this case, the image from the overhead camera 300 is supplied to the main camera 500. With this configuration, it is possible to reduce the number of devices required to realize a multi-camera imaging system.

[0197] <Third embodiment> Next, a third embodiment of the present invention will be described. This embodiment can be implemented in the imaging system described with reference to Figs. 1 and 2, so a description of the configuration will be omitted. In this embodiment, the roles that can be set for the sub-cameras 600 to 900 are four types shown in Fig. 9. Each role and its control content are the same as the role with the same name in Fig. 3. Note that "assist follow" in Fig. 9 corresponds to "assist follow L" in Fig. 3, and "assist counter" corresponds to "assist counter L" in Fig. 3, respectively.

[0198] <Explanation of the operation of each device> Next, the operation of each device in the imaging system according to this embodiment will be described. Here, the imaging control device 100 automatically controls the imaging operations of the sub-cameras 600 to 900 based on the video of the overhead camera 300, information obtained from the main camera 500, and the roles set for the sub-cameras 600 to 900.

[0199] 8 is a diagram showing a series of processes performed by the photography control device 100 when controlling the operations of the sub-cameras 600 to 900, focusing on the main operations and signal flows. The functional blocks shown in the photography control device 100 show the main operations in a schematic manner, and correspond to the main functions provided by the photography control application. Each functional block in FIG. 8 is realized by a combination of the CPU 101 that executes the photography control application and one or more of the functional blocks of the photography control device 100 shown in FIG. 2.

[0200] Fig. 10 is a flowchart showing the operation of CPU 101 as role determination unit 120. Fig. 11(a) to Fig. 11(d) are flowcharts relating to the operation of shooting control device 100, overhead camera 300, main camera 500, and sub camera 600, respectively. Since the operation of sub cameras 700, 800, and 900 is the same as that of sub camera 600, only sub camera 600 will be described below. Therefore, in the following description, the description of sub camera 600 also applies to sub cameras 700, 800, and 900.

[0201] In the following description, it is assumed that the three-dimensional coordinate values ​​of the viewpoint position and the shooting direction (optical axis direction) of the overhead camera 300 are known to the shooting control device 100. It is also assumed that known position information, such as the three-dimensional coordinate values ​​of the viewpoint positions of the sub camera 600 and the main camera 500 and the coordinate values ​​of markers arranged in the shooting range, is stored in advance in the ROM 103 as default position information REF_POSI. It is also assumed that the coordinate system of the position is predetermined depending on the type of position.

[0202] (Operation of role determination unit 120) First, the operation of the CPU 101 as the role determining section 120 in Fig. 8 will be described with reference to the flowchart shown in Fig. 10. The operation described below is realized by the CPU 101 executing a shooting control application.

[0203] 10 is not particularly limited to the timing for starting the operations shown in the flowchart of Fig. 10, but the operations are executed at least before starting control of the shooting operations of the sub-cameras 600 to 900. The operations are also executed when a notification is received from the role control device 400 via the network I / F 105 that the role settings for the sub-cameras 600 to 900 have been changed.

[0204] In S101, the CPU 101 as the role determination unit 120 acquires roles ROLE (role setting information) corresponding to the sub-cameras 600 to 900 from the role control device 400. The CPU 101 can acquire the above-mentioned role setting information from the role control device 400 by, for example, transmitting a role acquisition command to the role control device 400 via the network I / F 105. The CPU 101 stores the acquired role setting information in the RAM 102.

[0205] In S103, CPU 101 refers to the role setting information stored in RAM 102 based on the identification information of sub cameras 600-900, and acquires operation control content for sub cameras 600-900. Then, CPU 101 as role determination section 120 transmits the acquired operation control content (CAMERA_ROLE) to tracking subject determination section 123. In practice, CPU 101 stores the operation control content in a specific area of ​​RAM 102, and refers to it when functioning as tracking subject determination section 123.

[0206] In S104, the CPU 101 as the role determination unit 120 transmits the acquired operation control content (CAMERA_ROLE) to the zoom value calculation unit 125. In practice, the CPU 101 stores the operation control content in a specific area of ​​the RAM 102 and refers to it when functioning as the zoom value calculation unit 125.

[0207] (Operation of the imaging control device 100) Next, the operation of the shooting control device 100 to control shooting by the sub-cameras 600-900 will be described with reference to Fig. 8 and Fig. 11(a). The operation described below corresponds to the operation of the CPU 101 as the recognition unit 121, the target subject determination unit 122, the tracking subject determination unit 123, the pan / tilt value calculation unit 124, and the zoom value calculation unit 125 in Fig. 8. Note that the operation described below is realized by the CPU 101 executing a shooting control application.

[0208] In S201, CPU 101 transmits a shooting instruction command in a predetermined protocol to overhead camera 300 through network I / F 105. In response to this command, overhead camera 300 starts supplying a video signal (moving image data) IMG to video input unit 107. CPU 101 starts storing the video signal received by video input unit 107 in RAM 102, and then executes S202.

[0209] In S202, CPU 101 acquires information ANGLE indicating the shooting direction from main camera 500. Specifically, CPU 101 transmits a shooting direction acquisition command to main camera 500 via network I / F 105 in a predetermined protocol. In response to the shooting direction acquisition command, CPU 501 of main camera 500 transmits information ANGLE indicating the current shooting direction of main camera 500 to shooting control device 100. Information ANGLE may be, for example, the pan and tilt angles of drive unit 509. CPU 101 stores the acquired information ANGLE in RAM 102.

[0210] In S203, the recognition unit 121 executes the following process. (1) Apply subject area detection processing to the input frame image and store the detection results. (2) For each detected subject area, the position information (image coordinates) is converted. (3) Apply classification processing to each detected subject region to identify the classification information (if a new subject is detected, add classification processing information). (4) For each detected subject area, the identification information ID[n] and the position information POSITION[n] are associated and stored.

[0211] The recognition unit 121 is mainly realized by the CPU 101 and the inference unit 104. The CPU 101 reads one frame of the image received from the overhead camera 300 from the RAM 102 and inputs it to the inference unit 104.

[0212] The operation of the recognition unit 121 will be explained step by step below. (1) First, the inference unit 104 inputs a frame image into a machine learning model to detect a subject region. The inference unit 104 stores in the RAM 102 the position and size of each detected subject region, which are output by the machine learning model as a detection result, and the detection reliability. The position and size of the subject region may be any information capable of identifying the position and size of a rectangular region inscribed by the subject region. Here, the center coordinate of the bottom side, and the width and height of the rectangular region are used as the position and size of the subject region.

[0213] Furthermore, inference unit 104 stores the detection result for the first frame image in RAM 102 in association with the subject identification information ID[n]. Here, n is the subject number and is an integer ranging from 1 to the total number of detected subject regions. Furthermore, inference unit 104 stores the subject regions detected from the first frame image in RAM 102 in association with the subject identification information ID[n] as templates for identifying individual subjects. If template matching is not used to identify subjects, it is not necessary to store templates.

[0214] Fig. 13(a) shows an example of the result of subject detection processing by inference unit 104 for the image captured by overhead camera 300 shown in Fig. 12(a). Here, the areas of human subjects A to C present within shooting range 20 are detected, and the coordinates (foot coordinates) of the center of the bottom side of a rectangular area inscribed with the subject areas are output as the position.

[0215] For coordinate conversion, which will be described later, when a marker is placed at a known position within the shooting range 20 as shown in FIG. 12(b), the CPU 101 detects the image of the marker included in the frame image (FIG. 12(a)) and stores the position in the RAM 102. The detection of the marker image may also be performed by the inference unit 104. The detection of the marker image may be performed by any known method, such as pattern matching using a marker template. The marker image may also be detected using a pre-stored machine learning model for marker detection.

[0216] (2) Next, the coordinate conversion executed by the inference unit 104 will be described. Fig. 12(a) shows a schematic diagram of an image captured by the overhead camera 300, and Fig. 12(b) shows a schematic diagram of the shooting range 20 as viewed from directly above its center. The inference unit 104 converts the position of the subject area in the coordinate system of the overhead camera into values ​​in a coordinate system (planar coordinate system) when the shooting range 20 is viewed from directly above its center.

[0217] The reason for converting the coordinates into values ​​in the plane coordinate system here is that it is convenient to calculate pan values ​​(movement angles in a horizontal plane) for photographing a specific subject with the sub-cameras 600 to 900. Note that this is based on the premise that the sub-cameras 600 to 900 are installed so that the driving unit 609 performs panning within a horizontal plane parallel to the floor of the shooting range 20.

[0218] Coordinate conversion can be performed in various ways, but here, markers are placed at multiple known positions on the floor of the shooting range 20, and coordinate conversion is performed from the overhead camera coordinate system to the planar coordinate system based on the marker positions in the image captured by the overhead camera 300. Note that coordinate conversion may also be performed without using markers, for example, by using the viewpoint position and shooting direction of the overhead camera 300.

[0219] The coordinate transformation can be performed using a homography transformation matrix H according to Equation 1 below.

number

[0220] The homography transformation matrix can be calculated by substituting the coordinates of the four markers detected from the video and the coordinates (known) of the four markers arranged in the shooting range 20 into Equation 1 and solving the simultaneous equations. When the positional relationship between the shooting range 20 and the overhead camera 300 is fixed, the homography transformation matrix H can be calculated in advance during test shooting and stored in, for example, ROM 103.

[0221] CPU 101 sequentially reads out the positions of the subject regions from RAM 102 and converts the coordinates into values ​​in a planar coordinate system. Fig. 13(b) shows a schematic diagram of the state in which the foot coordinates (x, y) of each subject region detected in the overhead camera 300 video shown in Fig. 13(a) are converted into coordinate values ​​(X, Y) in a planar coordinate system using Equation 1 and the homography transformation matrix H stored in ROM 103. CPU 101 stores the converted foot coordinates in RAM 102 as POSITION[n].

[0222] (3) Next, the operation of the inference unit 104 to identify the identification information ID[n] of the subject will be described. Here, the subject is identified using template matching. The subject is identified for the processing results of the second and subsequent subject detections. For the first processing result, a new identification information ID[n] can be assigned to the subject region.

[0223] The inference unit 104 identifies the identification information ID[n] of the detected object region by template matching using the template stored in the RAM 102. This identifies the object within the shooting range. For example, the inference unit 104 calculates an evaluation value representing the correlation of each template for each detected object region. Then, the inference unit 104 identifies the identification information ID[n] corresponding to the template having a certain degree of correlation or higher and having the highest correlation as the identification information ID[n] of the object region. For the evaluation value, a known value such as the sum of absolute differences of pixel values ​​can be used.

[0224] Note that for a subject region that does not have a certain level of correlation with all the templates, the inference unit 104 assigns new identification information ID[n] and adds the image of the subject region to the template.

[0225] In addition, the inference unit 104 may update an existing template using a subject area detected in a most recent frame image, or delete a template in which a subject area having a certain degree of correlation or more has not existed for a certain period of time. Furthermore, the inference unit 104 may store in the ROM 103 templates corresponding to frequently appearing identification information ID[n].

[0226] Note that the subject may be identified by a method other than template matching. For example, the subject may be identified as the same identification information ID[n] as the subject area having the closest immediately preceding detection position and / or size. Alternatively, the position in the current frame image may be predicted by a Kalman filter or the like from the transition of positions in multiple past detection results associated with the same identification information, and the same identification information ID may be identified as the subject area that is closest to the predicted position. These methods may also be combined. By not using template matching, it is possible to improve the accuracy of identifying different subjects that look similar.

[0227] (4) The inference unit 104 associates the specified identification information ID[n] with the position (planar coordinate system) POSITION[n] of the corresponding subject region and stores them in the RAM 102.

[0228] Of the processes (1) to (4), the processes other than the subject detection may be executed by the CPU 101 instead of the inference unit 104.

[0229] Here, the image from the overhead camera 300 is used to determine the identification information ID[n] and the position POSITION[n] of the subject within the shooting range 20. However, the images from the sub-cameras 600 to 900 may also be used. The CPU 101 executes the operation shown in the flowchart of FIG. 11(a) for each sub-camera. The position of the subject area is output as a value in the coordinate system for each sub-camera. Thus, although the overhead camera 300 is not essential, it is considered that the subject detection accuracy is better when the overhead camera 300 is used.

[0230] Returning to the explanation of Fig. 11(a), in S204, CPU 101 as the subject of interest determination unit 122 in Fig. 8 determines a subject of interest as a tracking subject of main camera 500. CPU 101 can determine a subject of interest of main camera 500 from among the subjects detected in S203, based on the shooting direction of main camera 500 acquired in S202. CPU 101 stores identification information ID[n] corresponding to the subject area determined as the subject of interest of main camera 500 in RAM 102 as identification information MAIN_SUBJECT of the subject of interest.

[0231] For example, CPU 101 can determine that the subject closest to the shooting direction of main camera 500 in the planar coordinate system is the subject of interest of main camera 500. Note that, when there are multiple subjects whose distance from the shooting direction of main camera 500 is equal to or less than a threshold, the user may be allowed to select the subject of interest from among them.

[0232] When allowing the user to select a subject of interest, CPU 101 displays on display unit 108 or an external display device the frame image to which subject detection processing has been applied in S202, together with an indicator indicating the shooting direction and an indicator indicating a subject area that is a candidate for a subject of interest. The indicator for the subject area may be, for example, a rectangular frame indicating the outer edge of the subject area as shown in Fig. 13(a), but may be another indicator. CPU 101 may also display on display unit 108 a message prompting the user to select a subject of interest within the image.

[0233] The user can select a subject area corresponding to a desired subject of interest by operating the user input unit 106 (input device). There are no particular limitations on the selection method, but the selection method may be an operation of operating a mouse or keyboard to specify the desired subject area.

[0234] When the CPU 101 detects a user operation for designating a subject region, the CPU 101 stores in the RAM 102 the identification information ID[n] corresponding to the designated subject region as identification information MAIN_SUBJECT of the target subject.

[0235] Next, in S205, CPU 101 as tracking subject determination unit 123 in Fig. 8 acquires control content CAMERA_ROLE according to the role set for sub camera 600. Specifically, CPU 101 reads out control content CAMERA_ROLE acquired in the role determination process described with reference to Fig. 10 and stored in RAM 102. CPU 101 executes the processes of S205 to S207 for each sub camera.

[0236] In S206, CPU 101 as tracking subject determination unit 123 determines a subject to be tracked and photographed by sub camera 600 according to the control content CAMERA_ROLE. CPU 101 determines a tracking subject of sub camera 600 according to the tracking subject rule (FIG. 9) included in the control content CAMERA_ROLE.

[0237] When the tracking subject of the sub camera 600 is to be the same as the target subject of the main camera 500, the CPU 101 sets the identification information MAIN_SUBJECT of the target subject determined in S203 as the identification information SUBJECT_ID of the tracking subject of the sub camera 600.

[0238] When the tracking subject of the sub camera 600 is a subject located on the left side of the subject other than the target subject of the main camera 500, the CPU 101 detects a subject area located on the left edge of the subject area other than the target subject from among the subject areas detected in S203. Then, the CPU 101 sets the identification information ID[n] corresponding to the detected subject area as the identification information SUBJECT_ID of the tracking subject of the sub camera 600.

[0239] CPU 101 writes the identification information SUBJECT_ID of the determined tracking subject to RAM 102. If the tracking subject may differ depending on the sub camera, CPU 101 stores the identification information SUBJECT_ID of the tracking subject in association with the identification information of the sub camera. Note that if the tracking subject changes, CPU 101 retains the information of the previous tracking subject in RAM 102 without erasing it.

[0240] Here, the operation when the role set for the sub camera 600 is "main follow" will be described with reference to Fig. 14. The photography control device 100 controls the sub camera 600 set for the role "main follow" so as to track the subject of interest of the main camera 500.

[0241] Therefore, when the subject of interest of the main camera 500 is determined to be subject B as shown in Fig. 14(a), the CPU 101 determines subject B as the subject to be tracked by the sub camera 600. Thereafter, when the subject of interest of the main camera 500 is determined to have changed to subject A as shown in Fig. 14(b), the CPU 101 changes the subject to be tracked by the sub camera 600 to subject A. Similarly, when the subject of interest of the main camera 500 is determined to have changed to subject C as shown in Fig. 14(c), the CPU 101 changes the subject to be tracked by the sub camera 600 to subject C.

[0242] The operation of the sub camera 600 when the role set for the sub camera 600 is "assist follow" will be described with reference to Fig. 15. For the sub camera 600 set to the role "assist follow", the photography control device 100 controls the sub camera 600 so as to track a subject located on the left side of the subject other than the subject of interest of the main camera 500.

[0243] Therefore, when the subject of interest of the main camera 500 is determined to be subject B as shown in Fig. 15(a), the CPU 101 determines the subject A on the left side of subjects A and C as the subject to be tracked by the sub camera 600. Thereafter, when the subject of interest of the main camera 500 is determined to have changed to subject A as shown in Fig. 15(b), the CPU 101 changes the subject to be tracked by the sub camera 600 to subject B on the left side of subjects B and C. Also, when the subject of interest of the main camera 500 is determined to have changed to subject C as shown in Fig. 15(c), the CPU 101 changes the subject to be tracked by the sub camera 600 to subject A on the left side of subjects A and B.

[0244] By dynamically changing the roles set for the sub-cameras 600 to 900 using the role control device 400, it is possible to change the subjects tracked by the sub-cameras 600 to 900, enabling flexible automatic shooting.

[0245] 11(a), in S207, CPU 101 as pan / tilt value calculation unit 124 calculates the change amounts of pan angle and tilt angle required for sub camera 600 to track and photograph the tracking subject determined in S206. CPU 101 as zoom value calculation unit 125 calculates the zoom value of sub camera 600 according to the change in the angle of view of main camera 500. Although the sub camera 600 will be described below, the calculation of the change amounts of pan angle and tilt angle and the calculation of the zoom value are similarly performed for the other sub cameras 700 to 900.

[0246] First, there will be described the operation of the CPU 101 as the pan / tilt value calculation unit 124. Here, it is assumed that the following information is stored in advance in the ROM 103 as default position information REF_POSI for each sub camera. - 3D coordinates of the installation location (values ​​in a planar coordinate system) -Shooting direction corresponding to the initial values ​​of the pan and tilt angles of the drive unit Controllable range of pan and tilt angles

[0247] The CPU 101 reads out the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the subject being tracked by the sub camera 600 from the RAM 102. Then, the CPU 101 first determines the pan angle from the position information POSITION_OH and the installation position of the sub camera 600.

[0248] 16 is a diagram showing an example of the positional relationship between the sub camera 600 and a tracking subject in a plane coordinate system. Here, a pan angle θ for directing the optical axis direction of the sub camera 600 toward the subject position is determined. The CPU 101 calculates the pan angle θ using the following formula 2.

number

[0249] In Equation 2, px and py are the horizontal and vertical coordinates of the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracked subject. Also, subx and suby are the horizontal and vertical coordinates of the installation position of the sub-camera. Here, it is assumed that the current pan angle is the initial value of 0° and the optical axis direction is the vertical direction (Y-axis direction). If the current optical axis direction is not the vertical direction, the angle difference between the current optical axis direction and the vertical direction may be reflected in the angle obtained by Equation 2. Also, the direction of panning is counterclockwise if subx > px, and clockwise if subx < px.

[0250] Next, with reference to FIG. 17, a method for determining the tilt angle will be described. FIG. 13 shows a state of the sub-camera and the tracked subject as viewed from the side. It is assumed that the current optical axis of the sub-camera 600 is in the horizontal direction and the height is h1, and the height of the face of the tracked subject facing the optical axis is h2. Let ρ be the angle difference (tilt angle) in the height direction between the current optical axis direction and the target optical axis direction. The CPU 101 calculates the tilt angle ρ using the following Equations 3 and 4.

Equation

[0251] The coordinate values used in Equation 4 are the same as those used in Equation 2. It is assumed that h1 and h2 are input in advance to the shooting control application and stored in the RAM 102. In this case, the identification number associated with h2 for each subject is made equal to the identification number assigned in the subject detection process. Alternatively, h2 may be a value measured in real time using a sensor (not shown).

[0252] Here, it is assumed that the current tilt angle is the initial value of 0° and the optical axis direction is the horizontal direction (constant height). If the current optical axis direction is not the horizontal direction, the angle difference between the current optical axis direction and the horizontal direction may be reflected in the angle obtained by Equation 4. Also, the direction of tilting is downward if h1 > h2, and upward if h1 < h2.

[0253] CPU 101 periodically communicates with sub-camera 600 via communication network 200, acquires the current optical axis direction (pan angle and tilt angle of the drive unit), and stores it in RAM 102. The communication cycle can be set to, for example, the reciprocal of the frame rate or less. Alternatively, CPU 101 may hold in RAM 102 the sum of the pan angle and tilt angle controlled from the initial state for sub-camera 600, and use it as the current optical axis direction.

[0254] In this way, CPU 101 calculates the amount of change in the pan angle and tilt angle of sub camera 600, and stores the amount of change in RAM 102. CPU 101 also calculates the amount of change in the pan angle and tilt angle of each of the other sub cameras 700 to 900.

[0255] The change amount of the pan angle and tilt angle may be the angular velocity for the sub camera 600 to rotate in the direction of the tracking subject. For example, the CPU 101 acquires the current pan angle and tilt angle from the sub camera 600 through the communication network 200. Then, the CPU 101 calculates the angular velocity of the pan proportional to the difference between the pan angle θ read from the RAM 102 and the current pan angle. The CPU 101 also calculates the angular velocity of the tilt proportional to the difference between the tilt angle ρ read from the RAM 102 and the current tilt angle. The CPU 101 stores the angular velocity calculated in this manner in the RAM 102.

[0256] The change amount of the pan angle and the tilt angle may be calculated using the image of the sub camera 600 instead of the image of the overhead camera 300. In this case, the CPU 101 may calculate the change amount of the pan angle from the horizontal difference between the current optical axis direction and the direction of the tracked subject in the coordinate system of the sub camera 600, and may calculate the change amount of the tilt angle from the vertical difference. Also, the imaging system may change the shooting direction to track and shoot the tracked subject in only one of the pan direction and the tilt direction, and in such an imaging system, the change amount of only one of the pan angle and the tilt angle may be calculated.

[0257] Next, a description will be given of the operation of CPU 101 as zoom value calculation unit 125. CPU 101 as zoom value calculation unit 125 periodically acquires information MAIN_ZOOM indicating the angle of view of main camera 500, and stores it in RAM 102. Then, when information MAIN_ZOOM changes, CPU 101 calculates zoom value Z_VALUE for sub camera 600 according to control content CAMERA_ROLE according to the role set for sub camera 600.

[0258] CPU 101 can determine the zoom operation and its phase of main camera 500, for example, by detecting a change in the angle of view of the image of main camera 500. For example, the change in the angle of view may be detected from a change over time in the size or interval of the subject area.

[0259] 18 shows an example of mapping of zoom values ​​of the main camera and the sub-camera. Here, it is assumed that the main camera 500 and the sub-cameras 600 to 900 optically change the angle of view (the photographing optical system has a zoom function). However, a similar function may be realized by digital zoom using the image processing units 506 and 606.

[0260] The zoom value is a parameter having a value corresponding to the angle of view, and in this embodiment, the smaller (narrower) the angle of view, the smaller the zoom value, and the zoom value on the telephoto side is smaller than the zoom value on the wide-angle side. The sub-cameras 600 to 900 and the main camera 500 are capable of controlling the photographing optical system to an angle of view corresponding to the zoom value by transmitting a command specifying the zoom value. That is, the zoom value is information related to the angle of view and is information representing the zoom state. The zoom value may be, for example, the focal length (mm) of the photographing optical system corresponding to a 35 mm full-size image sensor, in which case the zoom value on the telephoto side is larger than the zoom value on the wide-angle side.

[0261] In Fig. 18, the range of the zoom value MAIN_ZOOM of the main camera 500 is main_min to main_max. The zoom range of the sub cameras 600 to 900 is sub_min to sub_max. main_min and sub_min are the zoom values ​​corresponding to the telephoto ends of the main camera 500 and the sub cameras 600 to 900, respectively, and main_max and sub_max are the zoom values ​​corresponding to the wide-angle ends of the main camera 500 and the sub cameras 600 to 900, respectively. Fig. 18 shows an example in which the range of the zoom value of the main camera 500 is wider than the range of the zoom value of the sub cameras 600 to 900 at both the telephoto end and the wide-angle end.

[0262] When controlling the zoom value SUB_ZOOM of the sub camera 600 to be in phase with the zoom value MAIN_ZOOM of the main camera 500, the CPU 101 calculates SUB_ZOOM corresponding to the current MAIN_ZOOM using the following formula 5.

number

[0263] On the other hand, when controlling the zoom value SUB_ZOOM of the sub camera 600 to be in the opposite phase to the zoom value MAIN_ZOOM of the main camera 500, the SUB_ZOOM corresponding to the current MAIN_ZOOM is calculated using the following formula 6. Specifically, the CPU 101 calculates the SUB_ZOOM corresponding to the current MAIN_ZOOM by substituting the SUB_ZOOM calculated using formula 5 into the right side of the following formula 6. SUB_ZOOM=sub_max-(SUB_ZOOM-sub_min) (Formula 6)

[0264] When main camera 500 performs digital zoom and controls the angle of view by cropping, CPU 101 can determine the zoom value SUB_ZOOM of sub camera 600 according to the size of the range to be cropped by main camera 500. Specifically, CPU 101 sets the zoom value SUB_ZOOM smaller (higher magnification) as the size of the range to be cropped by main camera 500 becomes smaller, and sets the zoom value SUB_ZOOM larger (lower magnification) as the size becomes larger.

[0265] In addition, the content of the zoom control associated with the role of the sub-cameras 600 to 900 is not limited to the control in the same phase or opposite phase as the main camera 500. For example, a zoom operation independent of the change in the angle of view of the main camera 500 may be associated with the role. For example, an auto-zoom operation that keeps the size of the tracking subject constant may be associated with the role. Also, the angle of view of the sub-cameras 600 to 900 may be fixed to a specific angle of view. By adding roles associated with these zoom controls to the control contents for each role shown in FIG. 9, or by changing the content of the zoom control for the role shown in FIG. 9, various zoom controls for the sub-cameras 600 to 900 are possible.

[0266] Returning to FIG. 11(a), in S207, CPU 101 reads out the change amounts of the pan and tilt angles calculated in S206 and the zoom value from RAM 102. Then, CPU 101 generates a control command PT_VALUE that instructs sub camera 600 to change the pan angle and tilt angle corresponding to these change amounts. CPU 101 also generates a control command Z_VALUE that instructs sub camera 600 to change the angle of view corresponding to the zoom value. The format of the control command is predefined. CPU 101 stores the generated control commands PT_VALUE and Z_VALUE in RAM 102. Note that S207 may be skipped when it is not necessary to generate a control command, such as when the tracking subject is stationary or when the angle of view of main camera 500 does not change.

[0267] Then, the CPU 101 reads out the control commands PT_VALUE and Z_VALUE from the RAM 102, and transmits them to the communication network 200 via the network I / F 105. The sub-camera 600 receives the control commands PT_VALUE and Z_VALUE via the network I / F 605.

[0268] The CPU 101 executes the process from S201 on the next frame image of the video captured by the overhead camera 300. Note that the process shown in Fig. 11(a) does not necessarily have to be executed for every frame.

[0269] (Operation of overhead camera 300) Next, the operation of the overhead camera 300 will be described with reference to Fig. 11(b). The operation described below is realized by the CPU 301 executing a program.

[0270] When the overhead camera 300 is powered on, each functional block is initialized by the CPU 301, and the camera enters a shooting standby state. In the shooting standby state, the CPU 301 may start a video shooting process for live view display, and output display image data generated by the image processing unit 306 to the shooting control device 100 via the network I / F 305.

[0271] In the shooting standby state, the CPU 301 waits for reception of a control command through the network I / F 305. When the CPU 301 receives a control command, it executes an operation according to the control command. Here, an operation when a shooting command is received as a control command from the shooting control device 100 will be described.

[0272] In S301 , the CPU 301 receives an imaging command from the imaging control device 100 via the network I / F 305 .

[0273] The shooting command may specify shooting parameters such as frame rate, resolution, etc. Also, the shooting command may include settings related to the processing to be applied by the image processing unit 306.

[0274] In S302, in response to receiving the shooting command, the CPU 301 starts a moving image shooting process for supplying the image to the shooting control device 100. In this moving image shooting process, a moving image with higher image quality is shot than that of the moving image shooting process for live view display. For example, at least one of the resolution of the moving image and the shooting frame rate is higher than that of the moving image for live view display. The image processing unit 306 applies processing to the image based on the settings for the moving image to be supplied to the shooting control device 100. The image processing unit 306 sequentially stores the generated moving image data in the RAM 302.

[0275] In S303, the CPU 101 reads the video data from the RAM 302, and transmits it to the shooting control device 100 via the network I / F 305. After that, the process from shooting to supply of video data is continued until a control command to stop shooting is received.

[0276] (Operation of main camera 500) Next, the operation of the main 500 will be described with reference to Fig. 11(c) The operation described below is realized by the CPU 501 executing a program.

[0277] When the main camera 500 is powered on, each functional block is initialized by the CPU 501, and then moving image shooting processing for supplying to the shooting control device 100 is started. The image processing unit 506 applies processing to the analog image signal obtained from the image sensor 507 based on the settings for moving images to be supplied to the shooting control device 100. The image processing unit 506 sequentially stores the generated moving image data in the RAM 502. The CPU 501 reads the moving image data from the RAM 502 and supplies it to the shooting control device 100 via the network I / F 505.

[0278] The CPU 501 waits for a control command via the network I / F 305 while supplying video data to the shooting control device 100. When the CPU 501 receives a control command, it executes an operation according to the control command. Here, an operation when a shooting direction acquisition command is received will be described. When a pan / tilt control command PT_VALUE or a zoom control command Z_VALUE is received, the CPU 501 drives the drive unit 509 according to the command.

[0279] In S501, the CPU 501 receives a shooting direction obtaining command via the network I / F 505. The CPU 501 stores the received shooting direction obtaining command in the RAM 502.

[0280] In S502 , in response to receiving the shooting direction acquisition command, the CPU 501 acquires the current pan angle and tilt angle from the drive unit 509 via the drive I / F 508 , and stores them in the RAM 502 .

[0281] In S503, the CPU 501 reads out the current pan angle and tilt angle from the RAM 502, and transmits them to the shooting control device 100 via the network I / F 305 as shooting direction information ANGLE.

[0282] (Operation of the sub camera 600) Next, the operation of the sub-camera 600 will be described with reference to Fig. 11(d). The operation described below is realized by the CPU 601 executing a program. The sub-cameras 700 to 900 also perform similar operations.

[0283] When the sub-camera 600 is powered on, each functional block is initialized by the CPU 601, and then moving image capturing processing for supplying to the shooting control device 100 is started. The image processing unit 606 applies processing to an analog image signal obtained from the image sensor 607 based on settings for moving images to be supplied to the shooting control device 100. The image processing unit 606 sequentially stores the generated moving image data in the RAM 602. The CPU 601 reads the moving image data from the RAM 602 and supplies it to the shooting control device 100 via the network I / F 605.

[0284] The CPU 601 waits for a control command via the network I / F 305 while supplying video data to the shooting control device 100. When the CPU 601 receives a control command, it executes an operation according to the control command. Here, an operation when a pan / tilt control command PT_VALUE and a zoom control command Z_VALUE are received from the shooting control device 100 will be described.

[0285] In S401, the CPU 601 receives at least one of a pan / tilt control command PT_VALUE and a zoom control command Z_VALUE from the shooting control device 100 via the network I / F 605. The CPU 601 stores the received control command in the RAM 602.

[0286] In S402, the CPU 601 reads out the operation direction and the corresponding operation amount from the control command stored in the RAM 602, and stores them in the RAM 602. Here, in the case of a pan / tilt control command PT_VALUE, the operation direction is the pan and / or tilt direction, and the operation amount is the target angle. Also, in the case of a zoom control command Z_VALUE, the operation amount is the zoom value, and since the operation direction can be specified from the zoom value, there is no need to read out and store the operation direction.

[0287] In S403, the CPU 601 generates drive parameters for the drive unit 609 based on the operation direction and operation amount read out in S403. The CPU 601 may obtain drive parameters according to a combination of the operation direction and operation amount, for example, by using a table previously stored in the ROM 603. Note that when the operation amount is given as a target value (target angle or zoom value), the CPU 410 obtains the drive parameters from the difference with the current value.

[0288] In S404, the CPU 601 controls the drive unit 609 through the drive I / F 608 based on the drive parameters acquired in S404. As a result, the drive unit 609 changes the shooting direction of the sub camera 600 to the operation direction and angle specified by the pan / tilt control command PT_VALUE. In addition, the drive unit 609 changes the angle of view of the shooting optical system to the zoom value specified by the zoom control command Z_VALUE.

[0289] Next, the operation of the photography control device 100 controlling the photography direction (pan and tilt) and the angle of view (zoom value) of the sub-camera 600 in accordance with the role set for the sub-camera 600 will be described in more detail with reference to the flowchart shown in Fig. 19. The operation shown in the flowchart in Fig. 19 is executed as part of the operation of S205 to S207 in Fig. 11(a). The photography control device 100 executes similar operations for the sub-cameras 700 to 900.

[0290] S601 corresponds to S205, and the CPU 101 reads out the control content CAMERA_ROLE stored in the RAM 102 in S103 of FIG.

[0291] S602 to S607 are performed in S206, for example. In S602, CPU 101 determines whether the rule for the tracking subject of sub camera 600 included in the control content CAMERA_ROLE indicates the tracking subject (target subject) of main camera 500. For example, if the rule for the tracking subject of sub camera 600 has a value indicating "same as main", CPU 101 determines that the rule for the tracking subject of sub camera 600 indicates the tracking subject of main camera 500 and executes S603. On the other hand, if the rule for the tracking subject of sub camera 600 has a value indicating "different from main (left side)", CPU 101 determines that the rule for the tracking subject of sub camera 600 does not indicate the tracking subject of main camera 500 and executes S604.

[0292] In S603, CPU 101 determines to control the shooting direction of sub camera 600 so as to track a tracking subject (target subject) of main camera 500. In S604, CPU 101 determines to control the shooting direction of sub camera 600 so as to track a subject located on the left side of the subject other than the subject of interest of main camera 500.

[0293] In S605, CPU 101 determines whether the zoom control provision of sub camera 600 included in control content CAMERA_ROLE indicates control in the same phase as main camera 500. For example, if the zoom control provision of sub camera 600 has a value indicating "same phase as main", CPU 101 determines that the zoom control provision of sub camera 600 indicates control in the same phase as main camera 500 and executes S606. On the other hand, if the tracking subject provision of sub camera 600 has a value indicating "opposite phase to main", CPU 101 determines that the zoom control provision of sub camera 600 does not indicate control in the same phase as main camera 500 and executes S607.

[0294] In S606, CPU 101 determines to control the zoom value (angle of view) of sub camera 600 in the same phase as the change in the zoom value of main camera 500. In S607, CPU 101 determines to control the zoom value (angle of view) of sub camera 600 in the opposite phase to the change in the zoom value of main camera 500.

[0295] An example of control of the sub-camera when the role set for the sub-camera 600 is "main follow" will be described with reference to FIG. 20. FIG. 20 shows a schematic diagram of how the shooting control device 100 controls the shooting direction and angle of view of the sub-camera 600 when the target subject and angle of view of the main camera 500 change over time during shooting. In the figure, time passes from left to right. In FIG. 20, the zoom state is shown in three stages: "telephoto end", "middle", and "wide-angle end". This is because the range of the zoom value may differ between the sub-camera and the main camera as shown in FIG. 18. "Telephoto end" corresponds to a state where the zoom is zoomed up to the telephoto end, "wide-angle end" corresponds to a state where the zoom is zoomed down to the wide-angle end, and "middle" corresponds to a zoom state between the telephoto end and the wide-angle end, but the actual zoom value may differ between the sub-camera and the main camera. For example, when the zoom state is "telephoto end", the zoom value of the main camera is main_min, and the zoom value of the sub-camera is sub_min.

[0296] Initially, the subject of interest (tracking subject) of main camera 500 is subject B, and the zoom state is "medium." Therefore, CPU 101 determines the subject to be tracked by sub camera 600 to be subject B, and controls the shooting direction so that sub camera 600 tracks subject B. CPU 101 also controls the zoom state of sub camera 600 to "medium."

[0297] Thereafter, the subject of interest of main camera 500 is changed from subject B to subject A, and the zoom state is changed to the "telephoto end." In response to this, CPU 101 changes the subject being tracked by sub camera 600 from subject B to subject A, and controls the shooting direction so that sub camera 600 tracks subject A. CPU 101 also controls the zoom state of sub camera 600 to the "telephoto end."

[0298] Thereafter, the subject of interest of main camera 500 is changed from subject A to subject C, and the zoom state is changed to the "wide-angle end." In response to this, CPU 101 changes the subject being tracked by sub camera 600 from subject A to subject C, and controls the shooting direction so that sub camera 600 tracks subject C. CPU 101 also controls the zoom state of sub camera 600 to the "wide-angle end."

[0299] In this way, when the role of the sub camera 600 is "main follow", the CPU 101 automatically changes the tracking subject and zoom value of the sub camera 600 so as to follow the change in the focus subject and angle of view (zoom value) of the main camera 500. In the example shown in FIG. 20, the sub camera 600 is controlled so that the degree of change in the zoom state is equal to the degree of change in the zoom state of the main camera 500, but the actual zoom value may be different as long as the direction of change in the zoom value is in phase. For example, when the zoom state of the main camera 500 is at the telephoto end, the zoom state of the main camera 500 does not have to be at the telephoto end. Whether or not to match the zoom value of the sub camera 600 to the zoom value of the main camera 500 may be set using role setting information.

[0300] Next, an example of control of the sub-camera when the role set for the sub-camera 600 is "assist counter" will be described with reference to FIG. 21, which is similar to FIG.

[0301] Initially, the subject of interest (subject to be tracked) of main camera 500 is subject B, and the zoom state is "telephoto end." Therefore, CPU 101 determines the subject to be tracked by sub camera 600 to be subject A on the left side of subjects A and C other than subject B, and controls the shooting direction so that sub camera 600 tracks subject A. CPU 101 also controls the zoom state of sub camera 600 to the "wide-angle end" which is in the opposite phase to main camera 500.

[0302] Thereafter, the focus subject of main camera 500 is changed from subject B to subject A, and the zoom state is changed to "middle". In response to this, CPU 101 changes the tracking subject of sub camera 600 to subject B on the left side of subjects B and C other than subject B, and controls the shooting direction so that sub camera 600 tracks subject B. In addition, since the zoom state of main camera 500 has changed from "telephoto end" to "middle", CPU 101 controls the zoom state of sub camera 600 from "wide-angle end" to "middle" (in the opposite phase).

[0303] Thereafter, the subject of interest of main camera 500 is changed from subject A to subject C, and the zoom state is changed to "wide-angle end." In response to this, CPU 101 changes the subject being tracked by sub camera 600 to subject A on the left side of subjects A and B other than subject C, and controls the shooting direction so that sub camera 600 tracks subject A. In addition, because the zoom state of main camera 500 has changed from "middle" to "wide-angle end," CPU 101 controls the zoom state of sub camera 600 from "middle" to "telephoto end" (in the opposite phase).

[0304] In this way, when the role of sub camera 600 is "assist counter," CPU 101 automatically changes the tracking subject of sub camera 600 to a subject other than the subject of interest of main camera 500 in response to a change in the subject of interest of main camera 500. CPU 101 also automatically changes the zoom value of sub camera 600 in the opposite direction to the change in the angle of view (zoom value) of main camera 500.

[0305] 21, the zoom state of the sub camera 600 is controlled at a degree of change equivalent to that of the main camera 500, but the amount of change in the zoom value may be different as long as the direction of change in the zoom value is in the opposite phase. For example, when the zoom state of the main camera 500 is at the telephoto end, the zoom state of the main camera 500 does not have to be at the wide-angle end. The degree of change in the zoom state of the sub camera 600 relative to the degree of change in the zoom state of the main camera 500 may be set using role setting information.

[0306] (Modification) So far, we have described an example of automatically controlling the tracking subject and zoom value of the sub camera 600 based on the target subject and zoom value of the main camera 500. In the above example, the sub camera is automatically controlled to track a single subject, but it can also be automatically controlled to track multiple subjects within the shooting range.

[0307] When the role set for the sub camera 600 is "assist follow", examples of control in the case where the sub camera 600 is made to track a single subject and in the case where the sub camera 600 is made to track multiple subjects will be described with reference to FIG. 22. For ease of understanding and explanation, a case will be described in which the angle of view of the main camera 500 does not change and only the control of the subject to be tracked is executed. Also, it is assumed that the angle of view of the sub camera 600 is always capable of photographing all subjects within the photographing range 20 regardless of the photographing direction. Note that the photographing direction of the sub camera 600 shown in the top row of FIG. 22 indicates the photographing direction when a single subject is tracked.

[0308] As in FIG. 21, initially, the subject of interest (tracking subject) of the main camera 500 is subject B. Therefore, the CPU 101 determines the tracking subject of the sub camera 600 to be subject A on the left side among subjects A and C other than subject B, and controls the shooting direction so that the sub camera 600 tracks subject A. If the shooting direction is controlled so that the tracking subject is located at the center of the screen, as shown in the second row from the bottom, the image of the sub camera 600 will be an image in which subjects A to C are shifted to the right, resulting in poor balance. Therefore, when multiple subjects including the tracking subject are present in the image of the sub camera 600, the shooting direction can be controlled so that the multiple subjects are tracked. For example, the CPU 101 can control the shooting direction so that the center of gravity of the positions of the multiple subjects A to C included in the image of the sub camera 600 is tracked. As a result, the sub camera 600 shoots an image as shown in the bottom row.

[0309] In the example shown in FIG. 22, even if the sub-camera 600 tracks any one of the subjects A to C, all of the subjects A to C are captured, so the capturing direction of the sub-camera 600 remains almost constant even if the subject of interest of the main camera 500 changes.

[0310] (Variation 2) Also, after determining the tracking subject of the sub camera 600, the CPU 101 may control the sub camera 600 to focus on the tracking subject. Basically, the CPU 601 continuously controls the focusing distance so that the sub camera 600 focuses on the designated tracking subject, but the AF frame of the sub camera 600 can be set to the position of the tracking subject from the shooting control device 100. This allows the sub camera 600 to quickly and reliably focus on the tracking subject. Note that if the AF frame is set when the pan speed slows down (below a threshold), there is a high possibility that the tracking target will be located in the center of the screen, and therefore the time required for focusing can be shortened.

[0311] (Variation 3) Also, the sub camera 600 can be controlled without using the overhead camera 300. In this case, the shooting direction of the sub camera 600 can be determined from the installation positions of the main camera 500 and the sub camera 600 and the shooting direction of the main camera 500 (the orientation of the main camera 500 with respect to the tracking subject). The main camera 500 executes a subject detection process, and the CPU 101 acquires and uses the result to control the sub camera 600. For example, the CPU 101 acquires an image of the subject area from the main camera 500 as a result of the subject detection process. Then, the CPU 101 can control the sub camera 600 to execute a subject tracking process using the acquired image as a template. Alternatively, the CPU 101 may use the acquired image as a template and control the sub camera 600 to track a subject area with a low correlation with the template.

[0312] (Variation 4) Although the shooting control device 100, the role control device 400, and the main camera 500 have been described as independent devices, the functions of the shooting control device 100 and the role control device 400 can also be incorporated into the main camera 500. In this case, the video of the overhead camera 300 is supplied to the main camera 500. With such a configuration, it is possible to reduce the number of devices required to realize a multi-camera imaging system.

[0313] As described above, according to this embodiment, when the operation of the sub camera is automatically controlled based on the state and image of the main camera, automatic control is executed according to the role set for the sub camera. Therefore, the photography control device of this embodiment can realize automatic photography control with a higher degree of freedom while realizing labor saving.

[0314] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0315] The disclosure of the present embodiment includes the following imaging control device, imaging control method, imaging system, and program. (Item 1) A control means for controlling the shooting direction and the angle of view of the sub-cameras among a plurality of cameras including a main camera and two or more sub-cameras based on the role set for the sub-cameras and the target subject and the angle of view of the main camera; A determination means for determining whether the main camera satisfies a predetermined condition; The photographing control device is characterized in that, when it is determined that the main camera satisfies the condition, the control means selects one or more of the two or more sub-cameras, and changes the control content of the selected sub-camera so that it tracks and photographs a subject of interest of the main camera with settings different from those of the main camera. (Item 2) The photographing control device described in item 1, characterized in that when it is determined that the main camera satisfies the condition, the control means controls the angle of view of the selected sub-camera in the same phase as the angle of view of the main camera. (Item 3) The determination means performs the determination with respect to a plurality of conditions, The plurality of conditions are: One or more conditions regarding the brightness of the image of the main camera; One or more conditions regarding one or more of the brightness, position, moving speed, size, and depth of field of the subject area in the image of the main camera; one or more conditions regarding the movement of the main camera; One or more conditions regarding the white balance of the main camera; 3. The imaging control device according to item 1 or 2, characterized in that it includes one or more of the above. (Item 4) The photographing control device according to item 3, characterized in that the control means changes the type of settings that are made different for the selected sub-camera from those for the main camera depending on which of the multiple conditions is satisfied by the main camera. (Item 5) The shooting control device described in item 4, characterized in that the control means makes the exposure setting of the selected sub-camera different from that of the main camera when the main camera satisfies a condition regarding the brightness of the image of the main camera or the brightness of the subject area. (Item 6) The photographing control device described in item 4 or 5, characterized in that the control means makes the angle of view setting of the selected sub-camera different from that of the main camera when the main camera satisfies one or more conditions regarding the position, movement speed, and size of the subject area. (Item 7) The photographing control device according to any one of items 4 to 6, characterized in that, when the main camera satisfies the condition regarding the depth of field, the control means causes the aperture setting of the selected sub-camera to be different from that of the main camera. (Item 8) The photographing control device according to any one of items 4 to 7, characterized in that the control means, when the main camera satisfies a condition regarding the movement of the main camera, sets the shutter speed and / or photographing sensitivity of the selected sub-camera different from that of the main camera. (Item 9) The photographing control device according to any one of items 4 to 8, characterized in that, when the main camera satisfies the white balance condition, the control means causes the white balance setting of the selected sub-camera to differ from that of the main camera. (Item 10) 10. The photographing control device according to any one of items 1 to 9, wherein the control means selects a sub-camera that has been preset from among the two or more sub-cameras. (Item 11) The imaging control device according to any one of items 1 to 9, characterized in that the control means selects, from among the two or more sub-cameras, a sub-camera other than the sub-camera currently capturing the image selected by an external device. (Item 12) The photographing control device according to any one of items 1 to 9, characterized in that the control means selects a sub-camera from among the two or more sub-cameras that can photograph an image with a composition similar to that of the main camera. (Item 13) A sub-camera capable of taking pictures with a similar composition to that of the main camera, The shooting control device described in item 12 is characterized in that the sub-camera is capable of shooting a range that includes all of the subjects to be shot by the main camera, and is capable of shooting the subjects at a size that is less than a threshold value compared to the size in the image of the main camera. (Item 14) A sub-camera capable of taking pictures with a similar composition to that of the main camera, The photographing control device described in item 12 is characterized in that the sub-camera is installed in a position closest to the main camera, the distance from the main camera is equal to or less than a threshold, and the focal length range of the photographing optical system overlaps with the focal length range of the photographing optical system of the main camera by more than a threshold percentage. (Item 15) The photographing control device according to any one of items 1 to 9, characterized in that, when there is no sub-camera among the two or more sub-cameras that can photograph an image with a similar composition to that of the main camera, the control means selects a sub-camera with a predetermined lowest priority among the two or more sub-cameras. (Item 16) 16. The photographing control device according to any one of items 1 to 15, wherein when a plurality of sub-cameras are selected, the control means varies the control content for each of the selected sub-cameras. (Item 17) A photographing control method executed by a photographing control device, Among a plurality of cameras including a main camera and two or more sub-cameras, controlling the shooting direction and the angle of view of the sub-cameras based on the role set for the sub-cameras and the target subject and the angle of view of the main camera; determining whether the main camera satisfies a predetermined condition; The controlling comprises: selecting one or more of the two or more sub-cameras when it is determined that the main camera satisfies the condition; and changing the control content of the selected sub-camera so as to track and photograph the subject of interest of the main camera with a setting different from that of the main camera. 23. A photographing control method comprising: (Item 18) A control means for controlling the shooting direction and the angle of view of each of the plurality of cameras based on at least a role set for each of the plurality of cameras; a determination means for determining whether or not there is a camera in the plurality of cameras that is in a first state and is not performing an operation associated with a set role; The control means, when it is determined that a camera in the first state exists, selects a camera from the plurality of cameras other than the camera in the first state, and changes the control content of the selected sub-camera so as to perform shooting that complements the image to be shot by the camera in the first state. (Item 19) The photographing control device described in item 18, characterized in that when the control contents of the angle of view associated with the roles set for the camera in the first state and the selected camera do not match, the control means changes the control contents of the selected camera to control contents in accordance with the role set for the camera in the first state. (Item 20) The control means When the control content of the angle of view associated with the role set for the camera in the first state and the selected camera matches, and the adjustment amount of the angle of view required for the selected camera and the camera in the first state to include a subject to be photographed within a photographing range is equal to or less than a threshold value, The photographing control device described in item 18, characterized in that the control content of the selected camera is changed so that the selected camera photographs an image that includes the subject to be photographed within its photographing range in accordance with the roles set for the selected camera and the camera in the first state. (Item 21) The control means When the control content of the angle of view associated with the role set for the camera in the first state and the selected camera matches, and the adjustment amount of the angle of view required for the selected camera and the camera in the first state to include a subject to be photographed within a photographing range is not equal to or less than a threshold value, If the distance between the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state is less than a threshold value, The photographing control device described in item 18, characterized in that the control content of the selected camera is changed in accordance with the roles set for the selected camera and the camera in the first state, and so that the selected camera alternately photographs the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state. (Item 22) The control means When the control content of the angle of view associated with the role set for the camera in the first state and the selected camera matches, and the adjustment amount of the angle of view required for the selected camera and the camera in the first state to include a subject to be photographed within a photographing range is not equal to or less than a threshold value, If the distance between the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state is not less than a threshold value, 20. The photographing control device according to item 18, characterized in that the control content of the selected camera is changed to control content in accordance with the role set for the camera in the first state. (Item 23) 23. The photographing control device according to any one of items 18 to 22, characterized in that the determination means determines that a camera that is autonomously performing photographing other than the set role is a camera in the first state. (Item 24) 24. The photographing control device according to any one of items 18 to 23, wherein the determination means determines that a camera in which an abnormality has been detected is a camera in the first state. (Item 25) 25. The photographing control device according to any one of items 18 to 24, characterized in that the determination means determines that a camera that is detected to have been operated by a means outside the control of the control means is a camera in the first state. (Item 26) The photographing control device according to any one of items 18 to 25, characterized in that the control means selects from among the plurality of cameras other than the camera in the first state a camera that has a position and capability similar to that of the camera in the first state and has a predetermined priority level lower than that of the camera in the first state. (Item 27) The photographing control device described in item 26, characterized in that a camera having a position and capabilities similar to that of the camera in the first state is a camera whose installation position is less than a threshold distance from the camera in the first state and whose pan / tilt driving ranges and zoom ranges overlap by a predetermined percentage or more. (Item 28) The control means When there is no camera having a position and capability similar to that of the camera in the first state, or a camera having a role set to the camera in the first state, among the plurality of cameras other than the camera in the first state, 26. The photographing control device according to any one of items 18 to 25, characterized in that a camera having the lowest predetermined priority among the plurality of cameras and which is not a camera in the first state is selected. (Item 29) The control means If there is no camera among the plurality of cameras other than the camera in the first state that has a position and capability similar to that of the camera in the first state, The photographing control device according to any one of items 18 to 25, characterized in that, among the cameras to which the same role as that set for the camera in the first state is set, a camera having the lowest predetermined priority for the multiple cameras and having a lower priority than the camera in the first state is selected. (Item 30) The shooting control device described in any one of items 18 to 29, characterized in that when there is a main camera different from the multiple cameras, the control means controls the shooting direction and angle of view of the multiple cameras based on the role set for each of the multiple cameras and the subject of interest and angle of view of the main camera. (Item 31) 31. The photographing control device according to any one of items 18 to 30, wherein the control means notifies a user when it is determined that a camera in the first state exists. (Item 32) A photographing control method executed by a photographing control device, Controlling the shooting direction and the angle of view of each of the plurality of cameras based on at least a role set for each of the plurality of cameras; determining whether or not there is a camera in a first state that is not performing an operation associated with a set role among the plurality of cameras; The controlling comprises: selecting one or more cameras other than the camera in the first state from among the plurality of cameras when it is determined that a camera in the first state is present; and changing the control content of the selected sub-camera so as to perform shooting that complements the image to be shot by the camera in the first state. 23. A photographing control method comprising: (Item 33) An imaging control device according to any one of items 1 to 16 and 18 to 31; The plurality of cameras are communicatively connected to the imaging control device; An imaging system comprising: (Item 34) A program for causing a computer to function as each of the means possessed by the imaging control device according to any one of claims 1 to 16 and 18 to 31.

[0316] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0317] 100... Shooting control device, 300... Bird's-eye view camera, 400... Role control device, 500... Main camera, 600, 700, 800, 900... Sub-camera, 101... CPU, 102... RAM, 103... ROM, 104... Inference unit, 105... Network I / F

Claims

1. A control means for controlling the shooting direction and the angle of view of the sub-cameras among a plurality of cameras including a main camera and two or more sub-cameras based on the role set for the sub-cameras and the target subject and the angle of view of the main camera; A determination means for determining whether the main camera satisfies a predetermined condition; The control means, when it is determined that the main camera satisfies the condition, selects one or more of the two or more sub-cameras, and changes the control content of the selected sub-camera so that it tracks and photographs a subject of interest of the main camera with settings different from those of the main camera.

2. 2. The photographing control device according to claim 1, wherein the control means controls the angle of view of the selected sub-camera in phase with the angle of view of the main camera when it is determined that the main camera satisfies the condition.

3. The determination means performs the determination with respect to a plurality of conditions, The plurality of conditions are: One or more conditions regarding the brightness of the image of the main camera; One or more conditions regarding one or more of the brightness, position, moving speed, size, and depth of field of a subject area in an image of the main camera; one or more conditions regarding the movement of the main camera; One or more conditions regarding the white balance of the main camera; 2. The imaging control device according to claim 1, further comprising one or more of the following:

4. 4. The photographing control device according to claim 3, wherein the control means changes the type of settings that are to be made different for the selected sub-camera from those for the main camera depending on which of the plurality of conditions is satisfied by the main camera.

5. The photographing control device according to claim 4, characterized in that the control means makes the exposure setting of the selected sub-camera different from that of the main camera when the main camera satisfies a condition regarding the brightness of the image of the main camera or the brightness of the subject area.

6. The photographing control device according to claim 4, characterized in that the control means causes the angle of view setting of the selected sub-camera to be different from that of the main camera when the main camera satisfies one or more conditions regarding the position, movement speed, and size of the subject area.

7. 5. The photographing control device according to claim 4, wherein the control means, when the main camera satisfies the condition regarding the depth of field, causes an aperture value setting of the selected sub-camera to differ from that of the main camera.

8. 5. The photographing control device according to claim 4, wherein the control means, when the main camera satisfies a condition regarding the movement of the main camera, makes the shutter speed and / or photographing sensitivity setting of the selected sub-camera different from that of the main camera.

9. 5. The photographing control device according to claim 4, wherein the control means, when the main camera satisfies the condition regarding white balance, causes a white balance setting of the selected sub-camera to differ from that of the main camera.

10. 2. The photographing control device according to claim 1, wherein the control means selects a preset sub-camera from among the two or more sub-cameras.

11. 2. The imaging control device according to claim 1, wherein the control means selects, from among the two or more sub-cameras, a sub-camera other than a sub-camera currently capturing an image selected by an external device.

12. 2. The photographing control device according to claim 1, wherein the control means selects, from among the two or more sub-cameras, a sub-camera capable of photographing an image with a similar composition to that of the main camera.

13. A sub-camera capable of taking pictures with a similar composition to that of the main camera, The photographing control device according to claim 12, characterized in that the sub-camera is capable of photographing a range including all of the subjects photographed by the main camera, and is capable of photographing the subjects at a size whose difference from the size in the image of the main camera is less than a threshold value.

14. A sub-camera capable of taking pictures with a similar composition to that of the main camera, The photographing control device according to claim 12, characterized in that the sub-camera is installed closest to the main camera, the distance from the main camera is equal to or less than a threshold, and the focal length range of the photographing optical system overlaps with the focal length range of the photographing optical system of the main camera by a threshold percentage or more.

15. 2. The photographing control device according to claim 1, wherein, when there is no sub-camera among the two or more sub-cameras that can photograph an image with a similar composition to that of the main camera, the control means selects a sub-camera having a predetermined lowest priority among the two or more sub-cameras.

16. 2. The photographing control device according to claim 1, wherein, when a plurality of sub-cameras are selected, the control means changes the control content for each of the selected sub-cameras.

17. A photographing control method executed by a photographing control device, Among a plurality of cameras including a main camera and two or more sub-cameras, a shooting direction and a field angle of the sub-camera are controlled based on a role set for the sub-camera and a target object and a field angle of the main camera; determining whether the main camera satisfies a predetermined condition; The controlling comprises: selecting one or more of the two or more sub-cameras when it is determined that the main camera satisfies the condition; and changing the control content of the selected sub-camera so as to track and photograph the subject of interest of the main camera with a setting different from that of the main camera.

23. A photographing control method comprising:

18. A control means for controlling the shooting direction and the angle of view of each of the plurality of cameras based on at least a role set for each of the plurality of cameras; a determination means for determining whether or not there is a camera in a first state that is not performing an operation associated with a set role among the plurality of cameras; The control means, when it is determined that a camera in the first state exists, selects a camera from the plurality of cameras other than the camera in the first state, and changes the control content of the selected sub-camera so as to perform shooting that complements the image to be shot by the camera in the first state.

19. The photographing control device according to claim 18, characterized in that, when the control contents of the angle of view associated with the roles set for the camera in the first state and the selected camera do not match, the control means changes the control contents of the selected camera to control contents in accordance with the role set for the camera in the first state.

20. The control means When the control content of the angle of view associated with the role set for the camera in the first state and the selected camera matches, and the adjustment amount of the angle of view required for the selected camera and the camera in the first state to include a subject to be photographed within a photographing range is equal to or less than a threshold value, The photographing control device according to claim 18, characterized in that the control content of the selected camera is changed so that the selected camera photographs an image that includes the subject to be photographed within its photographing range in accordance with the roles set for the selected camera and the camera in the first state.

21. The control means When the control content of the angle of view associated with the role set for the camera in the first state and the selected camera matches, and the adjustment amount of the angle of view required for the selected camera and the camera in the first state to include a subject to be photographed within a photographing range is not equal to or less than a threshold value, If the distance between the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state is less than a threshold value, The photographing control device according to claim 18, characterized in that the control content of the selected camera is changed so that the selected camera alternately photographs a subject to be photographed by the selected camera and a subject to be photographed by the camera in the first state in accordance with the roles set for the selected camera and the camera in the first state.

22. The control means When the control content of the angle of view associated with the role set for the camera in the first state and the selected camera matches, and the adjustment amount of the angle of view required for the selected camera and the camera in the first state to include a subject to be photographed within a photographing range is not equal to or less than a threshold value, If the distance between the subject to be photographed by the selected camera and the subject to be photographed by the camera in the first state is not less than a threshold value, 20. The photography control device according to claim 18, wherein the control content of the selected camera is changed to a control content according to a role set for the camera in the first state.

23. 19. The photographing control device according to claim 18, wherein the determining means determines that a camera that is autonomously performing photographing other than a role that has been set is the camera in the first state.

24. 19. The photographing control device according to claim 18, wherein the determining means determines that a camera in which an abnormality has been detected is the camera in the first state.

25. 19. The photographing control device according to claim 18, wherein the determining means determines that a camera that has been detected as having been operated by a means outside the control of the control means is a camera in the first state.

26. The photographing control device according to claim 18, characterized in that the control means selects from among the plurality of cameras other than the camera in the first state a camera that has a position and capabilities similar to those of the camera in the first state and has a predetermined priority level lower than that of the camera in the first state.

27. The shooting control device according to claim 26, characterized in that a camera having a position and capabilities similar to that of a camera in the first state is a camera whose installation position is less than a threshold distance from the camera in the first state and whose pan / tilt driving ranges and zoom ranges overlap by a predetermined percentage or more.

28. The control means When there is no camera having a position and capability similar to that of the camera in the first state, or a camera having a role set to the camera in the first state, among the plurality of cameras other than the camera in the first state, 19. The imaging control device according to claim 18, wherein a camera having the lowest predetermined priority among the plurality of cameras and which is not a camera in the first state is selected.

29. The control means If there is no camera among the plurality of cameras other than the camera in the first state that has a position and capability similar to that of the camera in the first state, The photographing control device according to claim 18, characterized in that, among the cameras to which the same role as that set for the camera in the first state is set, a camera having the lowest predetermined priority for the multiple cameras and a lower priority than the camera in the first state is selected.

30. 19. The photographing control device according to claim 18, characterized in that, when there is a main camera different from the plurality of cameras, the control means controls the photographing direction and angle of view of the plurality of cameras based on the role set for each of the plurality of cameras and the subject of interest and angle of view of the main camera.

31. 19. The photographing control device according to claim 18, wherein the control means notifies a user when it is determined that a camera in the first state exists.

32. A photographing control method executed by a photographing control device, Controlling the shooting direction and the angle of view of each of the plurality of cameras based on at least a role set for each of the plurality of cameras; determining whether or not there is a camera in a first state that is not performing an operation associated with a set role among the plurality of cameras; The controlling comprises: selecting one or more cameras other than the camera in the first state from among the plurality of cameras when it is determined that a camera in the first state is present; and changing the control content of the selected sub-camera so as to perform shooting that complements the image to be shot by the camera in the first state.

23. A photographing control method comprising:

33. An imaging control device according to any one of claims 1 to 16 and 18 to 31; The plurality of cameras are communicatively connected to the imaging control device; An imaging system comprising:

34. A program for causing a computer to function as each of the means included in the imaging control device according to any one of claims 1 to 16 and 18 to 31.

Images