Photographing control device and photographing control method, and imaging system

Abstract

To provide a photographing control device and a photographing control method capable of automatic photographing control with a higher degree of freedom, while achieving power saving.SOLUTION: A photographing control device acquires information from the state of or images from a main camera, of a plurality of cameras. The photographing control device controls the operation of a sub camera, of the plurality of cameras, on the basis of the information and a role set to the sub camera. The photographing control device acquires, as the information, information on a target subject of the main camera and information on the angle of view of the main camera. The photographing control device controls the photographing direction and angle of view of the sub camera on the basis of the information and the role set to the sub camera.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging control device, an imaging control method, and an imaging system, and more particularly to a technique for controlling a plurality of imaging devices. [Background technology]

[0002] Patent Document 1 describes an imaging system in which a plurality of cameras are divided into main cameras and sub-cameras, and the sub-cameras are controlled so as to capture the same subject as that of the main camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-25248 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, the photographing of the sub-camera can be automatically controlled, thereby realizing labor savings. However, there is room for improvement in the degree of freedom of photographing control of the sub-camera.

[0005] Therefore, in one aspect, the present invention provides an imaging control device and an imaging control method that realizes labor saving and allows automatic imaging control with a higher degree of freedom. [Means for solving the problem]

[0006] In one aspect, the present invention provides a shooting control device having an acquisition means for acquiring information from the state or image of a main camera among a plurality of cameras, and a control means for controlling the operation of a sub-camera among the plurality of cameras based on the role set for the sub-camera and the information, wherein the acquisition means acquires information on a subject of interest for the main camera and information on the angle of view of the main camera as information, and the control means controls the shooting direction and angle of view of the sub-camera based on the role set for the sub-camera and the information. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an imaging control device and an imaging control method that realize labor saving and enable automatic imaging control with a higher degree of freedom. [Brief explanation of the drawings]

[0008] [Figure 1] Schematic diagram of an imaging system according to a first embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of the functional configuration of each device in an imaging system according to a first embodiment. [Figure 3] FIG. 1 is a diagram illustrating the main operations and signal flows of an imaging control device according to an embodiment. [Figure 4] FIG. 10 shows examples of roles and control contents that can be set for a sub-camera in an embodiment. [Figure 5] Flowchart of role determination processing in the first embodiment [Figure 6] 1 is a flowchart illustrating the operation of each device in an imaging system according to an embodiment. [Figure 7] FIG. 1 is a diagram for explaining coordinate transformation in an embodiment. [Figure 8] 1 is a diagram relating to subject detection and coordinate transformation in an embodiment; [Figure 9] Schematic diagram of the operation control of the sub-camera in the first embodiment [Figure 10] 10 is a schematic diagram of another operational control of the sub-camera in the first embodiment; [Figure 11]FIG. 10 is a diagram for explaining pan value calculation in an embodiment. [Figure 12] FIG. 1 is a diagram for explaining tilt value calculation in an embodiment. [Figure 13] FIG. 10 is a diagram showing an example of mapping of zoom values ​​between the main camera and the sub camera in the first embodiment; [Figure 14] 10 is a flowchart showing a process for determining control contents according to the role of the sub-camera in the first embodiment; [Figure 15] Schematic diagram of control according to the role of the sub-camera in the first embodiment [Figure 16] Schematic diagram of control according to the role of the sub-camera in the first embodiment [Figure 17] Schematic diagram of a tracking operation of a plurality of subjects in a modified example of the first embodiment. [Figure 18] Schematic diagram of an imaging system according to a second embodiment. [Figure 19] FIG. 10 shows examples of roles and control contents that can be set in the second embodiment. [Figure 20] FIG. 11 is a schematic diagram illustrating an example of control related to a subject tracked by a sub-camera in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] <First embodiment> (Overview of the multi-camera imaging system) 1 is a schematic diagram showing an example of the configuration of a multi-camera imaging system 10 (hereinafter simply referred to as an imaging system) according to this embodiment. The imaging system 10 has multiple cameras 300, 400, and 500, an imaging control device 100, and a role control device 600. The multiple cameras 300, 400, and 500, the imaging control device 100, and the role control device 600 are communicably connected via a communication network 700.

[0011] The communication network 700 complies with known wired or wireless communication standards such as the IEEE802.3 series, IEEE802.11 series, etc. Each of the multiple cameras 300, 400, 500, the shooting control device 100, and the role control device 600 has a communication interface that complies with the standard of the communication network 700.

[0012] Of the multiple cameras 300, 400, and 500, camera 300 captures the entirety of a predetermined capture range. The capture range is set as the range in a studio, for example, where the subject to be captured may be present. Therefore, the video captured by camera 300 captures all of the subjects within the capture range.

[0013] The purpose of camera 300 is to capture images for detecting subjects within the capture range. Therefore, the capture direction and angle of view of camera 300 are determined according to the position of camera 300 and the capture range, and are basically fixed during capture. Furthermore, camera 300 preferably captures the entire capture range so that it is not obscured by objects outside the capture range. For this reason, camera 300 is installed in a position that overlooks the entire capture range. To distinguish camera 300 from other cameras 400 and 500, whose capture direction and angle of view during capture are not basically fixed, camera 300 will be referred to as a bird's-eye view camera below. However, the installation position of camera 300 is not limited to a position that overlooks the capture range. The operation of bird's-eye view camera 300 can be controlled by the capture control device 100.

[0014] The cameras 400 and 500 are, for example, PTZ cameras, and their operations, including the shooting direction (pan and tilt angles) and angle of view (zoom), can be controlled from an external device. Here, it is assumed that the user of the imaging system controls the operation of the camera 500, and the shooting control device 100 controls the operation of the camera 400. Hereinafter, the camera 500 will be referred to as the main camera and the camera 400 as the sub-camera, because the shooting control device 100 controls the operation of the camera 400 based on the state of the camera 500. For ease of explanation and understanding, only one sub-camera 400 is shown, but there may be multiple sub-cameras. The main camera 500 may be directly operated by the user. The cameras 400 and 500 may be configured so that the shooting direction (pan and tilt angles) can be controlled by attaching the camera body to a camera platform. The cameras 400 and 500 may also be configured so that a zoomable interchangeable lens is attached to the camera body.

[0015] In this embodiment, it is assumed that the role control device 600 has an operator. The shooting control device 100 may also have an operator (user), but this is not required. The operator of the role control device 600 may also be the user of the shooting control device 100. Since the shooting control device 100 controls the shooting of the overhead camera 300 and the sub-camera 400, no photographer is required. It is assumed that the main camera 500 has an operator or photographer. In this way, a configuration that does not require operators or photographers for some devices can achieve labor savings.

[0016] 1 illustrates communication of all signals over the communication network 700, but video signals and control signals may be communicated by different methods. For example, each of the multiple cameras 300, 400, and 500 may directly supply a video signal to the shooting control device 100 via a cable. The cameras 300, 400, and 500 and the shooting control device 100 have communication circuits that comply with the video signal standard. Examples of video signal standards include, but are not limited to, the SDI (Serial Digital Interface) standard and HDMI (High-Definition Multimedia Interface) (registered trademark).

[0017] The photography control device 100 detects a subject from the video signal received from the overhead camera 300. The photography control device 100 determines the photography direction and angle of view of the sub-camera based on the subject detection result, the state of the main camera 500, and the role set for the sub-camera. The photography control device 100 transmits a control command including the determined photography direction and angle of view to the sub-camera 400. By changing the role setting, it is possible to change the method for determining the photography direction and angle of view of the sub-camera 400, thereby increasing the degree of freedom in controlling the operation of the sub-camera 400.

[0018] (Example of functional configuration of each device) Figure 2 is a block diagram showing an example of the functional configuration of each device constituting the multi-camera imaging system shown in Figure 1. Note that the configuration expressed as a functional block in the drawing can be realized by an integrated circuit such as an ASIC or FPGA, by a discrete circuit, or by a combination of a memory and a processor that executes a program stored in the memory. Also, one functional block may be realized by multiple integrated circuit packages, or multiple functional blocks may be realized by a single integrated circuit package. Also, the same functional block may be implemented in different configurations depending on the operating environment, required capabilities, etc.

[0019] (imaging control device 100) First, we will explain an example of the functional configuration of the imaging control device 100. The imaging control device 100 may be a general-purpose computer device such as a personal computer or a workstation. The imaging control device 100 has a configuration in which a CPU 101, a RAM 102, a ROM 103, an inference unit 104, a network interface (I / F) 105, a user input unit 106, and a display unit 108 are interconnected via an internal bus 110.

[0020] The CPU 101 is a microprocessor capable of executing programmed instructions. The CPU 101, for example, loads a program stored in a ROM 103 into a RAM 102 and executes the program, thereby realizing the functions of the imaging control device 100, which will be described later. The CPU 101 can, for example, realize the functions of the imaging control device 100 by executing an imaging control application that runs on an operating system (OS).

[0021] The RAM 102 is used to load programs to be executed by the CPU 101 and to temporarily store data to be processed by the CPU 101, data currently being processed, etc. A part of the RAM 102 may be used as a video memory for the display unit 108.

[0022] The ROM 103 is a rewritable nonvolatile memory, and stores programs (OS and applications) executed by the CPU 101, user data, and the like.

[0023] The inference unit 104 executes a subject area detection process using a machine learning model on the image captured by the overhead camera 300. The inference unit 104 can be implemented using a hardware circuit capable of quickly executing calculations of the machine learning model, such as a GPU (Graphics Processing Unit) or an NPU (Neural Network Processing Unit). Alternatively, the inference unit 104 may be implemented using a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array). The CPU 101 may execute a program to realize the functions of the inference unit 104.

[0024] The machine learning model may be a convolutional neural network (CNN) trained according to the type of object to be detected. Here, the inference unit 104 detects a human body region or a human face region as an object region from an input image. The inference unit 104 also outputs the position and size of a rectangular region inscribed in the object region and the detection reliability for each detected object region. It is also possible to use multiple types of machine learning models to perform detection processing of different types of object regions on the same input image. It is also possible for the inference unit 104 to perform detection processing of the object region using a known method that does not use a machine learning model. The inference unit 104 can detect the object region using, for example, a method using local features such as SIFT or SURF, or a method using pattern matching.

[0025] The network I / F 105 is an interface for connecting the photography control device 100 to the communication network 700. The photography control device 100 (CPU 101) can communicate with external devices on the communication network 700, such as the overhead camera 300, sub-camera 400, main camera 500, and role control device 600, via the network I / F 105. The photography control device 100 may also communicate with external devices via other communication interfaces (USB, Bluetooth (registered trademark), etc.) not shown.

[0026] To communicate with each device (overhead camera 300, sub camera 400, main camera 500, role control device 600) on communication network 700, CPU 101 acquires the network address of each device at any timing and stores it in RAM 102. CPU 101 also acquires information about each device (device type, model name, etc.) at any timing (for example, at the time of the first communication) and stores it in RAM 102. In this way, CPU 101 is assumed to know at least the identification information and device type of the overhead camera 300, sub camera 400, main camera 500, and role control device 600. Note that the user may be able to assign any name to each device.

[0027] The user input unit 106 is an input device (not shown) such as a mouse, a keyboard, a touch panel, etc. The imaging control device 100 receives instructions from the user through the user input unit 106.

[0028] The display unit 108 is a display device such as a liquid crystal display (LCD), etc. The display unit 108 displays a GUI screen provided by the OS, a shooting control application, etc.

[0029] (Overhead camera 300) Next, an example of the functional configuration of the overhead camera 300 will be described. The CPU 301 is a microprocessor capable of executing programmed instructions. For example, the CPU 301 loads a program stored in the ROM 303 into the RAM 302 and executes it to control the operation of each functional block and realize the functions of the overhead camera 300, which will be described later.

[0030] The RAM 302 is used to load programs executed by the CPU 301 and to temporarily store data to be processed by the CPU 301, data currently being processed, etc. The RAM 302 may also be used as a buffer for video signals obtained by shooting.

[0031] The ROM 308 is a rewritable non-volatile memory. The ROM 308 stores programs executed by the CPU 301, setting values ​​for the overhead camera 300, user data, etc. The ROM 308 can also be used as a recording destination for video signals. The ROM 308 may include an internal memory and a removable memory card.

[0032] The image sensor 307 has a photographing optical system and an image sensor. The image sensor may be, for example, a known CCD or CMOS color image sensor with a primary color Bayer array color filter. The image sensor has a pixel array in which multiple pixels are arranged two-dimensionally, and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charge according to the amount of incident light through photoelectric conversion. By reading out from each pixel a signal having a voltage according to the amount of charge accumulated during the exposure period, a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface can be obtained.

[0033] The image processing unit 306 applies predetermined signal processing and image processing to the analog image signal output by the image sensor 307, generating signals and image data according to the application, and acquiring and / or generating various types of information.

[0034] The processing applied by the image processing unit 306 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, and the like. Color interpolation is performed when a color filter is provided on the image sensor 307, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data. Color interpolation is also called demosaic processing. The correction processing can include white balance adjustment, tone correction, correction of image degradation caused by optical aberrations in the imaging optical system (image restoration), correction of the effects of vignetting in the imaging optical system, color correction, and the like. The data processing may include processes such as cutting out an area (trimming), compositing, scaling, encoding and decoding, generating header information (generating a data file), etc. The data processing also includes generating a video signal to be output externally and generating video data to be recorded in the ROM 308. The evaluation value calculation process may include processes such as generating signals and evaluation values ​​used for autofocus (AF) and generating evaluation values ​​used for automatic exposure (AE). AF and AE are executed by the CPU 301. Special effect processing can include adding a blur effect, changing color tones, relighting, and the like. Note that these are examples of processing that can be applied by the image processing unit 306, and do not limit the processing that can be applied by the image processing unit 306. The image processing unit 306 outputs the acquired or generated information and data to the CPU 301, RAM 302, etc. depending on the application.

[0035] The type and settings of the processing applied by the image processing unit 306 can be controlled by sending commands from the photography control device 100 to the overhead camera 300.

[0036] The network I / F 305 is an interface for connecting the overhead camera 300 to the communication network 700. The overhead camera 300 (CPU 301) can communicate with external devices on the communication network 700, such as the shooting control device 100, sub-camera 400, main camera 500, and role control device 600, via the network I / F 305. The overhead camera 300 may also communicate with external devices via other communication interfaces (USB, Bluetooth, etc.) not shown.

[0037] (Sub-camera 400) Next, a description will be given of an example of the functional configuration of the sub-camera 400. Functional blocks with the same names in the sub-camera 400 and the overhead camera 300 have the same functions, and their description will be omitted.

[0038] As described above, the sub-camera 400 is a PTZ camera, and the shooting direction and angle of view can be controlled externally. Therefore, the sub-camera 400 has a driving unit 409 that is capable of panning, tilting, and zooming, and a driving I / F 408. The driving I / F 408 is a communication interface between the driving unit 409 and the CPU 401.

[0039] The drive unit 409 has a pan / tilt mechanism that supports the sub-camera 400 so that it can pan and tilt, a zoom mechanism that changes the angle of view of the imaging optical system, and motors that drive these mechanisms. The zoom mechanism may use the image processing unit 406 to enlarge and reduce the image. The drive unit 409 drives the motor in accordance with instructions received from the CPU 401 via the drive I / F 408, and adjusts the optical axis direction and angle of view of the imaging optical system.

[0040] (Main camera 500) Next, an example of the functional configuration of the main camera 500 will be described. Functional blocks with the same name in the main camera 500 and the sub-camera 400 have the same function, and their description will be omitted. The main camera 500 is operated by the user. Here, it is assumed that the user remotely controls the main camera 500 by sending commands via the communication network 700. However, if the main camera 500 is not a PTZ camera, the user may directly operate the main camera 500.

[0041] The imaging control device 100 (CPU 101) can acquire information on the imaging direction and angle of view of the sub camera 400 and the main camera 500 from the sub camera 400 and the main camera 500 via the network I / F 505. The imaging direction may be the pan and tilt angles of the drive units 409 and 509, with a predetermined reference direction set to 0°. The reference direction may be the direction directly facing the imaging range.

[0042] (Role control device 600) Next, an example of the functional configuration of the role control device 600 will be described. The CPU 601 is a microprocessor capable of executing programmed instructions. For example, the CPU 601 loads a role setting program stored in the ROM 603 into the RAM 602 and executes it to control the operation of each functional block and realize the functions of the role control device 600.

[0043] The RAM 602 is used to load programs executed by the CPU 601 and to temporarily store data to be processed by the CPU 601, data currently being processed, etc. A part of the RAM 602 may be used as a video memory for the display unit 608.

[0044] The ROM 603 is a rewritable non-volatile memory, and stores the programs executed by the CPU 601, setting values ​​for the role control device 600, user data, and the like.

[0045] The user input unit 611 is an input device such as a button, a dial, a joystick, a touch panel, etc. The role control device 600 receives user instructions regarding the setting of the role of the sub camera 400 through the user input unit 611.

[0046] The network I / F 605 is an interface for connecting the role control device 600 to the communication network 700. The role control device 600 (CPU 601) can communicate with external devices on the communication network 700, such as the overhead camera 300, the sub-camera 400, and the shooting control device 100, via the network I / F 605. Note that the role control device 600 may also communicate with external devices via other communication interfaces (USB, Bluetooth, etc.) not shown.

[0047] The display unit 608 is a display device such as a liquid crystal display (LCD), etc. The display unit 608 displays a GUI screen provided by the OS, the role setting application, etc.

[0048] The role control device 600 stores role setting information in, for example, the ROM 603. The role setting information is information in which identification information of the sub camera 400 is associated with information indicating the role that has been set. The CPU 601 executes a role setting application to display a role setting screen on the display unit 608. The role setting screen displays, for example, identification information of the sub camera 400 (such as a network address or a name set by the user) and the name of the currently set role in association with each other. The initial value of the currently set role may be a default role that has been set in advance. The user can change the current role that is displayed in association with a desired sub camera 400 by operating the user input unit 611.

[0049] When a user operation indicating the end of the setting operation, such as an operation of an OK button included in the role setting screen, is detected, the CPU 601 updates the role setting information stored in the ROM 103 in accordance with the content of the role setting screen.

[0050] When the CPU 601 receives a role acquisition command via the network I / F 605, it reads out the role setting information stored in the ROM 103 and transmits it to the sender of the role acquisition command.

[0051] 1 and 2, the role control device 600 is depicted as an independent device, but for example, a shooting control application executed by the shooting control device 100 may provide the same function as the role control device 600. Alternatively, a role may be set directly to the sub-camera 400, and the shooting control device 100 may acquire the role assigned to the sub-camera 400 from the sub-camera 400.

[0052] The role that can be set for the sub camera 400 is a predetermined role that determines how information obtained from the main camera 500 is used to control the operation of the sub camera 400. Here, as an example, it is assumed that information from the main camera is used to control the tracking subject and zoom operation of the sub camera 400.

[0053] Fig. 4 shows an example of the types of roles that can be set for the sub-camera 400 and the control contents associated with the roles. The control contents for each role can be stored in the ROM 603 of the role control device 600 and the ROM 103 of the shooting control device 100, for example, in the table format shown in Fig. 4. Here, it is assumed that any of "main follow," "main counter," "assist follow," and "assist counter" can be set as the role ROLE. If there are multiple sub-cameras 400, a role can be set for each sub-camera.

[0054] For a sub-camera 400 whose ROLE is "main follow," the photography control device 100 (CPU 101) sets the same tracking subject as the main camera 500, and when the main camera 500 is zoomed, the photography control device 100 also performs zoom control of the sub-camera 400 in the same phase. Here, "in phase" means that the zoom direction (telephoto direction or wide-angle direction) is the same, i.e., the direction of the change in the angle of view is the same. On the other hand, "opposite phase" means that the zoom direction (telephoto direction or wide-angle direction) is opposite, i.e., the direction of the change in the angle of view is opposite. Note that even if the zoom direction is in the same phase, the angle of view does not have to be the same as that of the main camera 500, and whether in the same phase or opposite phase, the degree of change in zoom (such as the speed or rate of change) does not have to be the same as that of the main camera 500.

[0055] For the sub-camera 400 whose role is "main counter," the photography control device 100 (CPU 101) sets the same tracking subject as the main camera 500, and when the main camera 500 is zoomed, the photography control device 100 (CPU 101) controls the sub-camera 400 to zoom down in the opposite direction. Therefore, when the main camera 500 is zoomed up, the photography control device 100 (CPU 101) controls the sub-camera 400 with this role to zoom down. Note that zooming up refers to changing the zoom in the telephoto direction (toward the telephoto end), and zooming down refers to changing the zoom in the wide-angle direction (toward the wide-angle end). When zoom control is performed by the image processing unit 406, zooming up refers to reducing the area cut out from the image and increasing the magnification rate of the cut-out area compared to before the area was changed. On the other hand, zooming down refers to increasing the area cut out from the image and decreasing the magnification rate of the cut-out area compared to before the area was changed.

[0056] For the sub-camera 400 whose ROLE is "assist follow", the photography control device 100 (CPU 101) sets a tracking subject different from that of the main camera 500, and when the main camera 500 is zoomed, the photography control device 100 (CPU 101) performs zoom control of the sub-camera 400 in the same phase.

[0057] For the sub-camera 400 whose role is "assist counter," the photography control device 100 (CPU 101) sets a tracking subject that is different from that of the main camera 500. Furthermore, when the main camera 500 is zoomed, the photography control device 100 performs zoom control on the sub-camera 400 in the opposite phase.

[0058] Here, for sub-cameras 400 with roles of "assist follow" and "assist counter," a subject that is present on the left side of the main camera 500 in the image, other than the subject of interest, is set as the tracking subject of the sub-camera 400. Note that the tracking subject of the sub-camera 400 may be set according to other conditions. For example, a subject that is present on the right, above, or below the main camera 500 in the image, other than the subject of interest, may be set as the tracking subject of the sub-camera. Alternatively, a subject that is present at the forefront or the back of the main camera 500 in the image, other than the subject of interest, may be set as the tracking subject of the sub-camera 400.

[0059] Also, only one of setting the subject to be tracked and zoom control may be performed, or another control item may be added.

[0060] In the role setting information stored in the ROM 603 by the role control device 600, information indicating the role ROLE (such as the above-mentioned type name or the number assigned to the type) is associated with identification information of the sub camera 400. The CPU 101 of the photography control device 100 acquires the role setting information from the role control device 600, and executes operation control of the sub camera 400 according to the type of role ROLE set for the sub camera 400.

[0061] The role control device 600 may notify an external device (for example, the shooting control device 100) when there is a change in the role setting for the sub-camera 400. This allows the change in the role setting to be immediately reflected in the operation control of the sub-camera 400.

[0062] <Explanation of the operation of each device> Next, the operation of each device in the multi-camera imaging system will be described. Here, the imaging control device 100 automatically controls the imaging operation of the sub-camera 400 based on the image from the overhead camera 300, information obtained from the main camera 500, and the role set for the sub-camera 400.

[0063] 3 is a diagram illustrating a series of processes performed by the shooting control device 100 when controlling the operation of the sub-camera 400, focusing on the main operations and signal flow. The functional blocks shown in the shooting control device 100 schematically illustrate the main operations and correspond to the main functions provided by the shooting control application. Each functional block in FIG. 3 is realized by a combination of the CPU 101 that executes the shooting control application and one or more of the functional blocks of the shooting control device 100 shown in FIG. 2.

[0064] Fig. 5 is a flowchart showing the operation of CPU 101 as role determination unit 120. Fig. 6(a) to Fig. 6(d) are flowcharts relating to the operations of the shooting control device 100, overhead camera 300, main camera 500, and sub-camera 400, respectively.

[0065] In the following description, it is assumed that the three-dimensional coordinate values ​​of the viewpoint position and the shooting direction (optical axis direction) of the overhead camera 300 are known to the shooting control device 100. It is also assumed that known position information, such as the three-dimensional coordinate values ​​of the viewpoint positions of the sub-camera 400 and main camera 500 and the coordinate values ​​of markers placed in the shooting range, is stored in advance in ROM 103 as default position information REF_POSI. It is also assumed that the coordinate system of the position is predetermined depending on the type of position.

[0066] (Operation of the role determination unit 120) First, the operation of the CPU 101 as the role determination unit 120 in Fig. 3 will be described with reference to the flowchart shown in Fig. 5. The operation described below is realized by the CPU 101 executing a shooting control application.

[0067] 5, there is no particular limitation on the timing for starting the operations shown in the flowchart of Fig. 5, but they are executed at least before starting control of the shooting operation of the sub-camera 400. They are also executed when a notification is received from the role control device 600 via the network I / F 105 that the role setting for the sub-camera 400 has been changed.

[0068] In S101, the CPU 101 as the role determination unit 120 acquires the role ROLE (role setting information) corresponding to the sub-camera 400 from the role control device 600. The CPU 101 can acquire the above-mentioned role setting information from the role control device 600 by, for example, transmitting a role acquisition command to the role control device 600 via the network I / F 105. The CPU 101 stores the acquired role setting information in the RAM 102.

[0069] In S103, CPU 101 refers to the role setting information stored in RAM 102 based on the identification information of sub camera 400, and acquires operation control content for sub camera 400. Then, CPU 101 as role determination section 120 transmits the acquired operation control content (CAMERA_ROLE) to tracking subject determination section 123. In practice, CPU 101 stores the operation control content in a specific area of ​​RAM 102, and refers to it when functioning as tracking subject determination section 123.

[0070] In S104, the CPU 101 as the role determination unit 120 transmits the acquired operation control content (CAMERA_ROLE) to the zoom value calculation unit 125. In practice, the CPU 101 stores the operation control content in a specific area of ​​the RAM 102 and refers to it when functioning as the zoom value calculation unit 125.

[0071] (Operation of the imaging control device 100) Next, the operation of the shooting control device 100 to control shooting by the sub-camera 400 will be described with reference to Figures 3 and 6(a). The operation described below corresponds to the operation of the CPU 101 as the recognition unit 121, target subject determination unit 122, tracking subject determination unit 123, pan / tilt value calculation unit 124, and zoom value calculation unit 125 in Figure 3. Note that the operation described below is realized by the CPU 101 executing a shooting control application.

[0072] In S201, CPU 101 transmits a shooting instruction command to overhead camera 300 using a predetermined protocol via network I / F 105. In response to this command, overhead camera 300 starts supplying a video signal (video data) IMG to video input unit 107. CPU 101 starts storing the video signal received by video input unit 107 in RAM 102, and then executes S202.

[0073] In S202, CPU 101 acquires information ANGLE indicating the shooting direction from main camera 500. Specifically, CPU 101 sends a shooting direction acquisition command to main camera 500 via network I / F 105 using a predetermined protocol. In response to the shooting direction acquisition command, CPU 501 of main camera 500 transmits information ANGLE indicating the current shooting direction of main camera 500 to shooting control device 100. Information ANGLE may be, for example, the pan and tilt angles of drive unit 509. CPU 101 stores the acquired information ANGLE in RAM 102.

[0074] In S203, the recognition unit 121 executes the following process. (1) Apply subject area detection processing to the input frame image and store the detection results. (2) For each detected subject area, coordinate conversion is performed on the position information (image coordinates). (3) Apply identification processing to each detected subject area to identify the identification information (add information for the identification processing in the case of a new subject). (4) For each detected subject area, the identification information ID[n] and the position information POSITION[n] are associated and stored.

[0075] The recognition unit 121 is mainly realized by the CPU 101 and the inference unit 104. The CPU 101 reads one frame of the video received from the overhead camera 300 from the RAM 102 and inputs it to the inference unit 104.

[0076] The operation of the recognition unit 121 will be explained below step by step. (1) First, the inference unit 104 inputs a frame image into a machine learning model to detect a subject region. The inference unit 104 stores in RAM 102 the position and size of each detected subject region, which are output by the machine learning model as detection results, and the detection reliability. The position and size of the subject region may be any information that can identify the position and size of a rectangular region inscribed with the subject region. Here, the center coordinate of the bottom side of the rectangular region, as well as the width and height, are used as the position and size of the subject region.

[0077] Furthermore, the inference unit 104 stores the detection result for the first frame image in RAM 102 in association with the subject identification information ID[n]. Here, n is the subject number and is an integer ranging from 1 to the total number of detected subject areas. Furthermore, the inference unit 104 stores the subject areas detected from the first frame image in RAM 102 in association with the subject identification information ID[n] as templates for identifying individual subjects. If template matching is not used to identify subjects, it is not necessary to store templates.

[0078] Fig. 8(a) shows an example of the result of subject detection processing by inference unit 104 for the image captured by overhead camera 300 shown in Fig. 7(a). Here, the areas of human subjects A to C present within shooting range 20 are detected, and the coordinates (foot coordinates) of the center of the bottom side of the rectangular area inscribed with the subject areas are output as positions.

[0079] For coordinate transformation, which will be described later, if a marker is placed at a known position within the shooting range 20 as shown in FIG. 7(b), the CPU 101 detects the image of the marker included in the frame image (FIG. 7(a)) and stores the position in the RAM 102. The detection of the marker image may also be performed by the inference unit 104. The detection of the marker image can be performed by any known method, such as pattern matching using a marker template. The marker image may also be detected using a pre-stored machine learning model for marker detection.

[0080] (2) Next, we will explain the coordinate transformation performed by the inference unit 104. Fig. 7(a) schematically shows the image from the overhead camera 300, and Fig. 7(b) schematically shows the state of the shooting range 20 as viewed from directly above its center. The inference unit 104 transforms the position of the subject area in the coordinate system of the overhead camera into values ​​in a coordinate system (planar coordinate system) when the shooting range 20 is viewed from directly above its center.

[0081] The coordinate conversion into values ​​in the planar coordinate system is convenient for calculating the pan value (movement angle in a horizontal plane) for photographing a specific subject with the sub-camera 400. Note that this is based on the premise that the sub-camera 400 is installed so that the drive unit 409 performs panning within a horizontal plane parallel to the floor of the photographing range 20.

[0082] Coordinate conversion can be performed in various ways, but here, markers are placed at multiple known positions on the floor of the shooting range 20, and coordinate conversion is performed from the overhead camera coordinate system to a planar coordinate system based on the marker positions in the image captured by the overhead camera 300. Note that coordinate conversion may also be performed without using markers, for example, by using the viewpoint position and shooting direction of the overhead camera 300.

[0083] The coordinate transformation can be performed using a homography transformation matrix H according to Equation 1 below.

number

[0084] The homography transformation matrix can be calculated by substituting the coordinates of the four markers detected from the video and the (known) coordinates of the four markers placed in the shooting range 20 into Equation 1 and solving the simultaneous equations. If the positional relationship between the shooting range 20 and the overhead camera 300 is fixed, the homography transformation matrix H can be calculated in advance during test shooting and stored in, for example, ROM 103.

[0085] CPU 101 sequentially reads out the positions of the subject areas from RAM 102 and converts the coordinates into values ​​in a planar coordinate system. Fig. 8(b) schematically shows the state in which the foot coordinates (x, y) of each subject area detected in the image from overhead camera 300 shown in Fig. 8(a) are converted into coordinate values ​​(X, Y) in a planar coordinate system using Equation 1 and the homography transformation matrix H stored in ROM 103. CPU 101 stores the converted foot coordinates in RAM 102 as POSITION[n].

[0086] (3) Next, the operation of the inference unit 104 to identify the identification information ID[n] of the subject will be described. Here, the subject is identified using template matching. The subject is identified for the processing results of the subject detection from the second onwards. For the first processing result, it is sufficient to newly assign identification information ID[n] to the subject region.

[0087] The inference unit 104 identifies the identification information ID[n] of the detected object region by template matching using the templates stored in RAM 102. This identifies the object within the shooting range. For example, the inference unit 104 calculates an evaluation value representing the correlation of each template for each detected object region. Then, the inference unit 104 identifies the identification information ID[n] corresponding to the template with a correlation above a certain level and with the highest correlation as the identification information ID[n] of the object region. The evaluation value can be a known value, such as the sum of absolute differences in pixel values.

[0088] For a subject region that does not have a certain level of correlation with all templates, the inference unit 104 assigns new identification information ID[n] and adds the image of the subject region to the template.

[0089] In addition, the inference unit 104 may update an existing template using a subject area detected in the most recent frame image, or delete a template in which a subject area having a certain level of correlation or higher has not existed for a certain period of time. Furthermore, the inference unit 104 may store in the ROM 103 templates corresponding to frequently appearing identification information ID[n].

[0090] Note that a subject may be identified using a method other than template matching. For example, the subject may be identified as having the same identification information ID[n] as the subject area that is closest in at least one of the most recent detection position and size. Alternatively, the position in the current frame image may be predicted using a Kalman filter or the like based on the positional transitions in multiple past detection results associated with the same identification information, and the same identification information ID may be identified as the subject area that is closest to the predicted position. These methods may also be combined. By not using template matching, the accuracy of identifying different subjects that look similar can be improved.

[0091] (4) The inference unit 104 associates the identified identification information ID[n] with the position (plane coordinate system) POSITION[n] of the corresponding subject area and stores them in the RAM 102.

[0092] Of the processes (1) to (4), the processes other than the subject detection may be executed by the CPU 101 instead of the inference unit 104.

[0093] Here, the image from the overhead camera 300 was used to determine the identification information ID[n] and position POSITION[n] of the subject within the shooting range 20. However, the image from the sub-camera 400 may also be used. When there are multiple sub-cameras 400, the CPU 101 executes the operation shown in the flowchart of FIG. 6(a) for each sub-camera 400. The position of the subject area is output as a value in the coordinate system for each sub-camera 400. Thus, although the overhead camera 300 is not essential, it is believed that using the overhead camera 300 will result in better subject detection accuracy.

[0094] Returning to the explanation of Fig. 6(a), in S204, CPU 101 serving as subject of interest determination unit 122 in Fig. 3 determines a subject of interest as a tracking subject of main camera 500. CPU 101 can determine a subject of interest of main camera 500 from among the subjects detected in S203, based on the shooting direction of main camera 500 acquired in S202. CPU 101 stores identification information ID[n] corresponding to the subject area determined as the subject of interest of main camera 500 in RAM 102 as identification information MAIN_SUBJECT of the subject of interest.

[0095] For example, CPU 101 can determine that the subject closest to the shooting direction of main camera 500 in the planar coordinate system is the subject of interest for main camera 500. Note that if there are multiple subjects whose distance from the shooting direction of main camera 500 is equal to or less than a threshold, the user may be allowed to select the subject of interest from among them.

[0096] When allowing the user to select a subject of interest, CPU 101 displays on display unit 108 or an external display device the frame image to which subject detection processing was applied in S202, along with an indicator indicating the shooting direction and an indicator indicating a subject area that is a candidate for the subject of interest. The indicator for the subject area may be, for example, a rectangular frame indicating the outer edge of the subject area as shown in FIG. 8(a), but may also be another indicator. CPU 101 may also display on display unit 108 a message prompting the user to select a subject of interest within the image.

[0097] The user can select a subject area corresponding to a desired subject of interest by operating the user input unit 106 (input device). There are no particular restrictions on the selection method, but the user may operate a mouse or keyboard to specify the desired subject area.

[0098] When the CPU 101 detects a user operation to designate a subject area, the CPU 101 stores the identification information ID[n] corresponding to the designated subject area in the RAM 102 as the identification information MAIN_SUBJECT of the subject of interest.

[0099] Next, in S205, CPU 101 as tracking subject determination unit 123 in Fig. 3 acquires control content CAMERA_ROLE according to the role set for sub-camera 400. Specifically, CPU 101 reads out control content CAMERA_ROLE acquired in the role determination process described with reference to Fig. 5 and stored in RAM 102. Note that if there are multiple sub-cameras 400, CPU 101 executes the processes of S205 to S207 for each sub-camera.

[0100] In S206, CPU 101 as tracking subject determination unit 123 determines a subject to be tracked and photographed by sub camera 400 in accordance with the control content CAMERA_ROLE. CPU 101 determines a tracking subject for sub camera 400 in accordance with the tracking subject rules (FIG. 4) included in the control content CAMERA_ROLE.

[0101] When the tracking subject of the sub camera 400 is to be the same as the target subject of the main camera 500, the CPU 101 sets the identification information MAIN_SUBJECT of the target subject determined in S203 as the identification information SUBJECT_ID of the tracking subject of the sub camera 400.

[0102] When the tracking subject of sub camera 400 is a subject located on the left side of the subject of interest of main camera 500, CPU 101 detects a subject area located on the left edge of the subject area other than the subject of interest from the subject areas detected in S203. Then, CPU 101 sets identification information ID[n] corresponding to the detected subject area as identification information SUBJECT_ID of the tracking subject of sub camera 400.

[0103] CPU 101 writes the identification information SUBJECT_ID of the determined tracking subject to RAM 102. If the tracking subject may differ depending on the sub-camera, CPU 101 stores the identification information SUBJECT_ID of the tracking subject in association with the identification information of the sub-camera. Note that if the tracking subject changes, CPU 101 does not delete the information of the previous tracking subject but keeps it in RAM 102.

[0104] Here, the operation when the role set for the sub camera 400 is "main follow" will be described with reference to Fig. 9. For the sub camera 400 set as the role "main follow", the photography control device 100 controls it so that it tracks the subject of interest of the main camera 500.

[0105] Therefore, when the subject of interest of main camera 500 is determined to be subject B as shown in Fig. 9(a), CPU 101 determines subject B as the subject to be tracked by sub camera 400. Thereafter, when the subject of interest of main camera 500 is determined to have changed to subject A as shown in Fig. 9(b), CPU 101 changes the subject to be tracked by sub camera 400 to subject A. Similarly, when the subject of interest of main camera 500 is determined to have changed to subject C as shown in Fig. 9(c), CPU 101 changes the subject to be tracked by sub camera 400 to subject C.

[0106] The operation when the role set for the sub camera 400 is "assist follow" will be described with reference to Fig. 10. For the sub camera 400 set to the role "assist follow", the photography control device 100 controls the sub camera 400 so that it tracks a subject located on the left side of the main camera 500, which is different from the subject of interest of the main camera 500.

[0107] 10(a), when the target subject of the main camera 500 is determined to be subject B, CPU 101 determines subject A, which is on the left of subjects A and C, as the target subject of the sub camera 400. Thereafter, when the target subject of the main camera 500 is determined to have changed to subject A, as shown in FIG. 10(b), CPU 101 changes the target subject of the sub camera 400 to subject B, which is on the left of subjects B and C. Also, when the target subject of the main camera 500 is determined to have changed to subject C, as shown in FIG. 10(c), CPU 101 changes the target subject of the sub camera 400 to subject A, which is on the left of subjects A and B.

[0108] By dynamically changing the role set for the sub-camera 400 using the role control device 600, it is possible to change the subject being tracked by the sub-camera 400, thereby enabling flexible automatic photography.

[0109] 6(a), in S207, CPU 101 as pan / tilt value calculation unit 124 calculates the amount of change in pan angle and tilt angle required for sub camera 400 to track and photograph the tracking subject determined in S206. CPU 101 as zoom value calculation unit 125 also calculates the zoom value of sub camera 400 according to the change in angle of view of main camera 500. The following describes the case where there is one sub camera 400, but if there are multiple sub cameras 400, the amount of change in pan angle and tilt angle and the zoom value are calculated for each sub camera.

[0110] First, we will explain the operation of the CPU 101 as the pan / tilt value calculation unit 124. Here, it is assumed that the following information is stored in advance in the ROM 103 as default position information REF_POSI for each sub camera 400: - 3D coordinates of the installation location (values ​​in a planar coordinate system) Shooting direction corresponding to the initial values ​​of the pan and tilt angles of the drive unit Pan and tilt angle control range

[0111] The CPU 101 reads out from the RAM 102 the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the subject being tracked by the sub camera 400. Then, the CPU 101 first determines the pan angle from the position information POSITION_OH and the installation position of the sub camera 400.

[0112] 11 is a diagram showing an example of the positional relationship between the sub-camera 400 and the tracking subject in a plane coordinate system. Here, it is assumed that a pan angle θ is determined so that the optical axis direction of the sub-camera 400 faces the subject position. The CPU 101 calculates the pan angle θ using the following equation 2.

number

[0113] In Equation 2, px and py are the horizontal and vertical coordinates of the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracked subject. Also, subx and suby are the horizontal and vertical coordinates of the installation position of the sub-camera. Here, it is assumed that the current pan angle is the initial value of 0° and the optical axis direction is the vertical direction (Y-axis direction). If the current optical axis direction is not the vertical direction, the angle difference between the current optical axis direction and the vertical direction should be reflected in the angle obtained by Equation 2. Also, the direction of the pan is counterclockwise if subx > px, and clockwise if subx < px.

[0114] Next, the method for determining the tilt angle will be described using FIG. 12. FIG. 12 shows a state of the sub-camera and the tracked subject as viewed from the side. It is assumed that the current optical axis of the sub-camera 400 is in the horizontal direction and the height is h1, and the height of the face of the tracked subject facing the optical axis is h2. Let ρ be the angle difference (tilt angle) in the height direction between the current optical axis direction and the target optical axis direction. The CPU 101 calculates the tilt angle ρ using the following Equations 3 and 4.

Equation

[0117] CPU 101 periodically communicates with sub-camera 400 via communication network 700, acquires the current optical axis direction (pan angle and tilt angle of the drive unit), and stores it in RAM 102. The communication cycle can be set, for example, to be equal to or less than the reciprocal of the frame rate. Alternatively, CPU 101 may store in RAM 102 the sum of the pan angle and tilt angle controlled for sub-camera 400 from the initial state, and use this as the current optical axis direction.

[0118] In this way, the CPU 101 calculates the amount of change in the pan angle and tilt angle of the sub camera 400, and stores the amount of change in the RAM 102. If there are multiple sub cameras 400, the CPU 101 calculates the amount of change in the pan angle and tilt angle for each sub camera.

[0119] The change amounts of the pan angle and tilt angle may be the angular velocity at which the sub camera 400 rotates in the direction of the tracking subject. For example, the CPU 101 acquires the current pan angle and tilt angle from the sub camera 400 via the communication network 700. The CPU 101 then calculates a pan angular velocity proportional to the difference between the pan angle θ read from the RAM 102 and the current pan angle. The CPU 101 also calculates a tilt angular velocity proportional to the difference between the tilt angle ρ read from the RAM 102 and the current tilt angle. The CPU 101 stores the angular velocities calculated in this manner in the RAM 102.

[0120] Note that the amount of change in the pan angle and tilt angle may be calculated using the image from sub camera 400 instead of the image from overhead camera 300. In this case, CPU 101 may calculate the amount of change in the pan angle from the horizontal difference between the current optical axis direction and the direction of the subject to be tracked in the coordinate system of sub camera 400, and may calculate the amount of change in the tilt angle from the vertical difference. Also, the imaging system may change the shooting direction to track and shoot the subject to be tracked in only one of the pan direction and tilt direction, and in such an imaging system, the amount of change in only one of the pan angle and tilt angle may be calculated.

[0121] Next, we will explain the operation of CPU 101 as zoom value calculation unit 125. CPU 101 as zoom value calculation unit 125 periodically acquires information MAIN_ZOOM that indicates the angle of view of main camera 500 and stores it in RAM 102. When information MAIN_ZOOM changes, CPU 101 calculates zoom value Z_VALUE for sub camera 400 in accordance with control content CAMERA_ROLE that corresponds to the role set for sub camera 400.

[0122] CPU 101 can determine the zoom operation and its phase of main camera 500, for example, by detecting a change in the angle of view of the image of main camera 500. For example, the change in the angle of view may be detected from a change over time in the size or spacing of the subject area.

[0123] 13 shows an example of mapping of zoom values ​​between the main camera and the sub camera. Here, it is assumed that the main camera 500 and the sub camera 400 optically change the angle of view (the photographing optical system has a zoom function). However, a similar function may be realized by digital zoom using the image processing units 406 and 506.

[0124] The zoom value is a parameter having a value corresponding to the angle of view. In this embodiment, the smaller (narrower) the angle of view, the smaller the zoom value, and the zoom value on the telephoto side is smaller than the zoom value on the wide-angle side. The sub-camera 400 and the main camera 500 can control the imaging optical system to the angle of view corresponding to the zoom value by transmitting a command specifying the zoom value. In other words, the zoom value is information related to the angle of view and represents the zoom state. The zoom value may be, for example, the focal length (mm) of the imaging optical system corresponding to a 35mm full-size image sensor, in which case the zoom value on the telephoto side is larger than the zoom value on the wide-angle side.

[0125] 13, the range of the zoom value MAIN_ZOOM of the main camera 500 is main_min to main_max. The zoom range of the sub camera 400 is sub_min to sub_max. main_min and sub_min are the zoom values ​​corresponding to the telephoto ends of the main camera 500 and the sub camera 400, respectively, and main_max and sub_max are the zoom values ​​corresponding to the wide-angle ends of the main camera 500 and the sub camera 400, respectively. FIG. 13 shows an example in which the range of the zoom value of the main camera 500 is wider than the range of the zoom value of the sub camera 400 at both the telephoto end and the wide-angle end.

[0126] When controlling the zoom value SUB_ZOOM of the sub camera 400 to be in phase with the zoom value MAIN_ZOOM of the main camera 500, the CPU 101 calculates SUB_ZOOM corresponding to the current MAIN_ZOOM using the following equation 5.

number

[0127] On the other hand, when controlling the zoom value SUB_ZOOM of the sub camera 400 to be in the opposite phase to the zoom value MAIN_ZOOM of the main camera 500, the SUB_ZOOM corresponding to the current MAIN_ZOOM is calculated using the following equation 6. Specifically, the CPU 101 calculates the SUB_ZOOM corresponding to the current MAIN_ZOOM by substituting the SUB_ZOOM calculated using equation 5 into the right side of the following equation 6. SUB_ZOOM=sub_max-(SUB_ZOOM-sub_min) (Formula 6)

[0128] When the main camera 500 performs digital zoom and controls the angle of view by cropping, the CPU 101 can determine the zoom value SUB_ZOOM of the sub camera 400 according to the size of the range to be cropped by the main camera 500. Specifically, the CPU 101 sets the zoom value SUB_ZOOM to a smaller value (higher magnification) as the size of the range to be cropped by the main camera 500 becomes smaller, and sets the zoom value SUB_ZOOM to a larger value (lower magnification) as the size becomes larger.

[0129] Furthermore, the zoom control content associated with the role of the sub-camera 400 is not limited to control in phase or opposite phase to that of the main camera 500. For example, a zoom operation independent of changes in the angle of view of the main camera 500 may be associated with the role. For example, an auto-zoom operation that maintains a constant size of a tracked subject may be associated with the role. The angle of view of the sub-camera 400 may also be fixed to a specific angle of view. By adding roles associated with these zoom controls to the control content for each role shown in FIG. 4 or by changing the zoom control content of the role shown in FIG. 4, various zoom controls for the sub-camera 400 become possible.

[0130] Returning to FIG. 6(a), in S207, CPU 101 reads from RAM 102 the change amounts of the pan and tilt angles and the zoom value calculated in S206. Then, CPU 101 generates a control command PT_VALUE that instructs sub camera 400 to change the pan angle and tilt angle corresponding to these change amounts. CPU 101 also generates a control command Z_VALUE that instructs sub camera 400 to change the angle of view corresponding to the zoom value. The format of the control command is assumed to be predetermined. CPU 101 stores the generated control commands PT_VALUE and Z_VALUE in RAM 102. Note that S207 may be skipped if there is no need to generate a control command, such as when the tracking subject is stationary or when the angle of view of main camera 500 does not change.

[0131] Then, the CPU 101 reads the control commands PT_VALUE and Z_VALUE from the RAM 102 and transmits them to the communication network 700 via the network I / F 105. The sub-camera 400 receives the control commands PT_VALUE and Z_VALUE via the network I / F 405.

[0132] The CPU 101 executes the process from S201 on the next frame image of the video from the overhead camera 300. Note that the process shown in Fig. 6(a) does not necessarily have to be executed for every frame.

[0133] (Operation of overhead camera 300) Next, the operation of the overhead camera 300 will be described with reference to Fig. 6(b). The operation described below is realized by the CPU 301 executing a program.

[0134] When the overhead camera 300 is powered on, each functional block is initialized by the CPU 301, and the camera enters a shooting standby state. In the shooting standby state, the CPU 301 may start a video shooting process for live view display, and output display image data generated by the image processing unit 306 to the shooting control device 100 via the network I / F 305.

[0135] In the shooting standby state, the CPU 301 waits for reception of a control command via the network I / F 305. When the CPU 301 receives a control command, it executes an operation according to the control command. Here, an operation when a shooting command is received as a control command from the shooting control device 100 will be described.

[0136] In step S301, the CPU 301 receives an imaging command from the imaging control device 100 via the network I / F 305.

[0137] The shooting command may specify shooting parameters such as frame rate, resolution, etc. The shooting command may also include settings related to the processing to be applied by the image processing unit 306.

[0138] In S302, in response to receiving the shooting command, the CPU 301 starts video shooting processing to supply the video to the shooting control device 100. In this video shooting processing, a video with higher image quality is shot than in video shooting processing for live view display. For example, at least one of the video resolution and shooting frame rate is higher than in video for live view display. The image processing unit 306 applies processing to the image based on the video settings to be supplied to the shooting control device 100. The image processing unit 306 sequentially stores the generated video data in the RAM 302.

[0139] In S303, the CPU 101 reads the video data from the RAM 302 and transmits it to the shooting control device 100 via the network I / F 305. After that, the process from shooting to supplying the video data continues until a control command to stop shooting is received.

[0140] (Main camera 500 operation) Next, the operation of the main 500 will be described with reference to Fig. 6(c). The operation described below is realized by the CPU 501 executing a program.

[0141] When the main camera 500 is powered on, the CPU 501 initializes each functional block and then starts video capture processing to supply the video to the imaging control device 100. The image processing unit 506 processes the analog image signal obtained from the image sensor 507 based on the video settings to be supplied to the imaging control device 100. The image processing unit 506 sequentially stores the generated video data in the RAM 502. The CPU 501 reads the video data from the RAM 502 and supplies it to the imaging control device 100 via the network I / F 505.

[0142] The CPU 501 supplies video data to the shooting control device 100 while waiting to receive a control command via the network I / F 305. When the CPU 501 receives a control command, it executes an operation according to the control command. Here, the operation when a shooting direction acquisition command is received will be described. When a pan / tilt control command PT_VALUE or a zoom control command Z_VALUE is received, the CPU 501 drives the drive unit 509 according to the command.

[0143] In S501, the CPU 501 receives an imaging direction acquisition command via the network I / F 505. The CPU 501 stores the received imaging direction acquisition command in the RAM 502.

[0144] In step S502 , in response to receiving the imaging direction acquisition command, the CPU 501 acquires the current pan angle and tilt angle from the driving unit 509 via the driving I / F 508 and stores them in the RAM 502 .

[0145] In S503, the CPU 501 reads the current pan angle and tilt angle from the RAM 502 and transmits them to the imaging control device 100 via the network I / F 305 as imaging direction information ANGLE.

[0146] (Operation of sub camera 400) Next, the operation of the sub-camera 400 will be described with reference to Fig. 6(d). The operation described below is realized by the CPU 401 executing a program.

[0147] When the sub-camera 400 is powered on, the CPU 401 initializes each functional block and then starts video capture processing to supply the video to the imaging control device 100. The image processing unit 406 processes the analog image signal obtained from the image sensor 407 based on the video settings to be supplied to the imaging control device 100. The image processing unit 406 sequentially stores the generated video data in the RAM 402. The CPU 401 reads the video data from the RAM 402 and supplies it to the imaging control device 100 via the network I / F 405.

[0148] The CPU 401 supplies video data to the shooting control device 100 while waiting for a control command to be received through the network I / F 305. Upon receiving a control command, the CPU 401 executes an operation according to the control command. Here, the operation when a pan / tilt control command PT_VALUE and a zoom control command Z_VALUE are received from the shooting control device 100 will be described.

[0149] In S401, the CPU 401 receives at least one of a pan / tilt control command PT_VALUE and a zoom control command Z_VALUE from the imaging control device 100 via the network I / F 405. The CPU 401 stores the received control command in the RAM 402.

[0150] In S402, the CPU 401 reads out the operation direction and the corresponding operation amount from the control command stored in the RAM 402, and stores them in the RAM 402. Here, in the case of a pan / tilt control command PT_VALUE, the operation direction is the pan and / or tilt direction, and the operation amount is the target angle. Also, in the case of a zoom control command Z_VALUE, the operation amount is the zoom value, and since the operation direction can be identified from the zoom value, there is no need to read out and store the operation direction.

[0151] In S403, the CPU 401 generates drive parameters for the drive unit 409 based on the operation direction and operation amount read out in S403. The CPU 401 may obtain drive parameters according to the combination of the operation direction and operation amount, for example, by using a table stored in advance in the ROM 403. Note that when the operation amount is given as a target value (target angle or zoom value), the CPU 410 obtains the drive parameters from the difference from the current value.

[0152] In S404, the CPU 401 controls the driver 409 through the driver I / F 408 based on the drive parameters acquired in S404. As a result, the driver 409 changes the shooting direction of the sub-camera 400 to the operation direction and angle specified by the pan / tilt control command PT_VALUE. The driver 409 also changes the angle of view of the shooting optical system to the zoom value specified by the zoom control command Z_VALUE.

[0153] Next, the operation of the imaging control device 100 to control the imaging direction (pan and tilt) and angle of view (zoom value) of the sub-camera in accordance with the role set for the sub-camera 400 will be described in more detail using the flowchart shown in Fig. 14. The operation shown in the flowchart in Fig. 14 is executed as part of the operation of S205 to S207 in Fig. 6(a).

[0154] S601 corresponds to S205, and the CPU 101 reads out the control content CAMERA_ROLE stored in the RAM 102 in S103 of FIG.

[0155] S602 to S607 are performed in S206, for example. In S602, CPU 101 determines whether the definition of the tracking subject of sub camera 400 included in the control content CAMERA_ROLE indicates the tracking subject (target subject) of main camera 500. For example, if the definition of the tracking subject of sub camera 400 has a value indicating "same as main", CPU 101 determines that the definition of the tracking subject of sub camera 400 indicates the tracking subject of main camera 500, and executes S603. On the other hand, if the definition of the tracking subject of sub camera 400 has a value indicating "different from main (left side)", CPU 101 determines that the definition of the tracking subject of sub camera 400 does not indicate the tracking subject of main camera 500, and executes S604.

[0156] In S603, CPU 101 determines to control the shooting direction of sub camera 400 so that main camera 500 tracks the tracking subject (target subject). In S604, CPU 101 determines to control the shooting direction of sub-camera 400 so as to track a subject other than the target subject of main camera 500, which is located on the left side.

[0157] In S605, CPU 101 determines whether the zoom control specification of sub camera 400 included in the control content CAMERA_ROLE indicates control in the same phase as main camera 500. For example, if the zoom control specification of sub camera 400 has a value indicating "same phase as main camera," CPU 101 determines that the zoom control specification of sub camera 400 indicates control in the same phase as main camera 500, and executes S606. On the other hand, if the zoom control specification of sub camera 400 has a value indicating "opposite phase to main camera," CPU 101 determines that the zoom control specification of sub camera 400 does not indicate control in the same phase as main camera 500, and executes S607.

[0158] In S606, the CPU 101 determines to control the zoom value (angle of view) of the sub camera 400 in the same phase as the change in the zoom value of the main camera 500. In S607, the CPU 101 determines to control the zoom value (angle of view) of the sub camera 400 in the opposite phase to the change in the zoom value of the main camera 500.

[0159] FIG. 15 illustrates an example of sub-camera control when the role assigned to the sub-camera 400 is "main follow." FIG. 15 shows a schematic diagram of how the shooting control device 100 controls the shooting direction and angle of view of the sub-camera 400 when the subject of interest and angle of view of the main camera 500 change over time during shooting. Time passes from left to right in the diagram. Note that FIG. 15 shows three zoom states: "telephoto end," "intermediate," and "wide-angle end." This is because, as shown in FIG. 13, the zoom value ranges of the sub-camera and the main camera may differ. "telephoto end" corresponds to zooming in to the telephoto end, "wide-angle end" corresponds to zooming down to the wide-angle end, and "intermediate" corresponds to a zoom state intermediate between the telephoto end and the wide-angle end. However, the actual zoom values ​​of the sub-camera and the main camera may differ. For example, when the zoom state is "telephoto end," the main camera's zoom value is "main_min," and the sub camera's zoom value is "sub_min."

[0160] Initially, the target subject (tracking subject) of main camera 500 is subject B, and the zoom state is "medium." Therefore, CPU 101 determines that the tracking subject of sub camera 400 is subject B, and controls the shooting direction so that sub camera 400 tracks subject B. CPU 101 also controls the zoom state of sub camera 400 to "medium."

[0161] Thereafter, the subject of interest of main camera 500 is changed from subject B to subject A, and the zoom state is changed to the "telephoto end." In response to this, CPU 101 changes the subject being tracked by sub camera 400 from subject B to subject A, and controls the shooting direction so that sub camera 400 tracks subject A. CPU 101 also controls the zoom state of sub camera 400 to the "telephoto end."

[0162] Thereafter, the subject of interest of main camera 500 is changed from subject A to subject C, and the zoom state is changed to "wide-angle end." In response to this, CPU 101 changes the subject being tracked by sub-camera 400 from subject A to subject C, and controls the shooting direction so that sub-camera 400 tracks subject C. CPU 101 also controls the zoom state of sub-camera 400 to "wide-angle end."

[0163] In this way, when the role of sub camera 400 is "main follow," CPU 101 automatically changes the tracking subject and zoom value of sub camera 400 so as to follow changes in the subject of interest and angle of view (zoom value) of main camera 500. Note that in the example shown in FIG. 15, control is performed so that the degree of change in the zoom state of sub camera 400 is equivalent to the degree of change in the zoom state of main camera 500, but the actual zoom values ​​may differ as long as the direction of change in the zoom value is in phase. For example, when the zoom state of main camera 500 is at the telephoto end, the zoom state of sub camera 400 does not have to be at the telephoto end. Whether the zoom value of sub camera 400 is matched to the zoom value of main camera 500 may be settable using role setting information.

[0164] Next, an example of control of the sub camera when the role set for the sub camera 400 is "assist counter" will be described with reference to FIG. 16, which is similar to FIG.

[0165] Initially, the subject of interest (subject to be tracked) of main camera 500 is subject B, and the zoom state is "telephoto end." Therefore, CPU 101 determines that the subject to be tracked by sub camera 400 is subject A on the left, of subjects A and C other than subject B, and controls the shooting direction so that sub camera 400 tracks subject A. CPU 101 also controls the zoom state of sub camera 400 to the "wide-angle end," which is in the opposite phase to that of main camera 500.

[0166] Thereafter, the target subject of main camera 500 is changed from subject B to subject A, and the zoom state is changed to "middle." In response to this, CPU 101 changes the tracking subject of sub camera 400 to subject B on the left of subjects B and C other than subject A, and controls the shooting direction so that sub camera 400 tracks subject B. Furthermore, because the zoom state of main camera 500 has changed from "telephoto end" to "middle," CPU 101 controls the zoom state of sub camera 400 from "wide-angle end" to "middle" (in the opposite phase).

[0167] Thereafter, the target subject of main camera 500 is changed from subject A to subject C, and the zoom state is changed to "wide-angle end." In response to this, CPU 101 changes the tracking subject of sub camera 400 to subject A on the left of subjects A and B other than subject C, and controls the shooting direction so that sub camera 400 tracks subject A. Furthermore, because the zoom state of main camera 500 has changed from "middle" to "wide-angle end," CPU 101 controls the zoom state of sub camera 400 from "middle" to "telephoto end" (in the opposite phase).

[0168] In this way, when the role of sub camera 400 is "assist counter," CPU 101 automatically changes the tracking subject of sub camera 400 to a subject other than the target subject of main camera 500 in accordance with a change in the target subject of main camera 500. CPU 101 also automatically changes the zoom value of sub camera 400 in the opposite direction to the change in the angle of view (zoom value) of main camera 500.

[0169] 16, the zoom state of the sub camera 400 is controlled at a rate equivalent to that of the main camera 500, but the amount of change in the zoom value may be different as long as the change direction of the zoom value is in the opposite phase. For example, when the zoom state of the main camera 500 is at the telephoto end, the zoom state of the sub camera 400 does not have to be at the wide-angle end. The rate of change in the zoom state of the sub camera 400 relative to the rate of change in the zoom state of the main camera 500 may be set using role setting information.

[0170] (Variation) So far, we have described an example in which the tracking subject and zoom value of the sub-camera 400 are automatically controlled based on the target subject and zoom value of the main camera 500. In the above example, the sub-camera is automatically controlled to track a single subject, but it can also be automatically controlled to track multiple subjects within the shooting range.

[0171] When the role set for the sub-camera 400 is "assist follow," examples of control when the sub-camera 400 is made to track a single subject and when it is made to track multiple subjects will be described using FIG. 17. For ease of understanding and explanation, a case will be described here in which the angle of view of the main camera 500 does not change and only control of the subject being tracked is performed. Also, it is assumed that the angle of view of the sub-camera 400 is capable of always capturing images of all subjects within the capturing range 20 regardless of the capturing direction. Note that the capturing direction of the sub-camera 400 shown in the top row of FIG. 17 indicates the capturing direction when a single subject is being tracked.

[0172] As in FIG. 16, initially, the subject of interest (tracking subject) of main camera 500 is subject B. Therefore, CPU 101 determines that the tracking subject of sub camera 400 is subject A on the left side of subjects A and C other than subject B, and controls the shooting direction so that sub camera 400 tracks subject A. If the shooting direction is controlled so that the tracking subject is located at the center of the screen, as shown in the second row from the bottom, the image captured by sub camera 400 will be unbalanced, with subjects A to C shifted to the right. Therefore, if the image captured by sub camera 400 contains multiple subjects, including the tracking subject, the shooting direction can be controlled to track these multiple subjects. For example, CPU 101 can control the shooting direction so that the shooting direction tracks the center of gravity of the positions of multiple subjects A to C included in the image captured by sub camera 400. As a result, the sub camera 400 captures an image as shown in the bottom row.

[0173] In the example shown in Figure 17, regardless of which of subjects A to C the sub-camera 400 tracks, all of the subjects A to C are photographed, so even if the subject of interest of the main camera 500 changes, the photographing direction of the sub-camera 400 remains approximately constant.

[0174] (Variation 2) Furthermore, after determining the subject to be tracked by the sub camera 400, the CPU 101 may control the sub camera 400 to focus on the subject to be tracked. Basically, the CPU 401 continuously controls the focusing distance so that the sub camera 400 focuses on the specified subject to be tracked, but the shooting control device 100 can set the AF frame of the sub camera 400 to the position of the subject to be tracked. This allows the sub camera 400 to quickly and reliably focus on the subject to be tracked. Note that if the AF frame is set when the pan speed slows down (below a threshold), the tracking target is likely to be located in the center of the screen, which may shorten the time required for focusing.

[0175] (Variation 3) Furthermore, it is possible to control the sub-camera 400 without using the overhead camera 300. In this case, the shooting direction of the sub-camera 400 can be determined from the installation positions of the main camera 500 and the sub-camera 400 and the shooting direction of the main camera 500 (the orientation of the main camera 500 with respect to the subject to be tracked). The main camera 500 executes a subject detection process, and the CPU 101 acquires and uses the result to control the sub-camera 400. For example, the CPU 101 acquires an image of the subject area from the main camera 500 as a result of the subject detection process. The CPU 101 can then control the sub-camera 400 to execute a subject tracking process using the acquired image as a template. Alternatively, the CPU 101 may use the acquired image as a template and control the sub-camera 400 to track a subject area with a low correlation with the template.

[0176] (Variation 4) Although the shooting control device 100, the role control device 600, and the main camera 500 have been described as independent devices, the functions of the shooting control device 100 and the role control device 600 can also be incorporated into the main camera 500. In this case, the image from the overhead camera 300 is supplied to the main camera 500. With this configuration, it is possible to reduce the amount of equipment required to realize a multi-camera shooting system.

[0177] As described above, according to this embodiment, when the operation of the sub-camera is automatically controlled based on the status and image of the main camera, automatic control is performed according to the role assigned to the sub-camera. Therefore, the photography control device of this embodiment can achieve more flexible automatic photography control while realizing labor savings.

[0178] <Second embodiment> Next, a second embodiment of the present invention will be described. In this embodiment, the tracking subject of the sub camera is determined taking into consideration the image of the sub camera in addition to information based on the state or image of the main camera and the role set for the sub camera.

[0179] Fig. 18 is a schematic diagram showing an example of the configuration of an imaging system 10' according to this embodiment. In Fig. 18, components similar to those in the imaging system 10 of the first embodiment are given the same reference numerals as in Fig. 1, and descriptions thereof will be omitted. The imaging system 10' according to this embodiment has two sub-cameras, A800 and B900. The functional configurations of the sub-camera A800 and sub-camera B900 are similar to those of the sub-camera 400 described in the first embodiment, and descriptions thereof will be omitted.

[0180] In this embodiment, a role is set in the imaging control device 100 from the role control device 600, and the role setting information includes different control contents for each sub-camera. The CPU 101 controls the operation of each sub-camera in accordance with the role set in the imaging control device 100. 19 is a diagram showing an example of role setting information in this embodiment. Here, only the role "assist follow" and the corresponding control content for each sub-camera are shown as an example. However, the types of roles that can be set in the shooting control device 100 and the control content corresponding to the role types are not limited to the example shown in FIG. 19.

[0181] Here, it is specified that the sub-camera A800 is controlled to set the left subject of the subjects different from the subject being tracked by the main camera as the tracking subject and to focus on the tracking subject. Furthermore, it is specified that the sub-camera B900 is controlled to set the right subject of the subjects different from the subject being tracked by the main camera as the tracking subject and to focus on the tracking subject. Here, since it is known in advance that the total number of subjects is three, it is specified as the left or right of the two subjects, but it may also be simply specified as the left or right. Note that zoom control may be specified as in the first embodiment, but for simplicity of explanation, zoom control will be omitted.

[0182] 19 is stored in advance in the ROM 103 of the imaging control device 100. Alternatively, the role setting information may be supplied from the role control device 600 to the imaging control device 100, and the CPU 101 may store it in the RAM 102.

[0183] Using FIG. 20, how the shooting control device 100 controls the sub-camera A800 and the sub-camera B900 based on the role setting information will be described.

[0184] Fig. 20(a) shows the positional relationship between shooting range 20, main camera 500, sub-camera A 800, and sub-camera B 900 at the start of shooting, and subjects A to C within shooting range 20. Fig. 20(b) shows a state in which, after shooting starts, subject A moves from the left to the right of subject B, and subject C moves from the right to the left of subject B.

[0185] At the stage of Fig. 20(a), CPU 101 of imaging control device 100 controls the operations of sub-camera A800 and sub-camera B900 based on the role setting information shown in Fig. 19. That is, CPU 101 determines that of subjects A and C other than the target subject (subject B) of main camera 500, subject A on the left side is the subject to be tracked by sub-camera A800, and subject C on the right side is the subject to be tracked by sub-camera B900.

[0186] The CPU 101 controls the shooting directions of the sub-camera A800 and the sub-camera B900 so that they track the determined subject to be tracked. The CPU 101 also controls the sub-camera A800 and the sub-camera B900 so that they focus on the determined subject to be tracked.

[0187] In the state of FIG. 20(b), subject A being tracked by sub camera A800 does not satisfy the condition of "subject on the left side" (other than the subject of interest of main camera 500). Therefore, CPU 101 changes the subject being tracked by sub camera A800 to subject C on the left side of subjects A and C other than the subject of interest of main camera 500 (subject B). Similarly, subject C being tracked by sub camera B900 also does not satisfy the condition of "subject on the right side". Therefore, CPU 101 changes the subject being tracked by sub camera B900 to subject A on the right side of subjects A and C other than the subject of interest of main camera 500 (subject B). CPU 101 also controls sub camera A800 and sub camera B900 to focus on the determined subject being tracked.

[0188] When the subject moves significantly, as shown in Figure 20(b), there is a high possibility that another subject is present in front, making it easier for the subject to be tracked to be hidden. By setting the sub-camera positioned to the left of the shooting range 20 to track a subject on the left side, it is possible to specify that the tracking subject will be changed effectively when the tracking subject moves significantly to the right. The same applies to the sub-camera positioned to the right of the shooting range 20.

[0189] In this embodiment, it is possible to specify the control content so that each sub-camera with the same role tracks a different subject. Therefore, the shooting control device 100 of this embodiment makes it possible to automatically control the sub-cameras to shoot footage tracking various subjects based on the status of the main camera and information obtained from the footage.

[0190] (Variation) In this embodiment, the shooting direction of the sub-cameras for tracking a specific subject is estimated by performing coordinate transformation on the position of the subject area detected from the video of the overhead camera 300. However, a similar estimation may be performed based on the video of the sub-cameras A800 and B900. In this case, although the processing load on the shooting control device 100 increases, the accuracy of controlling the shooting direction of the sub-cameras can be improved because coordinate transformation is not required.

[0191] (Variation 2) In this embodiment, the pan and tilt values ​​for controlling the sub-cameras A800 and B900 are calculated to automatically track the tracking target. However, automatic tracking is not essential. For example, the role setting information may specify that the pan values ​​of the sub-cameras A800 and B900 are controlled according to the zoom value of the main camera 500.

[0192] As an example, when the zoom state of the main camera 500 is at the "wide-angle end," the sub-camera A800 can be controlled to face in a direction diagonally 45 degrees to the left, and the sub-camera B900 can be controlled to face in a direction diagonally 45 degrees to the right. On the other hand, when the zoom state of the main camera 500 is at the "telephoto end," the sub-camera A800 and the sub-camera B900 can both be controlled to face in the center (0 degrees) direction.

[0193] By performing such control, it becomes possible to synchronize and control the shooting directions of multiple sub-cameras with the zoom state of main camera 500, thereby enhancing the dramatic effect. For example, when main camera 500 zooms in on a specific subject, sub-cameras A800 and B900 can change their shooting directions while zooming in on the same subject in tandem. By performing such control, it becomes possible to simultaneously view multiple images captured by multiple cameras controlled synchronously in a performance in which multiple camera images are displayed on multiple monitors. Furthermore, by automatically controlling each camera with the same role, the variation in the angle of view between cameras over time is reduced compared to when each camera is manually operated, and the sense of unity in the angle of view change is strengthened, resulting in a dramatic effect that enhances the sense of realism.

[0194] (Variation 3) Furthermore, in the present embodiment, an example has been shown in which the main camera 500 is controlled as the master, and the sub-cameras A800 and B900 are controlled as slaves. However, the master-slave relationship between the main camera 500 and the sub-camera A800 may be dynamically changeable. For example, among the main camera 500, the sub-camera A800, and the sub-camera B900, the camera capturing the main line video may be controlled as the master, and the other cameras may be controlled as slaves. In this case, the shooting control device 100 may obtain information indicating which camera's video has been selected as the main line video from an external device such as a video selection switcher, or may determine the information based on a tally signal. Similar control may be performed not only when the main line video is selected, but also when video for recording, or other video related to viewer viewing or recording, is selected.

[0195] The master-slave relationship between the cameras may also be switched when sub-camera A 800 or sub-camera B 900 is manually operated. In this case, when the user manually operates one of the sub-cameras, control of the other cameras begins with that sub-camera as the master, which is convenient.

[0196] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0197] The disclosure of the present embodiment includes the following imaging control device, imaging control method, imaging system, and program. (Item 1) an acquisition means for acquiring information from the state or image of a main camera among the plurality of cameras; a control means for controlling the operation of a sub-camera among the plurality of cameras based on a role set for the sub-camera and the information; the acquisition means acquires, as the information, information on a target subject of the main camera and information on an angle of view of the main camera; the control means controls the shooting direction and the angle of view of the sub-camera based on the role set for the sub-camera and the information. An imaging control device characterized by: (Item 2) The control means determines the target subject or a subject other than the target subject as the tracking subject of the sub-camera according to the role set for the sub-camera, and controls the shooting direction of the sub-camera so as to track the tracking subject. (Item 3) 3. The photographing control device according to item 2, wherein the control means controls the photographing direction of the sub-camera so as to track a plurality of subjects including the subject to be tracked. (Item 4) Item 3. The photographing control device according to item 3, characterized in that when the control means controls the photographing direction of the sub-camera so as to track a plurality of subjects including the tracking subject, the control means controls the photographing direction of the sub-camera so as to track the center of gravity of the positions of the plurality of subjects. (Item 5) 5. The photographing control device according to any one of items 1 to 4, wherein when there are a plurality of sub-cameras, the control means determines different tracking subjects for the plurality of sub-cameras. (Item 6) 6. The imaging control device according to any one of items 1 to 5, wherein the main camera and the sub camera can be dynamically changed. (Item 7) 7. The photographing control device according to item 6, wherein the main camera and the sub camera are switched when the sub camera is manually operated. (Item 8) 7. The imaging control device according to item 6, wherein the camera of the plurality of cameras whose video is selected by an external device is set as the main camera. (Item 9) The shooting control device described in any one of items 1 to 8, characterized in that the control means determines the shooting direction of the sub-camera based on images captured by a camera other than the multiple cameras, which captures the entire shooting range of the multiple cameras. (Item 10) 10. The imaging control device according to any one of items 1 to 9, wherein the sub-camera can be externally controlled in pan and tilt operations. (Item 11) the control means controls the angle of view of the sub-camera in the same phase as or in the opposite phase to the change in angle of view of the main camera, depending on the role set for the sub-camera. 11. The imaging control device according to any one of items 1 to 10, characterized in that: (Item 12) The information regarding the angle of view of the main camera is information indicating a zoom state of the main camera, The photographing control device according to any one of claims 1 to 11, characterized in that the control means controls the zoom operation of the sub-camera based on the role set for the sub-camera and information representing the zoom state of the main camera. (Item 13) 13. The photographing control device according to any one of items 1 to 12, wherein the sub-camera has an externally controllable zoom operation. (Item 14) An imaging control device according to any one of items 1 to 13; the plurality of cameras; a communication network that communicably connects the photography control device and the plurality of cameras; An imaging system comprising: (Item 15) A photography control method executed by a photography control device, Obtaining information from the status or video of a main camera among the multiple cameras; controlling an operation of a sub-camera among the plurality of cameras based on a role set for the sub-camera and the information; the acquiring includes acquiring, as the information, information on a target subject of the main camera and information on an angle of view of the main camera; The controlling comprises: and controlling the shooting direction and angle of view of the sub-camera based on the role set for the sub-camera and the information. (Item 16) A program for causing a computer to function as each of the means possessed by the imaging control device according to any one of items 1 to 13.

[0198] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0199] 100...control device, 300...overhead camera, 400...sub-camera, 500...main camera, 600...role control device, 101...CPU, 102...RAM, 103...ROM, 104...inference unit, 105...network I / F, 106...user input unit

Claims

[Claim 1] An imaging control device, an acquisition means for acquiring information about a first imaging device included in the plurality of imaging devices; a control unit that controls an operation of a second imaging device included in the plurality of imaging devices based on a set role of the second imaging device and information about the first imaging device, the acquiring means acquires, as the information about the first imaging device, information about a target subject of the first imaging device and information about an angle of view of the first imaging device; The control means controls the subject to be tracked and the angle of view of the second imaging device so that at least one of the subject to be tracked and the angle of view of the second imaging device differs when the set role of the second imaging device is a first role and when it is a second role different from the first role.

Images