Display control device and display control method, and imaging system

Abstract

To provide an imaging control device and an imaging control method capable of presenting information for improving user friendliness of an automatic imaging system using a plurality of cameras.SOLUTION: An imaging control device controls an imaging direction of an angle of view of a sub camera in a plurality of cameras including a main camera and the sub camera based on a role, which is set to the sub camera, and a subject of interest and an angle of view of the main camera. The imaging control device generates and outputs an image for display presenting at least one of information relating to the subject of interest of the main camera and a tracking subject of the sub camera and information relating to the angles of view of the main camera and the sub camera.SELECTED DRAWING: Figure 21

Description

[Technical field]

[0001] The present invention relates to an imaging control device and an imaging control method, a display control device and a display control method, and an imaging system, and in particular to a technique for controlling a plurality of imaging devices. [Background technology]

[0002] Patent Document 1 describes an imaging system that divides multiple cameras into a main camera and a sub-camera, and controls the sub-camera to capture the same subject as the main camera. Patent Document 1 also describes a system that allows the user to check the shooting direction and angle of view of each camera on a remote control screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-25248 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, the shooting of the sub-camera can be automatically controlled, which can reduce labor. However, in order to check the subject being shot by each camera, it is necessary to operate the remote control screen to display the image being shot by each camera.

[0005] Therefore, in one aspect, the present invention provides an imaging control device, an imaging control method, a display control device, and a display control method that are capable of presenting information that improves the usability of an automatic imaging system that uses multiple cameras. [Means for solving the problem]

[0006] In one aspect, the present invention provides a shooting control device comprising: a control means for controlling the shooting direction and angle of view of the sub-camera among a plurality of cameras including a main camera and a sub-camera based on the role set for the sub-camera and the subject of interest and angle of view of the main camera; a generation means for generating a display image presenting information about the main camera and the sub-camera; and an output means for outputting the display image, wherein the generation means generates a display image presenting at least one of information about the subject of interest of the main camera and the subject being tracked by the sub-camera, and information about the angles of view of the main camera and the sub-camera. Effect of the Invention

[0007] According to the present invention, it is possible to provide an imaging control device and imaging control method, and a display control device and display control method capable of presenting information that improves the usability of an automatic imaging system that uses multiple cameras. [Brief description of the drawings]

[0008] [Figure 1] Schematic diagram of an imaging system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing an example of the functional arrangement of each device in an imaging system according to a first embodiment; [Diagram 3] FIG. 1 is a diagram illustrating the imaging control device according to the first and second embodiments, focusing on main operations and signal flows. [Figure 4] FIG. 13 is a diagram showing an example of roles and control contents that can be set for a sub-camera in an embodiment. [Diagram 5] Flowchart of role determination process in the first embodiment [Figure 6] 1 is a flowchart showing the operation of each device in the imaging system according to the first and second embodiments. [Figure 7] FIG. 1 is a diagram for explaining coordinate conversion in an embodiment. [Figure 8] FIG. 1 is a diagram relating to subject detection and coordinate transformation in an embodiment. [Figure 9] Schematic diagram of the operation control of the sub-camera in the first embodiment. [Figure 10] FIG. 13 is a schematic diagram of another operational control of the sub camera in the first embodiment; [Figure 11] FIG. 1 is a diagram for explaining pan value calculation in an embodiment; [Figure 12] FIG. 1 is a diagram for explaining tilt value calculation in an embodiment; [Figure 13] FIG. 13 is a diagram showing an example of mapping of zoom values ​​between the main camera and the sub camera in the first embodiment; [Figure 14] 11 is a flowchart for determining the control content according to the role of the sub-camera in the first embodiment. [Figure 15] Schematic diagram of control according to the role of the sub-camera in the first embodiment. [Figure 16] Schematic diagram of control according to the role of the sub-camera in the first embodiment. [Figure 17] FIG. 13 is a schematic diagram of a tracking operation of a plurality of subjects in a modified example of the first embodiment; [Figure 18] Schematic diagram of an imaging system according to a second embodiment. [Figure 19] FIG. 11 is a diagram showing an example of roles and control contents that can be set in the second embodiment. [Figure 20] FIG. 13 is a schematic diagram showing an example of control regarding a tracking subject of a sub camera in the second embodiment; [Figure 21] Schematic diagram of an imaging system according to a third embodiment. [Figure 22] FIG. 13 is a block diagram showing an example of the functional arrangement of each device in the imaging system according to the third embodiment. [Diagram 23] FIG. 13 is a diagram illustrating the main operations and signal flows of an imaging control device according to a third embodiment. [Figure 24] Flowchart regarding the operation of the imaging control device in the third embodiment [Diagram 25] FIG. 13 is a diagram showing information associated with the subject's identification ID in the third and fourth embodiments; [Figure 26] FIG. 13 is a diagram showing an example of an overhead image in the third embodiment; [Figure 27] FIG. 13 is a diagram showing an example of an overhead image generated in the third embodiment; [Figure 28]FIG. 13 is a diagram showing an example of an overhead image generated in the third embodiment; [Figure 29] FIG. 13 is a diagram illustrating the main operations and signal flows of an imaging control device according to a fourth embodiment. [Diagram 30] Flowchart regarding the operation of the imaging control device in the fourth embodiment [Diagram 31] FIG. 13 is a diagram showing an example of an image generated in the fourth embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] <First embodiment> (Overview of the multi-camera imaging system) 1 is a schematic diagram showing an example of the configuration of a multi-camera imaging system 10 (hereinafter simply referred to as an imaging system) according to this embodiment. The imaging system 10 has multiple cameras 300, 400, 500, an imaging control device 100, and a role control device 600. The multiple cameras 300, 400, 500, the imaging control device 100, and the role control device 600 are communicatively connected via a communication network 700.

[0011] The communication network 700 complies with known wired or wireless communication standards such as the IEEE802.3 series, IEEE802.11 series, etc. Each of the multiple cameras 300, 400, 500, the shooting control device 100, and the role control device 600 has a communication interface that complies with the standard of the communication network 700.

[0012] Of the multiple cameras 300, 400, and 500, camera 300 captures the entirety of a predetermined shooting range. The shooting range is set as a range in which a subject to be shot may exist in a studio, for example. Therefore, the video of camera 300 captures all of the subjects within the shooting range.

[0013] The purpose of the camera 300 is to capture an image for detecting a subject of the shooting target existing within the shooting range. Therefore, the shooting direction and angle of view of the camera 300 are determined according to the position of the camera 300 and the shooting range, and are basically fixed during shooting. Furthermore, the focusing distance may also be basically fixed in order to obtain a pan-focus image of the shooting range. Moreover, it is preferable that the camera 300 captures the entire shooting range so that it is not hidden by objects outside the shooting range. Therefore, the camera 300 is installed in a position where it overlooks the entire shooting range. In order to distinguish the camera 300 from the other cameras 400 and 500 whose shooting direction and angle of view during shooting are not basically fixed, the camera 300 will be referred to as an overhead camera below. However, the installation position of the camera 300 is not limited to a position where it overlooks the shooting range. The operation of the overhead camera 300 can be controlled by the shooting control device 100.

[0014] The cameras 400 and 500 are, for example, PTZ cameras, and their operations, including the shooting direction (pan and tilt angles) and the angle of view (zoom), can be controlled from an external device. Here, it is assumed that the user of the imaging system controls the operation of the camera 500, and the shooting control device 100 controls the operation of the camera 400. In the following, the camera 500 is called the main camera and the camera 400 is called the sub-camera, since the shooting control device 100 controls the operation of the camera 400 based on the state of the camera 500. For ease of explanation and understanding, only one sub-camera 400 is shown, but there may be two or more sub-cameras. The main camera 500 may be directly operated by the user. The cameras 400 and 500 may be configured so that the shooting direction (pan and tilt angles) can be controlled by attaching the camera body to a tripod head. The cameras 400 and 500 may be configured so that a zoomable interchangeable lens is attached to the camera body.

[0015] Here, in this embodiment, it is assumed that the role control device 600 has an operator. The shooting control device 100 may also have an operator (user), but this is not essential. The operator of the role control device 600 may be the user of the shooting control device 100. Since the shooting of the overhead camera 300 and the sub camera 400 is controlled by the shooting control device 100, a photographer is not required. It is assumed that the main camera 500 has an operator or photographer. In this way, a configuration that does not require operators or photographers for some devices can achieve labor saving.

[0016] 1 shows that all signals are communicated through the communication network 700, but for example, the video signal and the control signal may be communicated in a different manner. For example, each of the multiple cameras 300, 400, 500 may directly supply the video signal to the shooting control device 100 via a cable. The cameras 300, 400, 500 and the shooting control device 100 have a communication circuit according to the standard of the video signal. The standard of the video signal is, for example, the SDI (Serial Digital Interface) standard or the HDMI (High-Definition Multimedia Interface) (registered trademark), but is not limited to these.

[0017] The photography control device 100 detects a subject from a video signal received from the overhead camera 300. The photography control device 100 determines the photography direction and angle of view of the sub-camera based on the subject detection result, the state of the main camera 500, and the role set for the sub-camera. The photography control device 100 transmits a control command including the determined photography direction and angle of view to the sub-camera 400. By changing the role setting, the method of determining the photography direction and angle of view of the sub-camera 400 can be changed, and the degree of freedom of the operation control of the sub-camera 400 can be increased.

[0018] (Example of functional configuration of each device) Fig. 2 is a block diagram showing an example of the functional configuration of each device constituting the multi-camera imaging system 10 shown in Fig. 1. The configuration expressed as a functional block in the drawing can be realized by an integrated circuit such as an ASIC or an FPGA, by a discrete circuit, or by a combination of a memory and a processor that executes a program stored in the memory. Also, one functional block may be realized by multiple integrated circuit packages, or multiple functional blocks may be realized by one integrated circuit package. Also, the same functional block may be implemented in different configurations depending on the operating environment, required capabilities, etc.

[0019] (Photography control device 100) First, a functional configuration example of the imaging control device 100 will be described. The imaging control device 100 may be a general-purpose computer device such as a personal computer or a workstation. The imaging control device 100 has a configuration in which a CPU 101, a RAM 102, a ROM 103, an inference unit 104, a network interface (I / F) 105, a user input unit 106, and a display unit 108 are mutually connected via an internal bus 110.

[0020] The CPU 101 is a microprocessor capable of executing programmed instructions. The CPU 101, for example, loads a program stored in a ROM 103 into a RAM 102 and executes the program to realize the functions of the imaging control device 100, which will be described later. The CPU 101, for example, can realize the functions of the imaging control device 100 by executing an imaging control application that runs on an operating system (OS).

[0021] The RAM 102 is used to load programs executed by the CPU 101, and to temporarily store data to be processed by the CPU 101, data being processed, etc. In addition, a part of the RAM 102 may be used as a video memory for the display unit 108.

[0022] The ROM 103 is a rewritable non-volatile memory, and stores the programs (OS and applications) executed by the CPU 101, user data, and the like.

[0023] The inference unit 104 executes a detection process of a subject area using a machine learning model for the image of the overhead camera 300. The inference unit 104 can be implemented using a hardware circuit capable of executing the calculation of the machine learning model at high speed, such as a GPU (Graphics Processing Unit) or an NPU (Neural network Processing Unit). Alternatively, the inference unit 104 may be implemented using a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array). The CPU 101 may execute a program to realize the function of the inference unit 104.

[0024] The machine learning model may be a convolutional neural network (CNN) trained according to the type of subject to be detected. Here, the inference unit 104 detects a human body region or a human face region as a subject region from an input image. The inference unit 104 outputs the position and size of a rectangular region inscribed in the subject region and the detection reliability for each detected subject region. Note that, a plurality of types of machine learning models may be used to execute detection processing of different types of subject regions for the same input image. Note that the inference unit 104 may execute detection processing of the subject region by a known method that does not use a machine learning model. The inference unit 104 can detect the subject region by using, for example, a method using local features such as SIFT or SURF, or a method using pattern matching.

[0025] The network I / F 105 is an interface for connecting the photography control device 100 to the communication network 700. The photography control device 100 (CPU 101) can communicate with external devices on the communication network 700, such as the overhead camera 300, the sub camera 400, the main camera 500, and the role control device 600, through the network I / F 105. The photography control device 100 may communicate with external devices through other communication interfaces (USB, Bluetooth (registered trademark), etc.) not shown.

[0026] To communicate with each device (overhead camera 300, sub camera 400, main camera 500, role control device 600) on communication network 700, CPU 101 acquires the network address of each device at any timing and stores it in RAM 102. CPU 101 also acquires information about each device (device type, model name, etc.) at any timing (for example, at the time of the first communication) and stores it in RAM 102. In this way, CPU 101 is assumed to know at least the identification information and device type of overhead camera 300, sub camera 400, main camera 500, and role control device 600. Note that the user may be allowed to give any name to each device.

[0027] The user input unit 106 is an input device such as a mouse, a keyboard, a touch panel, etc. The imaging control device 100 receives instructions from the user through the user input unit 106.

[0028] The display unit 108 is a display device such as a liquid crystal display (LCD), etc. The display unit 108 displays a GUI screen provided by the OS, a shooting control application, etc.

[0029] (Overhead camera 300) Next, an example of the functional configuration of the overhead camera 300 will be described. The CPU 301 is a microprocessor capable of executing programmed instructions. For example, the CPU 301 loads a program stored in the ROM 303 into the RAM 302 and executes the program to control the operation of each functional block and realize the function of the overhead camera 300, which will be described later.

[0030] The RAM 302 is used to load programs executed by the CPU 301 and to temporarily store data to be processed by the CPU 301, data being processed, etc. The RAM 302 may also be used as a buffer for video signals obtained by shooting.

[0031] The ROM 308 is a rewritable non-volatile memory. The ROM 308 stores the programs executed by the CPU 301, the setting values ​​of the overhead camera 300, user data, etc. The ROM 308 can also be used as a recording destination for video signals. The ROM 308 may include an internal memory and a removable memory card.

[0032] The image sensor 307 has a photographing optical system and an image sensor. The image sensor may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The image sensor has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from each pixel. Each pixel accumulates electric charge according to the amount of incident light by photoelectric conversion. A pixel signal group (analog image signal) representing the subject image formed on the imaging surface is obtained by reading out from each pixel a signal having a voltage according to the amount of electric charge accumulated during the exposure period.

[0033] The image processing unit 306 applies predetermined signal processing and image processing to the analog image signal output by the image sensor 307 to generate signals and image data according to the application, and to obtain and / or generate various types of information.

[0034] The processes applied by the image processing unit 306 can include, for example, pre-processing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effect processing. Pre-processing may include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on the image sensor 307, and is processing for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging optical system (image restoration), correction of the effect of peripheral light falloff of the imaging optical system, color correction, and the like. The data processing may include processes such as cutting out an area (trimming), compositing, scaling, encoding and decoding, generating header information (generating a data file), etc. The data processing also includes the generation of a video signal to be outputted to the outside and video data to be recorded in the ROM 308. The evaluation value calculation process may include processes such as generation of a signal and evaluation value used in automatic focus detection (AF), generation of an evaluation value used in automatic exposure control (AE), etc. AF and AE are executed by the CPU 301. The special effects processing may include adding a blur effect, changing color tones, relighting, and the like. Note that these are merely examples of processes that the image processing unit 306 can apply, and do not limit the processes that the image processing unit 306 can apply. The image processing unit 306 outputs the acquired or generated information and data to the CPU 301, RAM 302, etc. depending on the purpose.

[0035] The type and settings of the processing applied by the image processing unit 306 can be controlled by sending commands from the photography control device 100 to the overhead camera 300.

[0036] The network I / F 305 is an interface for connecting the overhead camera 300 to the communication network 700. The overhead camera 300 (CPU 301) can communicate with external devices on the communication network 700, such as the shooting control device 100, the sub camera 400, the main camera 500, and the role control device 600, via the network I / F 305. The overhead camera 300 may communicate with external devices via other communication interfaces (USB, Bluetooth, etc.) not shown.

[0037] (Sub-camera 400) Next, a description will be given of an example of the functional configuration of the sub-camera 400. It is assumed that functional blocks with the same names in the sub-camera 400 and the overhead camera 300 have the same functions, and the description will be omitted.

[0038] As described above, the sub-camera 400 is a PTZ camera, and the shooting direction and angle of view can be controlled externally. Therefore, the sub-camera 400 has a driving unit 409 capable of panning, tilting, and zooming, and a driving I / F 408. The driving I / F 408 is a communication interface between the driving unit 409 and the CPU 401.

[0039] The driving unit 409 has a pan / tilt mechanism that supports the sub-camera 400 so that it can pan and tilt, a zoom mechanism that changes the angle of view of the photographing optical system, and a motor that drives these mechanisms. The zoom mechanism may use the image enlargement / reduction by the image processing unit 406. The driving unit 409 drives the motor according to instructions received from the CPU 401 via the driving I / F 408, and adjusts the optical axis direction and the angle of view of the photographing optical system.

[0040] (Main camera 500) Next, an example of the functional configuration of the main camera 500 will be described. It is assumed that functional blocks with the same names in the main camera 500 and the sub camera 400 have the same functions, and their description will be omitted. The main camera 500 is operated by a user. Here, it is assumed that the user remotely controls the main camera 500 by transmitting commands through the communication network 700. However, when the main camera 500 is not a PTZ camera, the user may directly operate the main camera 500.

[0041] The shooting control device 100 (CPU 101) can obtain information on the shooting direction and angle of view of the sub camera 400 and the main camera 500 from the sub camera 400 and the main camera 500 via the network I / F 505. The shooting direction may be the pan and tilt angles of the driving units 409 and 509 with a predetermined reference direction set to 0°. The reference direction may be the direction directly facing the shooting range.

[0042] (Role control device 600) Next, an example of the functional configuration of the role control device 600 will be described. The CPU 601 is a microprocessor capable of executing programmed instructions. For example, the CPU 601 loads a role setting program stored in the ROM 603 into the RAM 602 and executes the program to control the operation of each functional block and realize the functions of the role control device 600.

[0043] The RAM 602 is used for loading programs executed by the CPU 601 and for temporarily storing data to be processed by the CPU 601, data being processed, etc. In addition, a part of the RAM 602 may be used as a video memory for the display unit 608.

[0044] The ROM 603 is a rewritable non-volatile memory, and stores the programs executed by the CPU 601, the setting values ​​of the role control device 600, user data, and the like.

[0045] The user input unit 611 is an input device such as a button, a dial, a joystick, a touch panel, etc. The role control device 600 receives, via the user input unit 611, a user's instruction regarding the setting of the role of the sub-camera 400.

[0046] The network I / F 605 is an interface for connecting the role control device 600 to the communication network 700. The role control device 600 (CPU 601) can communicate with external devices on the communication network 700, such as the overhead camera 300, the sub camera 400, the main camera 500, and the shooting control device 100, through the network I / F 605. The role control device 600 may communicate with external devices through other communication interfaces (USB, Bluetooth, etc.) not shown.

[0047] The display unit 608 is a display device such as a liquid crystal display (LCD), etc. The display unit 608 displays a GUI screen provided by the OS, a role setting application, etc.

[0048] The role control device 600 stores role setting information, for example, in the ROM 603. The role setting information is information in which identification information of the sub camera 400 is associated with information indicating the role that has been set. The CPU 601 executes a role setting application to display a role setting screen on the display unit 608. The role setting screen displays, for example, identification information of the sub camera 400 (such as a network address and a name set by the user) and the name of the currently set role in association with each other. The initial value of the currently set role may be a default role that has been set in advance. The user can change the current role that is displayed in association with a desired sub camera 400 by operating the user input unit 611.

[0049] When a user operation indicating the end of the setting operation, such as an operation of an OK button included in the role setting screen, is detected, the CPU 601 updates the role setting information stored in the ROM 103 according to the content of the role setting screen.

[0050] When the CPU 601 receives a role acquisition command via the network I / F 605, it reads out the role setting information stored in the ROM 103 and transmits it to the sender of the role acquisition command.

[0051] 1 and 2, the role control device 600 is described as an independent device, but for example, a shooting control application executed by the shooting control device 100 may provide a function similar to that of the role control device 600. Also, a role may be set directly to the sub-camera 400, and the shooting control device 100 may acquire the role assigned to the sub-camera 400 from the sub-camera 400.

[0052] The role that can be set for the sub camera 400 is a predetermined role that determines how information obtained from the main camera 500 is used to control the operation of the sub camera 400. As an example, the information from the main camera is used to control the tracking subject and zoom operation of the sub camera 400.

[0053] Fig. 4 shows an example of types of roles that can be set for the sub camera 400 and control contents associated with the roles. The control contents for each role can be stored in the ROM 603 of the role control device 600 and the ROM 103 of the shooting control device 100 in the table format shown in Fig. 4, for example. Here, it is assumed that any of "main follow (MF)", "main counter (MC)", "assist follow (AF)", and "assist counter (AC)" can be set as the role ROLE. When there are multiple sub cameras 400, a role can be set for each sub camera.

[0054] For the sub-camera 400 with the role "main follow", the photography control device 100 (CPU 101) sets the same tracking subject as the main camera 500, and when the main camera 500 is zoomed, the photography control device 100 (CPU 101) performs zoom control of the sub-camera 400 in the same phase. Here, "in phase" means that the zoom direction (telephoto direction or wide-angle direction) is the same, that is, the direction of the angle of view change is the same. On the other hand, "opposite phase" means that the zoom direction (telephoto direction or wide-angle direction) is the opposite direction, that is, the direction of the angle of view change is the opposite direction. Note that even if the zoom direction is in the same phase, the angle of view does not have to be the same as the main camera 500, and whether in the same phase or opposite phase, the degree of zoom change (such as the speed or rate of change) does not have to be the same as the main camera 500.

[0055] For the sub-camera 400 whose ROLE is "main counter", the photography control device 100 (CPU 101) sets the same tracking subject as the main camera 500, and when the main camera 500 is zoomed, the photography control device 100 (CPU 101) controls the sub-camera 400 to zoom down when the main camera 500 is zoomed up. Note that zooming up refers to changing the zoom in the telephoto direction (telephoto end direction), and zooming down refers to changing the zoom in the wide-angle direction (wide end direction). When zoom control is performed by the image processing unit 406, zooming up refers to making the area cut out from the image smaller and the magnification rate of the cut out area larger than before the area change. On the other hand, zooming down refers to making the area cut out from the image larger and the magnification rate of the cut out area smaller than before the area change.

[0056] For the sub-camera 400 whose ROLE is "assist follow", the photography control device 100 (CPU 101) sets a tracking subject different from that of the main camera 500, and when the main camera 500 is zoomed, the photography control device 100 (CPU 101) performs zoom control of the sub-camera 400 in the same phase.

[0057] For the sub-camera 400 whose ROLE is "assist counter", the photography control device 100 (CPU 101) sets a tracking subject different from that of the main camera 500. In addition, when the main camera 500 is zoomed, the photography control device 100 performs zoom control of the sub-camera 400 in the opposite phase.

[0058] Here, for the sub-camera 400 with the role of "assist follow" and "assist counter", a subject existing on the left side (left end) of the image other than the target subject of the main camera 500 is set as the tracking subject of the sub-camera 400. The tracking subject of the sub-camera 400 may be set according to other conditions. For example, a subject existing on the right side (right end), upper side (upper end), or lower side (lower end) of the image other than the target subject of the main camera 500 may be set as the tracking subject of the sub-camera. Alternatively, a subject existing on the front or back of the image other than the target subject of the main camera 500 may be set as the tracking subject of the sub-camera 400.

[0059] Also, only one of setting the subject to be tracked and zoom control may be performed, or another control item may be added.

[0060] In the role setting information stored in the ROM 603 by the role control device 600, information indicating the role ROLE (such as the above-mentioned type name or a number assigned to the type) is associated with identification information of the sub camera 400. The CPU 101 of the photography control device 100 acquires the role setting information from the role control device 600, and executes operation control of the sub camera 400 according to the type of role ROLE set to the sub camera 400.

[0061] The role control device 600 may notify an external device (for example, the shooting control device 100) when there is a change in the role setting for the sub-camera 400. This allows the change in the role setting to be immediately reflected in the operation control of the sub-camera 400.

[0062] <Explanation of operation of each device> Next, the operation of each device in the multi-camera imaging system will be described. Here, the imaging control device 100 automatically controls the imaging operation of the sub-camera 400 based on the image of the overhead camera 300, information obtained from the main camera 500, and the role set for the sub-camera 400.

[0063] 3 is a diagram showing a series of processes performed by the photography control device 100 when controlling the operation of the sub-camera 400, focusing on the main operations and signal flows. The functional blocks shown in the photography control device 100 show the main operations in a schematic manner, and correspond to the main functions provided by the photography control application. Each functional block in FIG. 3 is realized by a combination of the CPU 101 that executes the photography control application and one or more of the functional blocks of the photography control device 100 shown in FIG. 2.

[0064] Fig. 5 is a flowchart showing the operation of CPU 101 as role determination section 120. Fig. 6(a) to Fig. 6(d) are flowcharts relating to the operation of shooting control device 100, overhead camera 300, main camera 500, and sub-camera 400, respectively.

[0065] In the following description, it is assumed that the three-dimensional coordinate values ​​of the viewpoint position and the shooting direction (optical axis direction) of the overhead camera 300 are known to the shooting control device 100. It is also assumed that known position information, such as the three-dimensional coordinate values ​​of the viewpoint positions of the sub camera 400 and the main camera 500 and the coordinate values ​​of markers arranged in the shooting range, is stored in advance in the ROM 103 as default position information REF_POSI. It is also assumed that the coordinate system of the position is predetermined depending on the type of position.

[0066] (Operation of role determination unit 120) First, the operation of the CPU 101 as the role determination section 120 in Fig. 3 will be described with reference to the flowchart shown in Fig. 5. The operation described below is realized by the CPU 101 executing a shooting control application.

[0067] 5 is not particularly limited to the timing for starting the operations shown in the flowchart of Fig. 5, but the operations are executed at least before starting control of the shooting operation of the sub-camera 400. The operations are also executed when a notification is received from the role control device 600 via the network I / F 105 that the role setting for the sub-camera 400 has been changed.

[0068] In S101, the CPU 101 as the role determination unit 120 acquires a role ROLE (role setting information) corresponding to the sub camera 400 from the role control device 600. The CPU 101 can acquire the above-mentioned role setting information from the role control device 600 by transmitting a role acquisition command to the role control device 600 via the network I / F 105, for example. The CPU 101 stores the acquired role setting information in the RAM 102.

[0069] In S103, CPU 101 refers to the role setting information stored in RAM 102 based on the identification information of sub camera 400, and acquires operation control content for sub camera 400. Then, CPU 101 as role determination section 120 transmits the acquired operation control content (CAMERA_ROLE) to tracking subject determination section 123. In practice, CPU 101 stores the operation control content in a specific area of ​​RAM 102, and refers to it when functioning as tracking subject determination section 123.

[0070] In S104, the CPU 101 as the role determination unit 120 transmits the acquired operation control content (CAMERA_ROLE) to the zoom value calculation unit 125. In practice, the CPU 101 stores the operation control content in a specific area of ​​the RAM 102 and refers to it when functioning as the zoom value calculation unit 125.

[0071] (Operation of the imaging control device 100) Next, the operation of the shooting control device 100 to control shooting by the sub-camera 400 will be described with reference to Fig. 3 and Fig. 6(a). The operation described below corresponds to the operation of the CPU 101 as the recognition unit 121, the target subject determination unit 122, the tracking subject determination unit 123, the pan / tilt value calculation unit 124, and the zoom value calculation unit 125 in Fig. 3. Note that the operation described below is realized by the CPU 101 executing a shooting control application.

[0072] In S201, CPU 101 transmits a shooting instruction command in a predetermined protocol to overhead camera 300 through network I / F 105. In response to this command, overhead camera 300 starts supplying a video signal (moving image data) IMG to video input unit 107. CPU 101 starts storing the video signal received by video input unit 107 in RAM 102, and then executes S202.

[0073] In S202, CPU 101 acquires information ANGLE indicating the shooting direction from main camera 500. Specifically, CPU 101 transmits a shooting direction acquisition command to main camera 500 via network I / F 105 in a predetermined protocol. In response to the shooting direction acquisition command, CPU 501 of main camera 500 transmits information ANGLE indicating the current shooting direction of main camera 500 to shooting control device 100. Information ANGLE may be, for example, the pan and tilt angles of drive unit 509. CPU 101 stores the acquired information ANGLE in RAM 102.

[0074] In S203, the recognition unit 121 executes the following process. (1) Apply subject area detection processing to the input frame image and store the detection results. (2) For each detected subject area, the position information (image coordinates) is converted. (3) Apply classification processing to each detected subject region to identify the classification information (if a new subject is detected, add classification processing information). (4) For each detected subject area, the identification information ID[n] and the position information POSITION[n] are associated and stored.

[0075] The recognition unit 121 is mainly realized by the CPU 101 and the inference unit 104. The CPU 101 reads one frame of the image received from the overhead camera 300 from the RAM 102 and inputs it to the inference unit 104.

[0076] The operation of the recognition unit 121 will be explained step by step below. (1) First, the inference unit 104 inputs a frame image into a machine learning model to detect a subject region. The inference unit 104 stores in the RAM 102 the position and size of each detected subject region, which are output by the machine learning model as a detection result, and the detection reliability. The position and size of the subject region may be any information capable of identifying the position and size of a rectangular region inscribed by the subject region. Here, the center coordinate of the bottom side, and the width and height of the rectangular region are used as the position and size of the subject region.

[0077] Furthermore, inference unit 104 stores the detection result for the first frame image in RAM 102 in association with the subject identification information ID[n]. Here, n is the subject number and is an integer ranging from 1 to the total number of detected subject regions. Furthermore, inference unit 104 stores the subject regions detected from the first frame image in RAM 102 in association with the subject identification information ID[n] as templates for identifying individual subjects. If template matching is not used to identify subjects, it is not necessary to store templates.

[0078] Fig. 8(a) shows an example of the result of subject detection processing by inference unit 104 for the image captured by overhead camera 300 shown in Fig. 7(a). Here, the areas of human subjects A to C present within shooting range 20 are detected, and the coordinates (foot coordinates) of the center of the bottom side of a rectangular area inscribed with the subject areas are output as the position.

[0079] For coordinate conversion, which will be described later, when a marker is placed at a known position within the shooting range 20 as shown in FIG. 7(b), the CPU 101 detects the image of the marker included in the frame image (FIG. 7(a)) and stores the position in the RAM 102. The detection of the marker image may also be performed by the inference unit 104. The detection of the marker image may be performed by any known method, such as pattern matching using a marker template. The marker image may also be detected using a pre-stored machine learning model for marker detection.

[0080] (2) Next, we will explain the coordinate conversion executed by the inference unit 104. Fig. 7(a) shows a schematic diagram of the image captured by the overhead camera 300, and Fig. 7(b) shows a schematic diagram of the image capture range 20 as viewed from directly above its center. The inference unit 104 converts the position of the subject area in the coordinate system of the overhead camera into values ​​in a coordinate system (planar coordinate system) when the image capture range 20 is viewed from directly above its center.

[0081] The reason for converting the coordinates into values ​​in the plane coordinate system here is that it is convenient to calculate a pan value (a movement angle in a horizontal plane) for shooting a specific subject with the sub-camera 400. Note that this is based on the premise that the sub-camera 400 is installed so that the driving unit 409 performs a panning operation in a horizontal plane parallel to the floor of the shooting range 20.

[0082] Coordinate conversion can be performed in various ways, but here, markers are placed at multiple known positions on the floor of the shooting range 20, and coordinate conversion is performed from the overhead camera coordinate system to the planar coordinate system based on the marker positions in the image captured by the overhead camera 300. Note that coordinate conversion may also be performed without using markers, for example, by using the viewpoint position and shooting direction of the overhead camera 300.

[0083] The coordinate transformation can be performed using a homography transformation matrix H according to Equation 1 below.

number

[0084] The homography transformation matrix can be calculated by substituting the coordinates of the four markers detected from the video and the coordinates (known) of the four markers arranged in the shooting range 20 into Equation 1 and solving the simultaneous equations. When the positional relationship between the shooting range 20 and the overhead camera 300 is fixed, the homography transformation matrix H can be calculated in advance during test shooting and stored in, for example, ROM 103.

[0085] CPU 101 sequentially reads out the positions of the subject regions from RAM 102 and converts the coordinates into values ​​in a planar coordinate system. Fig. 8(b) shows a schematic diagram of the state in which the foot coordinates (x, y) of each subject region detected in the overhead camera 300 video shown in Fig. 8(a) are converted into coordinate values ​​(X, Y) in a planar coordinate system using Equation 1 and the homography transformation matrix H stored in ROM 103. CPU 101 stores the converted foot coordinates in RAM 102 as POSITION[n].

[0086] (3) Next, the operation of the inference unit 104 to identify the identification information ID[n] of the subject will be described. Here, the subject is identified using template matching. The subject is identified for the processing results of the second and subsequent subject detections. For the first processing result, a new identification information ID[n] can be assigned to the subject region.

[0087] The inference unit 104 identifies the identification information ID[n] of the detected object region by template matching using the template stored in the RAM 102. This identifies the object within the shooting range. For example, the inference unit 104 calculates an evaluation value representing the correlation of each template for each detected object region. Then, the inference unit 104 identifies the identification information ID[n] corresponding to the template having a certain degree of correlation or higher and having the highest correlation as the identification information ID[n] of the object region. For the evaluation value, a known value such as the sum of absolute differences of pixel values ​​can be used.

[0088] Note that, for a subject region that does not have a certain level of correlation with all the templates, the inference unit 104 assigns new identification information ID[n] and adds the image of the subject region to the template.

[0089] In addition, the inference unit 104 may update an existing template using a subject area detected in a most recent frame image, or delete a template in which a subject area having a certain degree of correlation or more has not existed for a certain period of time. Furthermore, the inference unit 104 may store in the ROM 103 templates corresponding to frequently appearing identification information ID[n].

[0090] Note that the subject may be identified by a method other than template matching. For example, the subject may be identified as the same identification information ID[n] as the subject area having the closest immediately preceding detection position and / or size. Alternatively, the position in the current frame image may be predicted by a Kalman filter or the like from the transition of positions in multiple past detection results associated with the same identification information, and the same identification information ID may be identified as the subject area that is closest to the predicted position. These methods may also be combined. By not using template matching, it is possible to improve the accuracy of identifying different subjects that look similar.

[0091] (4) The inference unit 104 associates the specified identification information ID[n] with the position (planar coordinate system) POSITION[n] of the corresponding subject region and stores them in the RAM 102.

[0092] Of the processes (1) to (4), the processes other than the subject detection may be executed by the CPU 101 instead of the inference unit 104.

[0093] Here, the image from the overhead camera 300 is used to determine the identification information ID[n] and position POSITION[n] of the subject within the shooting range 20. However, the image from the sub camera 400 may also be used. When there are multiple sub cameras 400, the CPU 101 executes the operation shown in the flowchart of FIG. 6(a) for each sub camera 400. The position of the subject area is output as a value in the coordinate system for each sub camera 400. Thus, although the overhead camera 300 is not essential, it is considered that the use of the overhead camera 300 provides better subject detection accuracy.

[0094] Returning to the explanation of Fig. 6(a), in S204, CPU 101 as the subject of interest determination unit 122 in Fig. 3 determines a subject of interest as a tracking subject of main camera 500. CPU 101 can determine a subject of interest of main camera 500 from among the subjects detected in S203, based on the shooting direction of main camera 500 acquired in S202. CPU 101 stores identification information ID[n] corresponding to the subject area determined as the subject of interest of main camera 500 in RAM 102 as identification information MAIN_SUBJECT of the subject of interest.

[0095] For example, CPU 101 can determine that the subject closest to the shooting direction of main camera 500 in the planar coordinate system is the subject of interest of main camera 500. Note that, when there are multiple subjects whose distance from the shooting direction of main camera 500 is equal to or less than a threshold, the user may be allowed to select the subject of interest from among them.

[0096] When allowing the user to select a subject of interest, CPU 101 displays on display unit 108 or an external display device the frame image to which subject detection processing has been applied in S202, together with an indicator indicating the shooting direction and an indicator indicating a subject area that is a candidate for a subject of interest. The indicator for the subject area may be, for example, a rectangular frame indicating the outer edge of the subject area as shown in Fig. 8(a), but may be another indicator. CPU 101 may also display on display unit 108 a message prompting the user to select a subject of interest within the image.

[0097] The user can select a subject area corresponding to a desired subject of interest by operating the user input unit 106 (input device). There are no particular limitations on the selection method, but the selection method may be an operation of operating a mouse or keyboard to specify the desired subject area.

[0098] When the CPU 101 detects a user operation for designating a subject region, the CPU 101 stores in the RAM 102 the identification information ID[n] corresponding to the designated subject region as identification information MAIN_SUBJECT of the target subject.

[0099] Next, in S205, CPU 101 as tracking subject determination unit 123 in Fig. 3 acquires control content CAMERA_ROLE according to the role set for sub camera 400. Specifically, CPU 101 reads out control content CAMERA_ROLE acquired in the role determination process described with reference to Fig. 5 and stored in RAM 102. Note that, if there are multiple sub cameras 400, CPU 101 executes the processes of S205 to S207 for each sub camera.

[0100] In S206, the CPU 101 as the tracking subject determination unit 123 determines a subject to be tracked and photographed by the sub camera 400 in accordance with the control content CAMERA_ROLE. The CPU 101 determines a tracking subject of the sub camera 400 in accordance with the tracking subject rule (FIG. 4) included in the control content CAMERA_ROLE.

[0101] When the tracking subject of the sub camera 400 is to be the same as the target subject of the main camera 500, the CPU 101 sets the identification information MAIN_SUBJECT of the target subject determined in S203 as the identification information SUBJECT_ID of the tracking subject of the sub camera 400.

[0102] When the tracking subject of the sub camera 400 is a subject located on the left side of the subject other than the target subject of the main camera 500, the CPU 101 detects a subject area located on the left edge of the subject area other than the target subject from among the subject areas detected in S203. Then, the CPU 101 sets the identification information ID[n] corresponding to the detected subject area as the identification information SUBJECT_ID of the tracking subject of the sub camera 400.

[0103] CPU 101 writes the identification information SUBJECT_ID of the determined tracking subject to RAM 102. If the tracking subject may differ depending on the sub camera, CPU 101 stores the identification information SUBJECT_ID of the tracking subject in association with the identification information of the sub camera. Note that if the tracking subject changes, CPU 101 retains the information of the previous tracking subject in RAM 102 without erasing it.

[0104] Here, the operation when the role set for the sub camera 400 is "main follow" will be described with reference to Fig. 9. The photography control device 100 controls the sub camera 400 set for the role "main follow" so as to track the subject of interest of the main camera 500.

[0105] Therefore, when the subject of interest of the main camera 500 is determined to be subject B as shown in Fig. 9(a), the CPU 101 determines subject B as the subject to be tracked by the sub camera 400. Thereafter, when the subject of interest of the main camera 500 is determined to have changed to subject A as shown in Fig. 9(b), the CPU 101 changes the subject to be tracked by the sub camera 400 to subject A. Similarly, when the subject of interest of the main camera 500 is determined to have changed to subject C as shown in Fig. 9(c), the CPU 101 changes the subject to be tracked by the sub camera 400 to subject C.

[0106] The operation of the sub camera 400 when the role set for the sub camera 400 is "assist follow" will be described with reference to Fig. 10. For the sub camera 400 set for the role "assist follow", the photography control device 100 controls the sub camera 400 so that it tracks a subject located on the left side of the subject other than the subject of interest of the main camera 500.

[0107] Therefore, when the subject of interest of the main camera 500 is determined to be subject B as shown in Fig. 10(a), the CPU 101 determines the subject A on the left side of subjects A and C as the subject to be tracked by the sub camera 400. Thereafter, when the subject of interest of the main camera 500 is determined to have changed to subject A as shown in Fig. 10(b), the CPU 101 changes the subject to be tracked by the sub camera 400 to subject B on the left side of subjects B and C. Also, when the subject of interest of the main camera 500 is determined to have changed to subject C as shown in Fig. 10(c), the CPU 101 changes the subject to be tracked by the sub camera 400 to subject A on the left side of subjects A and B.

[0108] By dynamically changing the role set for the sub-camera 400 using the role control device 600, the subject tracked by the sub-camera 400 can be changed, enabling flexible automatic shooting.

[0109] Returning to Fig. 6(a), in S207, CPU 101 as pan / tilt value calculation unit 124 calculates the change amounts of pan angle and tilt angle required for sub camera 400 to track and photograph the tracking subject determined in S206. CPU 101 as zoom value calculation unit 125 calculates the zoom value of sub camera 400 according to the change in angle of view of main camera 500. Although the following describes a case where there is one sub camera 400, when there are multiple sub cameras 400, calculation of the change amounts of pan angle and tilt angle and calculation of the zoom value are executed for each sub camera.

[0110] First, the operation of the CPU 101 as the pan-tilt value calculation unit 124 will be described. Here, it is assumed that the following information is stored in advance in the ROM 103 as the default position information REF_POSI for each sub-camera 400. · 3D coordinates of the installation position (values in a plane coordinate system) · Shooting direction corresponding to the initial values of the pan angle and tilt angle of the drive unit · Controllable range of the pan and tilt angles

[0111] The CPU 101 reads the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking subject of the sub-camera 400 from the RAM 102. Then, the CPU 101 determines the pan angle first from the position information POSITION_OH and the installation position of the sub-camera 400.

[0112] FIG. 11 is a diagram showing an example of the positional relationship between the sub-camera 400 and the tracking subject in a plane coordinate system. Here, it is assumed that the pan angle θ that directs the optical axis direction of the sub-camera 400 toward the subject position is determined. The CPU 101 calculates the pan angle θ using the following Equation 2.

Equation

[0113] In Equation 2, px and py are the horizontal and vertical coordinates of the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking subject. Also, subx and suby are the horizontal and vertical coordinates of the installation position of the sub-camera. Here, it is assumed that the current pan angle is the initial value of 0° and the optical axis direction is the vertical direction (Y-axis direction). If the current optical axis direction is not the vertical direction, the angle difference between the current optical axis direction and the vertical direction may be reflected in the angle obtained by Equation 2. Also, the direction of the pan is counterclockwise if subx > px, and clockwise if subx < px.

[0114] Next, with reference to FIG. 12, a method for determining the tilt angle will be described. FIG. 8 shows a state in which the sub-camera and the tracked subject are viewed from the side. Assume that the current optical axis of the sub-camera 400 is in the horizontal direction and has a height of h1, and the height of the face of the tracked subject facing the optical axis is h2. Let the angle difference (tilt angle) in the height direction between the current optical axis direction and the target optical axis direction be ρ. The CPU 101 calculates the tilt angle ρ using the following equations (3) and (4). [Number]

[0115] The coordinate values used in Equation (4) are the same as those used in Equation (2). Assume that h1 and h2 are input in advance to the shooting control application and stored in the RAM 102. In this case, the identification number associated with h2 for each subject is made equal to the identification number assigned in the subject detection process. Alternatively, h2 may be a value measured in real time using a sensor (not shown).

[0116] Here, assume that the current tilt angle is the initial value of 0°, and the optical axis direction is the horizontal direction (constant height). If the current optical axis direction is not the horizontal direction, the angle difference between the current optical axis direction and the horizontal direction may be reflected in the angle obtained by Equation (4). Also, the tilt direction is downward if h1 > h2, and upward if h1 < h2.

[0117] The CPU 101 communicates with the sub-camera 400 periodically through the communication network 700, acquires the current optical axis direction (the pan angle and tilt angle of the drive unit), and stores it in the RAM 102. Note that the communication cycle can be, for example, less than or equal to the reciprocal of the frame rate. Alternatively, the CPU 101 may hold the total values of the pan angle and tilt angle controlled from the initial state for the sub-camera 400 in the RAM 102 and use them as the current optical axis direction.

[0118] In this way, CPU 101 calculates the amount of change in the pan angle and tilt angle of sub camera 400, and stores the amount of change in RAM 102. If there are multiple sub cameras 400, CPU 101 calculates the amount of change in the pan angle and tilt angle for each sub camera.

[0119] The change amount of the pan angle and tilt angle may be the angular velocity for the sub camera 400 to rotate in the direction of the tracking subject. For example, CPU 101 acquires the current pan angle and tilt angle from sub camera 400 through communication network 700. Then, CPU 101 calculates the angular velocity of the pan proportional to the difference between the pan angle θ read from RAM 102 and the current pan angle. CPU 101 also calculates the angular velocity of the tilt proportional to the difference between the tilt angle ρ read from RAM 102 and the current tilt angle. CPU 101 stores the angular velocity calculated in this manner in RAM 102.

[0120] The change amount of the pan angle and the tilt angle may be calculated using the image of the sub camera 400 instead of the image of the overhead camera 300. In this case, the CPU 101 may calculate the change amount of the pan angle from the horizontal difference between the current optical axis direction and the direction of the tracked subject in the coordinate system of the sub camera 400, and may calculate the change amount of the tilt angle from the vertical difference. Also, the imaging system may change the shooting direction to track and shoot the tracked subject in only one of the pan direction and the tilt direction, and in such an imaging system, the change amount of only one of the pan angle and the tilt angle may be calculated.

[0121] Next, a description will be given of the operation of CPU 101 as zoom value calculation unit 125. CPU 101 as zoom value calculation unit 125 periodically acquires information MAIN_ZOOM indicating the angle of view of main camera 500, and stores it in RAM 102. Then, when the information MAIN_ZOOM changes, CPU 101 calculates a zoom value Z_VALUE for sub camera 400 according to control content CAMERA_ROLE according to the role set for sub camera 400.

[0122] CPU 101 can determine the zoom operation and its phase of main camera 500, for example, by detecting a change in the angle of view of the image of main camera 500. For example, the change in the angle of view may be detected from a change over time in the size or interval of the subject area.

[0123] 13 shows an example of mapping of zoom values ​​of the main camera and the sub camera. Here, it is assumed that the main camera 500 and the sub camera 400 optically change the angle of view (the photographing optical system has a zoom function). However, a similar function may be realized by digital zoom using the image processing units 406 and 506.

[0124] The zoom value is a parameter having a value corresponding to the angle of view, and in this embodiment, the smaller (narrower) the angle of view, the smaller the zoom value, and the zoom value on the telephoto side is smaller than the zoom value on the wide-angle side. The sub-camera 400 and the main camera 500 are capable of controlling the photographing optical system to an angle of view corresponding to the zoom value by transmitting a command specifying the zoom value. That is, the zoom value is information related to the angle of view and is information representing the zoom state. The zoom value may be, for example, the focal length (mm) of the photographing optical system corresponding to a 35 mm full-size image sensor, in which case the zoom value on the telephoto side is larger than the zoom value on the wide-angle side.

[0125] 13, the range of the zoom value MAIN_ZOOM of the main camera 500 is from main_min to main_max. The zoom range of the sub camera 400 is from sub_min to sub_max. main_min and sub_min are the zoom values ​​corresponding to the telephoto ends of the main camera 500 and the sub camera 400, respectively, and main_max and sub_max are the zoom values ​​corresponding to the wide-angle ends of the main camera 500 and the sub camera 400, respectively. FIG. 13 shows an example in which the range of the zoom value of the main camera 500 is wider than the range of the zoom value of the sub camera 400 at both the telephoto end and the wide-angle end.

[0126] When controlling the zoom value SUB_ZOOM of the sub camera 400 to be in phase with the zoom value MAIN_ZOOM of the main camera 500, the CPU 101 calculates SUB_ZOOM corresponding to the current MAIN_ZOOM using the following formula 5.

number

[0127] On the other hand, when controlling the zoom value SUB_ZOOM of the sub camera 400 to be in the opposite phase to the zoom value MAIN_ZOOM of the main camera 500, the SUB_ZOOM corresponding to the current MAIN_ZOOM is calculated using the following formula 6. Specifically, the CPU 101 calculates the SUB_ZOOM corresponding to the current MAIN_ZOOM by substituting the SUB_ZOOM calculated using formula 5 into the right side of the following formula 6. SUB_ZOOM=sub_max-(SUB_ZOOM-sub_min) (Formula 6)

[0128] When the main camera 500 performs digital zoom and controls the angle of view by cropping, the CPU 101 can determine the zoom value SUB_ZOOM of the sub camera 400 according to the size of the range to be cropped by the main camera 500. Specifically, the CPU 101 sets the zoom value SUB_ZOOM smaller (higher magnification) as the size of the range to be cropped by the main camera 500 becomes smaller, and sets the zoom value SUB_ZOOM larger (lower magnification) as the size becomes larger.

[0129] In addition, the content of the zoom control associated with the role of the sub camera 400 is not limited to the control in the same phase or opposite phase to the main camera 500. For example, a zoom operation independent of the change in the angle of view of the main camera 500 may be associated with the role. For example, an auto-zoom operation that maintains the size of a tracking subject constant may be associated with the role. Also, the angle of view of the sub camera 400 may be fixed to a specific angle of view. By adding roles associated with these zoom controls to the control contents for each role shown in FIG. 4, or by changing the content of the zoom control of the role shown in FIG. 4, various zoom controls for the sub camera 400 are possible.

[0130] Returning to FIG. 6(a), in S207, CPU 101 reads out the change amounts of the pan and tilt angles calculated in S206 and the zoom value from RAM 102. Then, CPU 101 generates a control command PT_VALUE that instructs sub camera 400 to change the pan angle and tilt angle corresponding to these change amounts. CPU 101 also generates a control command Z_VALUE that instructs sub camera 400 to change the angle of view corresponding to the zoom value. The format of the control command is predefined. CPU 101 stores the generated control commands PT_VALUE and Z_VALUE in RAM 102. Note that S207 may be skipped when it is not necessary to generate a control command, such as when the tracking subject is stationary or when the angle of view of main camera 500 does not change.

[0131] Then, the CPU 101 reads out the control commands PT_VALUE and Z_VALUE from the RAM 102, and transmits them to the communication network 700 via the network I / F 105. The sub-camera 400 receives the control commands PT_VALUE and Z_VALUE via the network I / F 405.

[0132] The CPU 101 executes the process from S201 on the next frame image of the video captured by the overhead camera 300. Note that the process shown in Fig. 6(a) does not necessarily have to be executed for every frame.

[0133] (Operation of overhead camera 300) Next, the operation of the overhead camera 300 will be described with reference to Fig. 6(b). The operation described below is realized by the CPU 301 executing a program.

[0134] When the overhead camera 300 is powered on, each functional block is initialized by the CPU 301, and the camera enters a shooting standby state. In the shooting standby state, the CPU 301 may start a video shooting process for live view display, and output display image data generated by the image processing unit 306 to the shooting control device 100 via the network I / F 305.

[0135] In the shooting standby state, the CPU 301 waits for reception of a control command through the network I / F 305. When the CPU 301 receives a control command, it executes an operation according to the control command. Here, an operation when a shooting command is received as a control command from the shooting control device 100 will be described.

[0136] In S301 , the CPU 301 receives an imaging command from the imaging control device 100 via the network I / F 305 .

[0137] The shooting command may specify shooting parameters such as frame rate, resolution, etc. Also, the shooting command may include settings related to the processing to be applied by the image processing unit 306.

[0138] In S302, in response to receiving the shooting command, the CPU 301 starts a moving image shooting process for supplying the image to the shooting control device 100. In this moving image shooting process, a moving image with higher image quality is shot than that of the moving image shooting process for live view display. For example, at least one of the resolution of the moving image and the shooting frame rate is higher than that of the moving image for live view display. The image processing unit 306 applies processing to the image based on the settings for the moving image to be supplied to the shooting control device 100. The image processing unit 306 sequentially stores the generated moving image data in the RAM 302.

[0139] In S303, the CPU 101 reads the video data from the RAM 302, and transmits it to the shooting control device 100 via the network I / F 305. After that, the process from shooting to supply of video data is continued until a control command to stop shooting is received.

[0140] (Operation of main camera 500) Next, the operation of the main 500 will be described with reference to Fig. 6(c). The operation described below is realized by the CPU 501 executing a program.

[0141] When the main camera 500 is powered on, each functional block is initialized by the CPU 501, and then moving image shooting processing for supplying to the shooting control device 100 is started. The image processing unit 506 applies processing to the analog image signal obtained from the image sensor 507 based on the settings for moving images to be supplied to the shooting control device 100. The image processing unit 506 sequentially stores the generated moving image data in the RAM 502. The CPU 501 reads the moving image data from the RAM 502 and supplies it to the shooting control device 100 via the network I / F 505.

[0142] The CPU 501 waits for a control command via the network I / F 305 while supplying video data to the shooting control device 100. When the CPU 501 receives a control command, it executes an operation according to the control command. Here, an operation when a shooting direction acquisition command is received will be described. When a pan / tilt control command PT_VALUE or a zoom control command Z_VALUE is received, the CPU 501 drives the drive unit 509 according to the command.

[0143] In S501, the CPU 501 receives a shooting direction obtaining command via the network I / F 505. The CPU 501 stores the received shooting direction obtaining command in the RAM 502.

[0144] In S502 , in response to receiving the shooting direction acquisition command, the CPU 501 acquires the current pan angle and tilt angle from the drive unit 509 via the drive I / F 508 , and stores them in the RAM 502 .

[0145] In S503, the CPU 501 reads out the current pan angle and tilt angle from the RAM 502, and transmits them to the shooting control device 100 via the network I / F 305 as shooting direction information ANGLE.

[0146] (Operation of sub camera 400) Next, the operation of the sub-camera 400 will be described with reference to Fig. 6(d). The operation described below is realized by the CPU 401 executing a program.

[0147] When the sub-camera 400 is powered on, each functional block is initialized by the CPU 401, and then moving image capturing processing for supplying to the shooting control device 100 is started. The image processing unit 406 applies processing to an analog image signal obtained from the image sensor 407 based on settings for moving images to be supplied to the shooting control device 100. The image processing unit 406 sequentially stores the generated moving image data in the RAM 402. The CPU 401 reads the moving image data from the RAM 402 and supplies it to the shooting control device 100 via the network I / F 405.

[0148] The CPU 401 waits for a control command via the network I / F 305 while supplying video data to the shooting control device 100. When the CPU 401 receives a control command, it executes an operation according to the control command. Here, an operation when a pan / tilt control command PT_VALUE and a zoom control command Z_VALUE are received from the shooting control device 100 will be described.

[0149] In S401, the CPU 401 receives at least one of a pan / tilt control command PT_VALUE and a zoom control command Z_VALUE from the shooting control device 100 via the network I / F 405. The CPU 401 stores the received control command in the RAM 402.

[0150] In S402, the CPU 401 reads out the operation direction and the corresponding operation amount from the control command stored in the RAM 402, and stores them in the RAM 402. Here, in the case of a pan / tilt control command PT_VALUE, the operation direction is the pan and / or tilt direction, and the operation amount is the target angle. Also, in the case of a zoom control command Z_VALUE, the operation amount is the zoom value, and since the operation direction can be specified from the zoom value, there is no need to read out and store the operation direction.

[0151] In S403, the CPU 401 generates drive parameters for the drive unit 409 based on the operation direction and operation amount read out in S403. For example, the CPU 401 may obtain drive parameters according to a combination of the operation direction and operation amount by using a table previously stored in the ROM 403. Note that when the operation amount is given as a target value (target angle or zoom value), the CPU 410 obtains the drive parameters from the difference with the current value.

[0152] In S404, the CPU 401 controls the drive unit 409 through the drive I / F 408 based on the drive parameters acquired in S404. As a result, the drive unit 409 changes the shooting direction of the sub-camera 400 to the operation direction and angle specified by the pan / tilt control command PT_VALUE. In addition, the drive unit 409 changes the angle of view of the shooting optical system to the zoom value specified by the zoom control command Z_VALUE.

[0153] Next, the operation of the shooting control device 100 to control the shooting direction (pan and tilt) and the angle of view (zoom value) of the sub-camera in accordance with the role set for the sub-camera 400 will be described in more detail with reference to the flowchart shown in Fig. 14. The operation shown in the flowchart in Fig. 14 is executed as part of the operation of S205 to S207 in Fig. 6(a).

[0154] S601 corresponds to S205, and the CPU 101 reads out the control content CAMERA_ROLE stored in the RAM 102 in S103 of FIG.

[0155] S602 to S607 are performed in S206, for example. In S602, CPU 101 determines whether the definition of the tracking subject of sub camera 400 included in the control content CAMERA_ROLE indicates the tracking subject (target subject) of main camera 500. For example, if the definition of the tracking subject of sub camera 400 has a value indicating "same as main", CPU 101 determines that the definition of the tracking subject of sub camera 400 indicates the tracking subject of main camera 500 and executes S603. On the other hand, if the definition of the tracking subject of sub camera 400 has a value indicating "different from main (left side)", CPU 101 determines that the definition of the tracking subject of sub camera 400 does not indicate the tracking subject of main camera 500 and executes S604.

[0156] In S603, CPU 101 determines to control the shooting direction of sub camera 400 so as to track a tracking subject (target subject) of main camera 500. In S604, CPU 101 determines to control the shooting direction of sub camera 400 so as to track a subject located on the left side of the subject other than the subject of interest of main camera 500.

[0157] In S605, CPU 101 determines whether the zoom control provision of sub camera 400 included in the control content CAMERA_ROLE indicates control in the same phase as main camera 500. For example, if the zoom control provision of sub camera 400 has a value indicating "same phase as main", CPU 101 determines that the zoom control provision of sub camera 400 indicates control in the same phase as main camera 500 and executes S606. On the other hand, if the tracking subject provision of sub camera 400 has a value indicating "opposite phase to main", CPU 101 determines that the zoom control provision of sub camera 400 does not indicate control in the same phase as main camera 500 and executes S607.

[0158] In S606, CPU 101 determines to control the zoom value (angle of view) of sub camera 400 in the same phase as the change in the zoom value of main camera 500. In S607, CPU 101 determines to control the zoom value (angle of view) of sub camera 400 in the opposite phase to the change in the zoom value of main camera 500.

[0159] An example of control of the sub-camera when the role set for the sub-camera 400 is "main follow" will be described with reference to FIG. 15. FIG. 15 shows a schematic diagram of how the shooting control device 100 controls the shooting direction and angle of view of the sub-camera 400 when the target subject and angle of view of the main camera 500 change over time during shooting. In the figure, time passes from left to right. In FIG. 15, the zoom state is shown in three stages: "telephoto end", "middle", and "wide-angle end". This is because the range of the zoom value may differ between the sub-camera and the main camera as shown in FIG. 13. "Telephoto end" corresponds to a state where the zoom is zoomed up to the telephoto end, "wide-angle end" corresponds to a state where the zoom is zoomed down to the wide-angle end, and "middle" corresponds to a zoom state between the telephoto end and the wide-angle end, but the actual zoom value may differ between the sub-camera and the main camera. For example, when the zoom state is "telephoto end", the zoom value of the main camera is main_min, and the zoom value of the sub-camera is sub_min.

[0160] Initially, the subject of interest (tracking subject) of main camera 500 is subject B, and the zoom state is "medium." Therefore, CPU 101 determines the subject to be tracked by sub camera 400 to be subject B, and controls the shooting direction so that sub camera 400 tracks subject B. CPU 101 also controls the zoom state of sub camera 400 to be "medium."

[0161] After that, the subject of interest of main camera 500 is changed from subject B to subject A, and the zoom state is changed to the "telephoto end." In response to this, CPU 101 changes the subject being tracked by sub camera 400 from subject B to subject A, and controls the shooting direction so that sub camera 400 tracks subject A. CPU 101 also controls the zoom state of sub camera 400 to the "telephoto end."

[0162] After that, the subject of interest of main camera 500 is changed from subject A to subject C, and the zoom state is changed to "wide-angle end." In response to this, CPU 101 changes the subject being tracked by sub camera 400 from subject A to subject C, and controls the shooting direction so that sub camera 400 tracks subject C. CPU 101 also controls the zoom state of sub camera 400 to "wide-angle end."

[0163] In this way, when the role of the sub camera 400 is "main follow", the CPU 101 automatically changes the tracking subject and zoom value of the sub camera 400 so as to follow the change in the focus subject and angle of view (zoom value) of the main camera 500. In the example shown in FIG. 15, the sub camera 400 is controlled so that the degree of change in the zoom state is equal to the degree of change in the zoom state of the main camera 500, but the actual zoom value may be different as long as the direction of change in the zoom value is in phase. For example, when the zoom state of the main camera 500 is at the telephoto end, the zoom state of the main camera 500 does not have to be at the telephoto end. Whether or not to match the zoom value of the sub camera 400 to the zoom value of the main camera 500 may be settable using role setting information.

[0164] Next, an example of control of the sub-camera when the role set for the sub-camera 400 is "assist counter" will be described with reference to FIG. 16 which is similar to FIG.

[0165] Initially, the subject of interest (tracking subject) of main camera 500 is subject B, and the zoom state is "telephoto end." Therefore, CPU 101 determines the subject to be tracked by sub camera 400 to be subject A on the left side out of subjects A and C other than subject B, and controls the shooting direction so that sub camera 400 tracks subject A. CPU 101 also controls the zoom state of sub camera 400 to the "wide-angle end" which is in the opposite phase to main camera 500.

[0166] Thereafter, the focus subject of main camera 500 is changed from subject B to subject A, and the zoom state is changed to "middle". In response to this, CPU 101 changes the tracking subject of sub camera 400 to subject B on the left side of subjects B and C other than subject B, and controls the shooting direction so that sub camera 400 tracks subject B. In addition, since the zoom state of main camera 500 has changed from "telephoto end" to "middle", CPU 101 controls the zoom state of sub camera 400 from "wide-angle end" to "middle" (in the opposite phase).

[0167] After that, the subject of interest of main camera 500 is changed from subject A to subject C, and the zoom state is changed to "wide-angle end." In response to this, CPU 101 changes the subject being tracked by sub camera 400 to subject A on the left side of subjects A and B other than subject C, and controls the shooting direction so that sub camera 400 tracks subject A. In addition, because the zoom state of main camera 500 has changed from "middle" to "wide-angle end," CPU 101 controls the zoom state of sub camera 400 from "middle" to "telephoto end" (in the opposite phase).

[0168] In this way, when the role of sub camera 400 is "assist counter", CPU 101 automatically changes the tracking subject of sub camera 400 to a subject other than the subject of interest of main camera 500 in response to a change in the subject of interest of main camera 500. CPU 101 also automatically changes the zoom value of sub camera 400 in the opposite direction to the change in the angle of view (zoom value) of main camera 500.

[0169] 16, the zoom state of the sub camera 400 is controlled at a degree of change equal to that of the main camera 500, but the amount of change in the zoom value may be different as long as the direction of change in the zoom value is in the opposite phase. For example, when the zoom state of the main camera 500 is at the telephoto end, the zoom state of the main camera 500 does not have to be at the wide-angle end. The degree of change in the zoom state of the sub camera 400 relative to the degree of change in the zoom state of the main camera 500 may be set using role setting information.

[0170] (Modification) So far, we have described an example of automatically controlling the tracking subject and zoom value of the sub camera 400 based on the target subject and zoom value of the main camera 500. In the above example, the sub camera is automatically controlled to track a single subject, but it can also be automatically controlled to track multiple subjects within the shooting range.

[0171] When the role set for the sub-camera 400 is "assist follow", examples of control in the case where the sub-camera 400 is made to track a single subject and in the case where the sub-camera 400 is made to track multiple subjects will be described with reference to FIG. 17. For ease of understanding and explanation, a case will be described in which the angle of view of the main camera 500 does not change and only the control of the subject to be tracked is executed. Also, it is assumed that the angle of view of the sub-camera 400 is always capable of photographing all subjects within the photographing range 20 regardless of the photographing direction. Note that the photographing direction of the sub-camera 400 shown in the top row of FIG. 17 indicates the photographing direction when a single subject is tracked.

[0172] As in FIG. 16, initially, the subject of interest (tracking subject) of the main camera 500 is subject B. Therefore, the CPU 101 determines the tracking subject of the sub camera 400 to be subject A on the left side among subjects A and C other than subject B, and controls the shooting direction so that the sub camera 400 tracks subject A. If the shooting direction is controlled so that the tracking subject is located at the center of the screen, as shown in the second row from the bottom, the image of the sub camera 400 will be an image in which subjects A to C are shifted to the right, resulting in poor balance. Therefore, when multiple subjects including the tracking subject are present in the image of the sub camera 400, the shooting direction can be controlled so that the multiple subjects are tracked. For example, the CPU 101 can control the shooting direction so that the center of gravity of the positions of the multiple subjects A to C included in the image of the sub camera 400 is tracked. As a result, the sub camera 400 shoots an image as shown in the bottom row.

[0173] In the example shown in FIG. 17, even if the sub-camera 400 tracks any one of the subjects A to C, all of the subjects A to C are captured, so the capturing direction of the sub-camera 400 remains almost constant even if the subject of interest of the main camera 500 changes.

[0174] (Variation 2) Also, after determining the tracking subject of the sub camera 400, the CPU 101 may control the sub camera 400 to focus on the tracking subject. Basically, the CPU 401 continuously controls the focusing distance so that the sub camera 400 focuses on the designated tracking subject, but the AF frame of the sub camera 400 can be set to the position of the tracking subject from the shooting control device 100. This allows the sub camera 400 to quickly and reliably focus on the tracking subject. Note that if the AF frame is set when the pan speed slows down (below the threshold), there is a high possibility that the tracking target will be located in the center of the screen, and therefore the time required for focusing can be shortened.

[0175] (Variation 3) Also, the sub camera 400 can be controlled without using the overhead camera 300. In this case, the shooting direction of the sub camera 400 can be determined from the installation positions of the main camera 500 and the sub camera 400 and the shooting direction of the main camera 500 (the orientation of the main camera 500 with respect to the tracking subject). The main camera 500 executes a subject detection process, and the CPU 101 acquires and uses the result to control the sub camera 400. For example, the CPU 101 acquires an image of the subject area from the main camera 500 as a result of the subject detection process. Then, the CPU 101 can control the sub camera 400 to execute a subject tracking process using the acquired image as a template. Alternatively, the CPU 101 may use the acquired image as a template and control the sub camera 400 to track a subject area with a low correlation with the template.

[0176] (Variation 4) Although the shooting control device 100, the role control device 600, and the main camera 500 have been described as independent devices, the functions of the shooting control device 100 and the role control device 600 can also be incorporated into the main camera 500. In this case, the video of the overhead camera 300 is supplied to the main camera 500. With this configuration, it is possible to reduce the number of devices required to realize a multi-camera imaging system.

[0177] As described above, according to this embodiment, when the operation of the sub camera is automatically controlled based on the state and image of the main camera, automatic control is executed according to the role set for the sub camera. Therefore, the photography control device of this embodiment can realize automatic photography control with a higher degree of freedom while realizing labor saving.

[0178] <Second embodiment> Next, a second embodiment of the present invention will be described. In this embodiment, a tracking subject of a sub camera is determined taking into account information based on the state or image of the main camera, the role set for the sub camera, and the image of the sub camera.

[0179] Fig. 18 is a schematic diagram showing a configuration example of an imaging system 10' according to this embodiment. In Fig. 18, the same components as those in the imaging system 10 according to the first embodiment are given the same reference numerals as those in Fig. 1, and the description thereof will be omitted. The imaging system 10' according to this embodiment has two sub-cameras, a sub-camera A800 and a sub-camera B900. The functional configurations of the sub-camera A800 and the sub-camera B900 are similar to those of the sub-camera 400 described in the first embodiment, and therefore the description thereof will be omitted.

[0180] In this embodiment, a role is set in the shooting control device 100 from the role control device 600, and the role setting information includes different control contents for each sub camera. The CPU 101 controls the operation of each sub camera according to the role set in the shooting control device 100.

[0181] Fig. 19 is a diagram showing an example of role setting information in this embodiment. Here, only the role "assist follow" and the corresponding control contents for each sub camera are shown as an example. However, the types of roles that can be set in the shooting control device 100 and the control contents corresponding to the types of roles are not limited to the example shown in Fig. 19.

[0182] Here, it is stipulated that the sub camera A800 is controlled to set the left subject of the subjects different from the main camera's tracking subject as the tracking subject and to focus on the tracking subject. Also, it is stipulated that the sub camera B900 is controlled to set the right subject of the subjects different from the main camera's tracking subject as the tracking subject and to focus on the tracking subject. Here, since it is known in advance that the total number of subjects is 3, it is stipulated as the left or right of two people, but it may simply be stipulated as the left or right. Note that zoom control may be stipulated as in the first embodiment, but zoom control is omitted for the sake of simplicity of explanation.

[0183] 19 is stored in advance in the ROM 103 of the shooting control device 100. Alternatively, the role setting information may be supplied from the role control device 600 to the shooting control device 100, and the CPU 101 may store it in the RAM 102.

[0184] Using FIG. 20, a description will be given of how the shooting control device 100 controls the sub-camera A 800 and the sub-camera B 900 based on the role setting information.

[0185] Fig. 20(a) shows the positional relationship between shooting range 20, main camera 500, sub camera A 800, and sub camera B 900 at the start of shooting, and subjects A to C within shooting range 20. Fig. 20(b) shows a state in which, after shooting starts, subject A moves from the left side to the right side of subject B, and subject C moves from the right side to the left side of subject B.

[0186] At the stage of Fig. 20(a), the CPU 101 of the shooting control device 100 controls the operations of the sub-camera A800 and the sub-camera B900 based on the role setting information shown in Fig. 19. That is, the CPU 101 determines that of the subjects A and C other than the target subject (subject B) of the main camera 500, the subject A on the left side is the subject to be tracked by the sub-camera A800, and the subject C on the right side is the subject to be tracked by the sub-camera B900.

[0187] The CPU 101 controls the shooting directions of the sub-camera A 800 and the sub-camera B 900 so as to track the determined tracking subject, and also controls the sub-camera A 800 and the sub-camera B 900 so as to focus on the determined tracking subject.

[0188] In the state of FIG. 20(b), the subject A tracked by the sub camera A800 does not satisfy the condition of "subject on the left side" (other than the subject of interest of the main camera 500). Therefore, the CPU 101 changes the subject tracked by the sub camera A800 to subject C on the left side of subjects A and C other than the subject of interest of the main camera 500 (subject B). Similarly, the subject C tracked by the sub camera B900 does not satisfy the condition of "subject on the right side". Therefore, the CPU 101 changes the subject tracked by the sub camera B900 to subject A on the right side of subjects A and C other than the subject of interest of the main camera 500 (subject B). The CPU 101 also controls the sub camera A800 and the sub camera B900 to focus on the determined subject to be tracked.

[0189] When a subject moves significantly, as in Fig. 20(b), there is a high possibility that another subject exists in front, making it easier for the subject to be tracked to be hidden. By setting the sub-camera positioned to the left of the shooting range 20 to track a subject that exists on the left side, it is possible to specify that the tracking subject is effectively changed when the tracking subject moves significantly to the right. The same applies to the sub-camera positioned to the right of the shooting range 20.

[0190] In this embodiment, it is possible to define the control contents so that each sub-camera with the same role tracks a different subject. Therefore, according to the shooting control device 100 of this embodiment, it is possible to automatically control the sub-camera so that it shoots images tracking various subjects based on the state of the main camera and information obtained from the image.

[0191] (Modification) In this embodiment, the shooting direction of the sub-camera for tracking a specific subject is estimated by performing coordinate conversion on the position of the subject area detected from the image of the overhead camera 300. However, a similar estimation may be performed based on the images of the sub-camera A800 and the sub-camera B900. In this case, although the processing load of the shooting control device 100 increases, since coordinate conversion is not required, the accuracy of the control of the shooting direction of the sub-camera can be improved.

[0192] (Variation 2) In the present embodiment, the pan / tilt values ​​for controlling the sub-camera A800 and the sub-camera B900 are calculated so as to automatically track the tracking target. However, automatic tracking is not essential. For example, the role setting information may specify that the pan values ​​of the sub-camera A800 and the sub-camera B900 are controlled according to the zoom value of the main camera 500.

[0193] As an example, when the zoom state of the main camera 500 is the "wide-angle end", the sub-camera A800 can be controlled to face in a direction diagonally 45 degrees to the left, and the sub-camera B900 can be controlled to face in a direction diagonally 45 degrees to the right. On the other hand, when the zoom state of the main camera 500 is the "telephoto end", both the sub-camera A800 and the sub-camera B900 can be controlled to face in the center 0 degree direction.

[0194] By performing such control, it becomes possible to control the shooting directions of the multiple sub-cameras in synchronization with the zoom state of the main camera 500, and the performance effect can be enhanced. For example, when the main camera 500 zooms in on a specific subject, the sub-cameras A800 and B900 can change the shooting directions while zooming in on the same subject in conjunction with the main camera 500. By performing such control, it becomes possible to simultaneously view multiple images shot by multiple cameras that are synchronously controlled in a performance in which multiple camera images are displayed on multiple monitors. In addition, by automatically controlling each camera with the same role, the variation in the time change in the angle of view between the cameras is smaller than when each camera is manually operated, and the sense of unity of the angle of view change is stronger, and a performance effect that enhances the sense of realism can be expected.

[0195] (Variation 3) In the present embodiment, the main camera 500 is controlled as the master, and the sub-camera A800 and the sub-camera B900 are controlled as the slaves. However, the master-slave relationship between the main camera 500 and the sub-camera A800 may be dynamically changed. For example, the camera capturing the main line video among the main camera 500, the sub-camera A800, and the sub-camera B900 may be controlled as the master, and the other cameras may be controlled as the slaves. In this case, the shooting control device 100 may obtain information indicating which camera's video is selected as the main line video from an external device such as a switcher that selects the video, or may determine the information based on a tally signal. The same control may be performed not only when the main line video is selected as the video related to the viewer's viewing or recording, such as the video for recording.

[0196] The master-slave relationship between the cameras may be switched when the sub-camera A 800 or the sub-camera B 900 is manually operated. In this case, when the user manually operates one of the sub-cameras, the control of the other cameras is started with the sub-camera as the master, which is convenient.

[0197] <Third embodiment> Next, a third embodiment of the present invention will be described. The imaging system according to this embodiment improves the usability of the imaging system by providing a user with visualized information on the imaging states of the main camera and the sub camera.

[0198] FIG. 21 is a schematic diagram showing a configuration example of an imaging system 10″ according to a third embodiment of the present invention. Here, for ease of explanation and understanding, a configuration example based on the imaging system 10 according to the first embodiment is shown. The explanation of the first embodiment is applied to the configuration other than the display device 800.

[0199] The display device 800 presents information about the sub camera 400 and the main camera 500, particularly information about the subject being photographed, to a user who uses the imaging system. In this embodiment, the display device 800 is described as a separate configuration from the photography control device 100 and the role control device 600, but the function of the display device 800 may be implemented by at least one of the photography control device 100 and the role control device 600.

[0200] The display device 800 displays the photographing information generated by the photographing control device 100. When the photographing control device 100 generates image data for display, the display device 800 may be a device that provides only a display function, such as a liquid crystal display, or any electronic device that can function as an external display device for the photographing control device 100. The display device 800 may also be any electronic device (such as an information processing device, a smartphone, or a tablet terminal) that has a display function and can communicate with the photographing control device 100. The display device 800 may generate and display image data for display based on the photographing information obtained from the photographing control device 100.

[0201] FIG. 22 is a block diagram showing an example of the functional configuration of each device constituting the imaging system 10'' shown in FIG. 21. For devices other than the display device 800, the description in the first embodiment with reference to FIG. 2 is cited.

[0202] The display device 800 has a configuration in which a CPU 801 , a RAM 802 , a ROM 803 , a user input unit 804 , a network interface (I / F) 805 , an image processing unit 806 , and a display unit 811 are interconnected via an internal bus 810 .

[0203] The CPU 801 is a microprocessor capable of executing programmed instructions. The CPU 801, for example, loads a program stored in a ROM 803 into a RAM 802 and executes the program to realize the functions of an imaging control device 800, which will be described later. The CPU 801, for example, executes a display control application that runs on an operating system (OS) to realize the functions of the display device 800.

[0204] The RAM 802 is used to load programs executed by the CPU 801, and to temporarily store data to be processed by the CPU 801, data currently being processed, etc. In addition, a part of the RAM 802 may be used as a video memory for the display unit 811.

[0205] The ROM 803 is a rewritable non-volatile memory, and stores the programs (OS and applications) executed by the CPU 801, user data, and the like.

[0206] The user input unit 804 is an input device such as a mouse, a keyboard, a touch panel, etc. The display device 800 receives instructions from a user through the user input unit 804.

[0207] The network I / F 805 is an interface for connecting the display device 800 to the communication network 700. The display device 800 (CPU 801) can communicate with external devices on the communication network 700, such as the imaging control device 100, through the network I / F 805. Note that the display device 800 may communicate with external devices through other communication interfaces (USB, Bluetooth (registered trademark), etc.) not shown.

[0208] The CPU 801 acquires information (eg, network addresses) required for communication with devices on the communication network 700 at any timing, and stores the information in the RAM 802 .

[0209] The image processing unit 806 generates image data (display image data) to be displayed on the display unit 811 from the image data acquired via the network I / F 805. The image processing unit 806 also generates display image data based on the shooting states of the main camera 500 and the sub camera 400 received from the shooting control device 100. The image processing unit 806 stores the generated display image data in the RAM 802. The image processing unit 806 may apply compression processing to the display image data as necessary to reduce the amount of data. The image processing unit 806 may also apply image quality adjustments such as color correction, exposure control, and sharpness correction to the received image data.

[0210] Display unit 811 is a display device such as a liquid crystal display (LCD). Display unit 811 displays a GUI screen provided by an OS, a display control application, etc. Display unit 811 also displays an image showing the shooting state of main camera 500 and sub camera 400.

[0211] The operation of the display device 800 is controlled by user instructions given through a user input unit 804 and instructions (commands) received through a network I / F 805 from the imaging control device 100 or the like.

[0212] 23 is a diagram showing a series of processes performed when the shooting control device 100 controls the operation of the sub-camera 400, focusing on main operations and signal flows, similar to FIG. 3. The functional blocks shown in the shooting control device 100 show the main operations in a schematic manner, and correspond to the main functions provided by the shooting control application. Each functional block in FIG. 23 is realized by a combination of the CPU 101 that executes the shooting control application and one or more of the functional blocks of the shooting control device 100 shown in FIG. 22.

[0213] As can be seen from the comparison between FIG. 23 and FIG. 3, the shooting control device 100 of this embodiment has the functions of the subject information superimposing unit 126 and the angle of view information superimposing unit 127. The subject information superimposing unit 126 superimposes information indicating the tracking subject in each of the main camera 500 and the sub camera 400 on the image acquired from the overhead camera 300. The angle of view information superimposing unit 127 superimposes information indicating the angle of view of each of the main camera 500 and the sub camera 400 on the image acquired from the overhead camera 300. The operations of the subject information superimposing unit 126 and the angle of view information superimposing unit 127 can be individually and dynamically switched between enabled and disabled. Note that the subject information superimposing unit 126 and the angle of view information superimposing unit 127 may be included in a display control device different from the shooting control device 100, and the display control device may receive various information from the shooting control device 100 and output an image generated by the display control device to the display device 800.

[0214] 24, an operation of generating an image on which shooting information from the main camera 500 and the sub camera 400 is superimposed will be described. The shooting control device 100 can execute the operation described here in parallel with other operations such as an automatic shooting control operation of the sub camera.

[0215] Fig. 24(a) is a flowchart relating to the operation of superimposing information on a tracked subject, which is one of the pieces of shooting information, on the image from overhead camera 300. Fig. 24(b) is a flowchart relating to the operation of superimposing information on the angle of view, which is one of the pieces of shooting information, on the image from overhead camera 300. Here, the information on the tracked subject is superimposed by subject information superimposing section 126, and further the information on the angle of view is superimposed by angle of view information superimposing section 127. However, the order of superimposition may be reversed, and the presence or absence of superimposition can be controlled individually.

[0216] Moreover, the operations shown in the flowcharts of FIG. 24(a) and FIG. 24(b) are executed every predetermined number of frames of the video captured by the overhead camera 300 (for example, every other frame, every other frame, etc.).

[0217] In S701, the CPU 101 starts acquiring a video signal (image data) IMG from the overhead camera 300 in the same manner as in S201.

[0218] In S702, CPU 101 acquires information ANGLE indicating the shooting direction from main camera 500, similarly to S202.

[0219] In S703, CPU 101 (recognition unit 121) reads one frame's worth of image data IMG from RAM 102, and executes the subject detection process described in S203. Then, CPU 101 determines the position at which to superimpose an index for each subject region based on the subject detection result. Here, it is assumed that a frame indicating a rectangular region inscribed in the subject region shown in FIG. 8(a) is superimposed as an index. Therefore, CPU 101 determines rectangular information DETECT_POSI for each subject region based on the output of inference unit 104 as information specifying the position of the rectangular region, and stores it in RAM 102.

[0220] Specifically, the rectangle information DETECT_POSI may be the image coordinates of the diagonal vertices of the rectangular area, which are the coordinates (xul, yul) of the upper left vertex and the coordinates (xdr, ydr) of the lower right vertex.

[0221] In S704, CPU 101 as the target subject determination unit 122 determines the target subject of main camera 500, similarly to S204. CPU 101 stores identification information ID[n] corresponding to the subject area determined as the target subject of main camera 500 in RAM 102 as identification information MAIN_SUBJECT of the target subject.

[0222] In S705, the CPU 101 as the tracking subject determination unit 123 acquires the control content CAMERA_ROLE according to the role set for the sub camera 400, similarly to S205.

[0223] In S706, CPU 101 as tracking subject determination unit 123 determines a subject to be tracked and photographed by sub camera 400 in accordance with control content CAMERA_ROLE, similar to S206. CPU 101 determines a tracking subject of sub camera 400 in accordance with the tracking subject rule (FIG. 4) included in control content CAMERA_ROLE.

[0224] When displaying an index indicating the shooting angle of view of the tracked subject without displaying an index for the subject area and an index for the tracked subject for each camera, CPU 101 executes S801 in FIG. 24B after completing the process of S706.

[0225] In S707, CPU 101 as subject information superimposition section 126 reads from RAM 102 the same image data IMG as that read in S703. Then, CPU 101 reads from RAM 102 rectangular information DETECT_POSI for each subject region, and superimposes an index based on the rectangular information DETECT_POSI on the image data IMG. As described above, the index is a rectangular frame whose diagonal vertices are the coordinates indicated by the rectangular information DETECT_POSI. Also, here, the color of the rectangular frame is made different for each subject region. CPU 101 (subject information superimposition section 126) stores the image data onto which the index for the subject region has been superimposed in RAM 102.

[0226] FIG. 25 is a diagram showing an example of the relationship between the identification ID of a subject detected from a frame image of the video of the overhead camera 300, the rectangle information DETECT_POSI, and the display color of the index for each subject region. The position information (rectangle information DETECT_POSI) of the rectangular area circumscribing the region of the subject with the identification ID "ID1" is (xul1, yul1), (xdr1, ydr1), and the color of the index "red" is associated with it. This indicates that for a subject determined to have an identification ID of ID1, a red index (here, a rectangular frame) is superimposed at the position indicated by the rectangle information in the image. The same applies to the other identification IDs, ID2 and ID3. The colors to be assigned to the identification IDs can be determined in advance.

[0227] 26 shows an example of an image represented by image data on which an index of a subject region is superimposed. Here, three subjects A to C are detected from an overhead image, and are assigned IDs ID1 to ID3, respectively. The overhead image is one frame of the video from the overhead camera 300.

[0228] When the colors of the indicators are defined as shown in FIG. 25, red, green and blue rectangular frame-shaped indicators 261 to 263 are superimposed on the areas of the subjects A to C, respectively.

[0229] The index for each subject is not limited to the rectangular frame-shaped index shown as an example. For example, the rectangular frame may include at least a part of the subject area. In addition, the area of ​​the rectangular frame other than the subject area may be filled with the same color as the frame. In addition, for example, the index may be semi-transparent. By making the index semi-transparent, it becomes possible to visually recognize the subject area even if the index overlaps the subject area. The index for each subject may or may not be adjacent to the associated subject area as long as the correspondence relationship with the subject area is clear.

[0230] In S708, CPU 101 reads out the image data generated in S707 from RAM 102. Then, CPU 101 generates image data SUBJECT_IMG in which identification information of the camera that has the subject as the tracking subject is superimposed near the area of ​​the subject tracked by main camera 500 and sub camera 400.

[0231] First, CPU 101 reads out MAIN_SUBJECT indicating the ID of the target subject (tracking subject) of main camera 500 and SUBJECT_ID indicating the ID of the tracking subject of sub camera 400 from RAM 102. CPU 101 superimposes the identification information of main camera 500 (here, the characters "Main Camera") near the subject area corresponding to the identification ID of MAIN_SUBJECT. Similarly, CPU 101 superimposes the identification information of sub camera 400 (here, the characters "Sub Camera") near the subject area corresponding to SUBJECT_ID.

[0232] In addition to the identification information of the sub camera 400, identification information of the role set to the sub camera 400 may be superimposed. For example, when the role "main follow" is set to the sub camera 400, the characters "sub camera (MF)" can be superimposed.

[0233] The position where the camera identification information is superimposed may be, for example, directly above a rectangular frame-shaped subject index. In this case, CPU 101 can determine the position where the camera identification information is superimposed from the rectangle information. Note that CPU 101 may also determine the position where the camera identification information is superimposed from position information of the subject area. Also, the camera identification information may be made semi-transparent.

[0234] The position at which the camera identification information is superimposed may be determined individually depending on the position of the corresponding subject area. For example, if the subject area is close to the edge of the image, the display position may be changed. Specifically, the camera identification information may be superimposed on the bottom of the subject area for a subject area close to the top edge of the image, and on the right of the subject area for a subject area close to the left edge. The position may also be dynamically changed taking into account the distance between the subject areas, overlap, etc.

[0235] Furthermore, the identification information of the camera is not limited to the role (main or sub) of the camera. For example, any information that can identify the camera, such as the IP address or model name of the camera, may be used. Furthermore, multiple types of identification information may be superimposed simultaneously or switchably.

[0236] Furthermore, the color of the camera identification information may be the color associated with the corresponding subject, i.e., the same color as the indicator for each subject area, or may be a fixed color that is determined in advance for each main camera and sub camera.

[0237] CPU 101 writes image data in which an index of the subject area and information indicating the tracking subject for each camera are superimposed on an overhead image as SUBJECT_IMG to RAM 102. When the angle of view information is not superimposed, CPU 101 may output image data SUBJECT_IMG to display device 800 as display image data OUT_IMAGE.

[0238] Fig. 27(a) to Fig. 27(c) show an example in which the main camera 500 and the sub camera 400 track different subjects for human subjects A to C present within the shooting range. Specifically, this is the case in which the sub camera 400 automatically shoots according to the role "assist follow" or "assist counter". Fig. 27(d) to Fig. 27(f) show image data SUBJECT_IMG generated corresponding to Fig. 27(a) to Fig. 27(c), respectively.

[0239] Regardless of the subjects tracked by main camera 500 and sub camera 400, rectangular frame-shaped indicators having colors based on the relationships shown in FIG. 25 are superimposed on the areas of subjects A to C in the overhead image.

[0240] 27(a) shows a state in which main camera 500 is tracking subject B, and sub camera 400 is tracking subject A. Therefore, characters 272 of "main camera" are superimposed near the area of ​​subject B, which is the tracking subject of main camera 500. Also, characters 271 of "sub camera" are superimposed near the area of ​​subject A, which is the tracking subject of sub camera 400.

[0241] 27(b) shows a state in which main camera 500 is tracking subject A, and sub camera 400 is tracking subject B. Therefore, characters 272 of "main camera" are superimposed near the area of ​​subject A, which is the tracking subject of main camera 500. Also, characters 271 of "sub camera" are superimposed near the area of ​​subject B, which is the tracking subject of sub camera 400.

[0242] 27(c) shows a state in which the main camera 500 is tracking subject C, and the sub camera 400 is tracking subject A. Therefore, the characters 272 of "main camera" are superimposed near the area of ​​subject C, which is the tracking subject of the main camera 500. Also, the characters 271 of "sub camera" are superimposed near the area of ​​subject A, which is the tracking subject of the sub camera 400.

[0243] Next, a process of adding an index indicating the shooting angle of view of the tracked subject to the overhead image will be described with reference to the flowchart shown in FIG.

[0244] In S801, CPU 101 as angle of view information superimposition unit 127 reads data SUBJECT_IMG of the overhead image on which the subject area index and the tracking subject index for each camera generated in S708 are superimposed from RAM 102. Note that if the subject area index and the tracking subject index for each camera are not superimposed, CPU 101 reads image data IMG of the same frame as that read in S703 from RAM 102.

[0245] In S802, CPU 101 obtains the current zoom values ​​of main camera 500 and sub camera 400. For the zoom value MAIN_ZOOM of main camera 500, the value most recently obtained from main camera 500 by zoom value calculation unit 125 and stored in RAM 102 can be used. For the zoom value of sub camera 400, the zoom value Z_VALUE most recently determined by zoom value calculation unit 125 in S207 and stored in RAM 102 can be used. Note that CPU 101 may obtain the zoom values ​​from main camera 500 and sub camera 400 by transmitting a zoom value obtainment command via network I / F 105.

[0246] In S803, the CPU 101 reads image data used as an index representing the shooting angle of view, for example, from the ROM 103. Here, a specific image is scaled to a size according to the angle of view and used as an index representing the shooting angle of view. Note that the index representing the shooting angle of view may be in other forms, such as an index representing the zoom value by characters.

[0247] In S804, CPU 101 resizes the image acquired in S803 according to the zoom value of main camera 500 acquired in S802, and generates an index indicating the shooting angle of view of the tracking subject of main camera 500. CPU 101 similarly generates an index indicating the shooting angle of view of the tracking subject for sub camera 400.

[0248] The correspondence between the resize rate and the zoom value is stored in advance in ROM 103 so that the smaller the zoom value (narrower the angle of view), the larger the image is resized, and the larger the zoom value (wider the angle of view), the smaller the image is resized. In this embodiment, a position DRAW_POSI on the image for superimposing an index indicating the shooting angle of view of the tracking subject on the overhead image is stored in ROM 103 in advance. The position DRAW_POSI specifies the coordinates of the diagonal vertices of a rectangular area for displaying the index for each of the main camera 500 and the sub-camera 400. Note that individual positions DRAW_POSI may be stored according to the number of sub-cameras 400.

[0249] Here, as shown in Fig. 28(d) and Fig. 28(e), an index indicating the shooting angle of view is displayed at the bottom of the overhead image. In this case, the position DRAW_POSI includes the coordinates (drawM_Xs, drawM_Ys) of the upper left vertex and the coordinates (drawM_Xe, drawM_Ye) of the lower right vertex of a rectangular area displaying the index of the shooting angle of view of the main camera 500. The position DRAW_POSI also includes the coordinates (drawS_Xs, drawS_Ys) of the upper left vertex and the coordinates (drawS_Xe, drawS_Ye) of the lower right vertex of a rectangular area displaying the index of the shooting angle of view of the sub camera 400.

[0250] The CPU 101 superimposes the scaled image within a rectangular area of ​​the overhead image indicated by the position DRAW_POSI.

[0251] The operation of generating and superimposing an index indicating the shooting angle of view will be described in more detail with reference to Fig. 28. Fig. 28(a) shows a state where, among human subjects A to C present within the shooting range, the main camera 500 is tracking subject B, and the sub camera 400 is tracking subject A. Specifically, this is the case where the sub camera 400 is automatically shooting according to the role "assist follow". Fig. 28(b) shows the image of the main camera 500, and Fig. 28(c) shows the image of the sub camera 400.

[0252] Fig. 28(d) shows an example of the display of an index indicating the shooting angle of view when the shooting angles of view of the main camera 500 and the sub camera 400 are both wide-angle in the state of Fig. 28(a). Also, Fig. 28(e) shows an example of the display of an index indicating the shooting angle of view when the shooting angles of view of the main camera 500 and the sub camera 400 are both telephoto in the state of Fig. 28(a).

[0253] Here, an image (human body model image) of a simple shape that imitates the torso and head of a human body is used to generate the index. Also, for ease of understanding and explanation, it is assumed that the zoom value ranges of the main camera 500 and the sub camera 400 are the same, and the zoom value of the sub camera 400 is controlled to be equal to the zoom value of the main camera 500 by the role "assist follow". As shown in Fig. 28(d) and Fig. 28(e), when the shooting angle of view is wide angle, a small index is displayed, and when the shooting angle of view is telephoto, a large index is displayed.

[0254] In S805, CPU 101 executes a process for clarifying which camera the shooting angle of view index is for. Specifically, CPU 101 sets the outline of the rectangular area displaying the index to a color according to the relationship between the identification ID and the index color shown in Fig. 25. Specifically, CPU 101 presents at least a part of the outline of the rectangular area displaying the index in a color corresponding to the identification ID of the subject indicated by the index. Alternatively, CPU 101 superimposes an image in a color corresponding to the identification ID of the subject indicated by the index on at least a part of the outline of the rectangular area displaying the index.

[0255] In the example shown in FIG. 28(a), the subject being tracked by main camera 500 is subject B. Also, as shown in FIG. 25, the color of the indicator for subject B (identification ID2) is green. Therefore, CPU 101 sets the outline of a rectangular area displaying an indicator representing the shooting angle of view of subject B to green. Similarly, CPU 101 sets the outline of a rectangular area displaying an indicator representing the shooting angle of view of subject A (identification ID1) to red.

[0256] The area displaying the indicator representing the shooting angle of view may also include identification information of the camera that tracks the subject corresponding to the indicator. By including the camera identification information, it is possible to more directly grasp which camera's shooting angle of view it is.

[0257] CPU101 generates image data including an index indicating the shooting angle of view and identification information of the subject corresponding to the index as display image data OUT_IMAGE, as shown in FIG. 28(d) and FIG. 28(e). CPU101 then transmits the display image data OUT_IMAGE to the display device 800. The display image data OUT_IMAGE includes at least one of (1) the index of the subject area shown in FIG. 27 and information indicating the tracking subject for each camera, and (2) an index indicating the shooting angle of view and identification information of the subject corresponding to the index. CPU101 may display the display image data OUT_IMAGE on the display unit 108.

[0258] When the CPU 801 of the display device 800 receives the display image data OUT_IMAGE via the network I / F 805, it stores it in the RAM 802. Then, the CPU 801 displays the display image data OUT_IMAGE on the display unit 811. The CPU 801 may convert the display image data OUT_IMAGE into data suitable for display on the display unit 811 as necessary. For example, the CPU 801 can scale the display image data OUT_IMAGE to match the resolution of the display unit 811.

[0259] An index representing the angle of view by size is generated by scaling an image in which the torso and head are deformed according to the zoom value. However, an index representing the angle of view by something other than size may be generated. For example, the size of the range used as the index in a predetermined image (for example, an image in which the entire body of a human body is deformed) may be varied according to the zoom value. Specifically, the narrower the angle of view, the narrower the range used as the index, such as using the entire image (the entire body) as the index when the shooting angle of view is wide, and using only the upper body of the image as the index when the shooting angle of view is telephoto.

[0260] Also, an image that imitates a subject does not have to be used as an index. For example, instead of an image with a deformed shape of the torso and head, an image of a fruit (e.g., an apple) or a musical instrument (e.g., a guitar) may be scaled and used as an index of the shooting angle of view. Alternatively, an image of a subject cut out from an overhead image may be used. Instead of scaling, one of a plurality of images of different sizes prepared in advance may be selected based on the zoom value. An index of different sizes according to the zoom value can be generated using any known method.

[0261] When there are multiple sub cameras 400, an indicator may be displayed for each sub camera 400. The indicators indicating the imaging angle of view may be displayed in the order of main camera / sub camera 1 / sub camera 2... from the leftmost rectangular area. Alternatively, an indicator indicating the imaging angle of view of the main camera may be displayed in a rectangular area located at the center in the horizontal direction, and indicators of the imaging angle of view of each sub camera may be displayed in a predetermined order in the rectangular areas on the left and right.

[0262] In this embodiment, each camera tracks one subject. However, the tracking subject may be multiple people. An index indicating the shooting angle of view of a camera tracking multiple subjects can be generated using an image according to the number of subjects. For example, for a camera tracking three subjects (subjects A to C) shown in FIG. 26, an image of the three subjects lined up can be scaled to generate an index indicating the shooting angle of view. The same applies to other numbers of subjects.

[0263] Alternatively, when multiple subjects are divided into groups and tracked on a group basis, an image of one person may be scaled and then indexes may be generated by superimposing group identification information (e.g., group name). In this case, a different index color may be assigned to the group of subjects than to each individual subject. Alternatively, subjects belonging to the same group may be represented using an index including all the colors assigned to each individual subject. For example, in the example shown in FIG. 25, if subjects A to C belong to the same group, an index (e.g., a rectangular frame) consisting of three colors, red, green, and blue, may be superimposed on each of the areas of subjects A to C.

[0264] In addition, in this embodiment, a rectangular frame of a color associated with the subject is used to visually associate the tracking subject and the shooting angle of view for the same camera, but other methods may be used. For example, an image such as a solid line, a dotted line, a dashed line, or a double line that connects the area of ​​the tracking subject for the same camera and the display area of ​​the indicator representing the shooting angle of view may be used. In addition to the rectangular frame, an arrow indicating the subject area, or a figure frame such as a circle or a trapezoid that surrounds the subject area may be used.

[0265] Also, an icon of the corresponding camera may be displayed in association with an index indicating the shooting angle of view. For example, when the external shapes of the cameras are different, it is possible to more clearly indicate which camera's shooting angle of view. For example, when the main camera 500 is a horizontally long video camera and the sub-camera 400 is a round PTZ camera, by displaying an icon imitating the camera's external appearance in association with an index indicating the shooting angle of view, it becomes possible to recognize at a glance which camera's shooting angle of view. Also, the type of camera can be ascertained from the icon.

[0266] The shooting angle of view may also be presented by text information. For example, a number indicating a zoom value or an image showing text such as "pull back" (wide angle) / "close up" (telephoto) may be used as an indicator of the shooting angle of view. Any other expression capable of indicating the shooting angle of view may also be used.

[0267] In addition, in the present embodiment, an index that visualizes the tracking subject and the shooting angle of view for each camera is exemplified as the shooting information. However, the shooting information is not limited to this. For example, the tally state of each camera may be visualized. Specifically, for the indicator of the tracking subject and the shooting angle of view of a camera whose tally state is broadcasting or streaming, a mark of the same color as the color of the tally lamp or text information such as "broadcasting" or "streaming" may be associated and displayed. When selecting a video to be output from the imaging system using a switcher, by displaying the tally state superimposed on the overhead image, it becomes possible to easily grasp the subject and the angle of view of the video currently being output, in addition to the subject and the angle of view of each camera.

[0268] As described above, according to this embodiment, in an imaging system having a sub-camera that automatically captures images in cooperation with a main camera, it is possible to grasp in real time imaging information such as the subject being photographed by each camera and the imaging angle of view. Therefore, it is possible to easily execute remote operations such as switching the main camera and appropriately changing the angle of view of each camera.

[0269] <Fourth embodiment> Next, a fourth embodiment of the present invention will be described. In the third embodiment, an example in which the shooting information of each camera is superimposed on an overhead image and presented to a user has been described. In the fourth embodiment, the shooting information of each camera is presented to a user in a method different from the method of superimposing the shooting information on an overhead image.

[0270] 29 is a diagram showing a series of processes performed by the shooting control device 100 when controlling the operation of the sub-camera 400, focusing on main operations and signal flows, similar to Fig. 3 and Fig. 23. The functional blocks shown in the shooting control device 100 show the main operations in a schematic manner, and correspond to the main functions provided by the shooting control application. Each functional block in Fig. 29 is realized by a combination of the CPU 101 that executes the shooting control application and one or more of the functional blocks of the shooting control device 100 shown in Fig. 22.

[0271] As can be seen from a comparison between FIG. 29 and FIG. 23, the shooting control device 100 of this embodiment has the function of a GUI generation unit 129 in addition to the configuration in the third embodiment. The GUI generation unit 129 generates a GUI (Graphical User Interface) as a second display image for presenting the shooting information of each camera to the user. By generating a GUI for presenting the shooting information to the user, it becomes possible to use an expression different from the first display image in which the shooting information is superimposed on the overhead image. Therefore, it becomes possible to provide another means for the user to grasp the shooting information of each camera.

[0272] 30, an operation of generating a GUI image for presenting shooting information of the main camera 500 and the sub camera 400 will be described among the operations performed by the shooting control device 100 in this embodiment. The shooting control device 100 can execute the operation described here in parallel with the automatic shooting control operation of the sub camera and the operation described in the third embodiment.

[0273] As an example, a GUI image is generated that presents the tracking subject and shooting angle of each camera as well as the linkage details of the sub-cameras according to the set roles as shooting information. Any type of GUI may be generated, but as an example, a window that presents shooting information is generated here.

[0274] In S901, CPU 101 generates data of an overhead image in which an image of a rectangular frame in a color corresponding to the identification ID of the subject is superimposed around the subject region, in the same manner as in S707 of Fig. 24. CPU 101 stores the generated image data in RAM 102.

[0275] In S902, CPU 101 adds, to the image data generated in S901, an image of the identification information of the camera to be superimposed near the subject area corresponding to the tracking subject of main camera 500 and sub camera 400, in the same manner as in S708. Here, images of the characters "main camera" and "sub camera" are added as the image of the identification information of the camera. CPU 101 stores in RAM 102 the data of the overhead image (first display image) on which the shooting information is superimposed, obtained by the steps of S901 and S902.

[0276] In S903, the CPU 101 acquires the role CAMERA_ROLE set for the sub-camera 400 in the same manner as in S705.

[0277] In S904, the CPU 101 acquires the zoom value of each camera in the same manner as in S802.

[0278] In S905, CPU 101 as GUI generating unit 129 starts generating a GUI image for presenting shooting information. First, CPU 101 draws an area for each camera in a window image of a predetermined size. Here, as an example, a rectangle corresponding to each camera is drawn in a color associated with the identification ID of the tracking subject of each camera. FIG. 31(e) shows an example of window 310 and rectangular frames 311 and 312 corresponding to each camera. In FIG. 31(e), as shown in FIG. 31(a), both of the tracking subjects of main camera 500 and sub camera 400 are subject B. Therefore, CPU 101 draws rectangular frame 311 corresponding to main camera 500 and rectangular frame 312 corresponding to sub camera 400 in green.

[0279] Furthermore, in order to indicate which camera the area corresponds to, CPU 101 draws camera identification information 313, 314 on the upper part of rectangular frames 311, 312. As an example, text information such as "main camera" and "sub camera" is drawn in Fig. 31(e), but other types of images such as icons of the external shapes of the cameras may be drawn as described above.

[0280] Furthermore, CPU 101 renders, in the area corresponding to sub camera 400, information indicating the control content of the tracking subject, among the control content according to the role CAMERA_ROLE set for sub camera 400.

[0281] In the example shown in Fig. 31(e), the area (area within a rectangle) corresponding to each camera is divided into upper and lower parts. The control content of the tracking subject is drawn in the upper part, and the information on the shooting angle of view is drawn in the lower part. Note that how to divide the area and what shooting information is presented in the divided areas can be appropriately determined according to the type and number of shooting information to be presented.

[0282] In the example shown in Fig. 31(e), the role "main follow" is set for the sub camera. Therefore, CPU 101 draws information 317 indicating that the tracking subject of sub camera 400 is the same as the tracking subject of the main camera, at the top of the area corresponding to sub camera 400. In Fig. 31(e), as an example, text information "Tracking subject same as main" is drawn.

[0283] In S906, the CPU 101 generates an index indicating the shooting angle of view of each camera in the same manner as in S804. Here, the index is generated using the same image as in the third embodiment, but as described above, the index may be generated using other types of images or text information.

[0284] In S907, CPU 101 draws the indices generated in S906 in the areas corresponding to the respective cameras. In the example shown in Fig. 31(e), the indices are drawn in the lower part of the areas. Then, CPU 101 stores data of the GUI image (second display image) obtained through the steps of S905 to S907 in RAM 102.

[0285] The CPU 101 transmits the data of the overhead image (first display image) stored in the RAM 102 in S902 and the data of the GUI image (second display image) stored in the RAM 102 in S907 as display image data OUT_IMAGE to the display device 800. In the present embodiment as well, the display image data OUT_IMAGE can be displayed on the display unit 108.

[0286] When the CPU 801 of the display device 800 receives the display image data OUT_IMAGE via the network I / F 805, it stores it in the RAM 802. Then, the CPU 801 displays the display image data OUT_IMAGE on the display unit 811. The CPU 801 may convert the display image data OUT_IMAGE into data suitable for display on the display unit 811 as necessary. For example, the CPU 801 can scale the overhead image and the GUI image so that the overhead image and the GUI image are displayed as separate windows on the display unit 811.

[0287] An example of the display image data OUT_IMAGE generated by the CPU 101 in this embodiment will be described with reference to FIG.

[0288] 31(a) and 31(b) show a state where the same scene as in FIG. 28(a) is automatically captured by the main camera 500 and the sub camera 400 linked to the main camera 500. In FIG. 31(a), the sub camera 400 is automatically capturing images according to the role of "main follow". In FIG. 31(b), the sub camera 400 is automatically capturing images according to the role of "assist counter". In this case, in the role of "assist counter", a subject on the right side of the main camera 500, which is a different subject from the subject of interest, is set as the tracking subject of the sub camera 400.

[0289] Also, in Fig. 31(a), the shooting angle of view of the main camera 500 and the sub camera 400 are the same. On the other hand, in Fig. 31(b), the angle of view of the main camera 500 is narrower (telephoto side) than in the state of Fig. 31(a). Therefore, the shooting angle of view of the sub camera 400 is controlled to be wider (wide-angle side) than in the state of Fig. 31(a) according to its role as an "assist counter."

[0290] In Fig. 31(a), both the main camera 500 and the sub-camera 400 are tracking subject B. In Fig. 31(b), the main camera 500 is tracking subject A, and the sub-camera 400 is tracking subject C.

[0291] Fig. 31(c) shows an image corresponding to the overhead image data generated in S902 in the shooting state of Fig. 31(a). A rectangular frame of a color associated with the identification ID of the subject is superimposed on the area of ​​each subject. In addition, an image showing the identification information of main camera 500 and sub camera 400 (the characters "Main Camera" and "Sub Camera") is superimposed on the upper part of the rectangular frame of subject B, which is the tracking subject of main camera 500 and sub camera 400.

[0292] Fig. 31(d) shows an image corresponding to the overhead image data generated in S902 in the shooting state of Fig. 31(b). A rectangular frame of a color associated with the identification ID of the subject is superimposed on the area of ​​each subject. Also, an image indicating the identification information of main camera 500 (the characters "Main Camera") is superimposed on the upper part of the rectangular frame of subject B, which is the tracking subject of main camera 500. Furthermore, an image indicating the identification information of sub camera 400 (the characters "Sub Camera") is superimposed on the upper part of the rectangular frame of subject C, which is the tracking subject of sub camera 400.

[0293] FIG. 31(e) shows an image corresponding to the GUI image data generated in S907 in the shooting state of FIG. 31(a). In the window 310, a rectangular frame 311 showing an area corresponding to the main camera 500 and a rectangular frame 312 showing an area corresponding to the sub camera 400 are drawn. The rectangular frames 311 and 312 are drawn in a color associated with the identification ID of the tracking subject of the corresponding camera. In addition, in the upper part of the rectangular frames 311 and 312, identification information 313 and 314 of the corresponding camera are drawn. In the upper part of the rectangular frame 312 corresponding to the sub camera 400, information 317 showing the content of the tracking subject control in the role currently set for the sub camera 400 is drawn. In addition, in the lower part of the rectangular frames 311 and 312, indicators 315 and 316 showing the shooting angle of view of the main camera 500 and the sub camera 400 are drawn. The sizes of the indicators 315 and 316 are equal.

[0294] Fig. 31(f) shows an image corresponding to the GUI image data generated in S907 in the shooting state of Fig. 31(b). Compared with Fig. 31(e), the colors of rectangular frames 311 and 312 are different because the tracking subject is different. Also, the shooting angle of view of main camera 500 is narrow and the shooting angle of view of sub-camera 400 is wide, so the sizes of indicators 315 and 316 are different. Also, the content of information 317 indicating the content of tracking subject control is different because the role of sub-camera 400 is different.

[0295] The color associated with the ID of the subject tracked by the camera may be applied to something other than the rectangle showing the area corresponding to the camera. For example, it may be applied to the background within the area. In addition, the two images may be displayed in the same window by superimposing a GUI image on an overhead image on which shooting information is superimposed.

[0296] In this embodiment, the control content of the tracking subject is shown as information related to the role of the sub camera 400. However, the name of the role of the sub camera 400 (such as "assist counter") or the identification information of the tracking subject (such as the subject name) may be shown instead or in addition.

[0297] According to this embodiment, by generating an image that presents shooting information separately from the overhead image, in addition to the effects of the third embodiment, it becomes possible to check the shooting information without being influenced by the overhead image, making it even easier to understand the shooting information.

[0298] In the third and fourth embodiments, examples have been described in which information visualizing the shooting status of the main camera and the sub camera is provided to the user, but when the photographer operates the main camera, the subject and shooting angle of view of the main camera may be changed in advance. In such a case, since the subject and shooting angle of view of the main camera can be grasped to some extent, information visualizing the shooting status of the sub camera without visualizing the shooting status of the main camera may be provided to the user. Alternatively, the user may be allowed to select whether or not to visualize the shooting status of the main camera.

[0299] In addition, in a multi-camera imaging system with a large number of sub-cameras, if information that visualizes the shooting status of all the cameras is provided to the user, the amount of information may be too large and the user may not be able to accurately recognize the information. Therefore, it may be possible to allow the user to select the camera of the multi-camera imaging system for which the shooting status is to be visualized. However, since the effect of improving the usability of the automatic imaging system is small even if information that visualizes the shooting status of the main camera without visualizing the shooting status of the sub-cameras is provided to the user, it is preferable to provide the user with information that visualizes the shooting status of at least one sub-camera. These selection operations may be performed using user input 804.

[0300] (Other embodiments) The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0301] The disclosure of the present embodiment includes the following imaging control device, imaging control method, display control device, display control method, imaging system, and program. (Item 1) A control means for controlling a shooting direction and a field angle of the sub-camera among a plurality of cameras including a main camera and a sub-camera, based on a role set for the sub-camera and a target object and a field angle of the main camera; a generating means for generating a display image presenting information about the main camera and the sub camera; an output means for outputting the display image; having The image capturing control device is characterized in that the generation means generates a display image that presents at least one of information regarding the subject of interest of the main camera and the subject being tracked by the sub camera, and information regarding the angles of view of the main camera and the sub camera. (Item 2) The image capturing control device described in item 1 is characterized in that the generation means generates the display image by superimposing information from the main camera and the sub-camera on an image captured by a camera other than the multiple cameras that captures the entire shooting range of the multiple cameras. (Item 3) The image capture control device described in item 2 is characterized in that the generation means presents information about the target subject and the tracking subject by superimposing indices associated with the area of ​​the target subject and the area of ​​the tracking subject in the video captured by the other camera. (Item 4) 4. The photographing control device according to item 3, wherein the indices include a rectangular frame-shaped indice including an area of ​​the target subject, and a rectangular frame-shaped indice including an area of ​​the tracking subject. (Item 5) 5. The photographing control device according to item 4, wherein the color of the rectangular frame-shaped indicator is a color that is predetermined for each subject. (Item 6) 6. The shooting control device according to item 4 or 5, wherein the indicator further includes identification information of the main camera and the sub camera. (Item 7) The shooting control device described in any one of items 2 to 6, characterized in that the generation means presents information regarding the angles of view of the main camera and the sub camera by superimposing an indicator indicating the angle of view of the main camera and an indicator indicating the angle of view of the sub camera on the image shot by the other camera. (Item 8) 8. The imaging control device according to item 7, wherein the index indicating the angle of view indicates the size of the angle of view by a size. (Item 9) 8. The imaging control device according to item 7, wherein the index indicating the angle of view indicates the size of the angle of view by the size of a range in a predetermined image used as the index. (Item 10) The shooting control device according to any one of items 7 to 9, characterized in that at least a part of the contour of an area on which an index indicating the angle of view of the main camera is superimposed is presented in a predetermined color with respect to the subject of interest of the main camera. (Item 11) The image capturing control device according to item 1, characterized in that the generation means generates, as the display images, a first display image based on an image captured by a camera other than the multiple cameras, which captures the entire shooting range of the multiple cameras, and a second display image not based on an image captured by the other camera. (Item 12) indices associated with the area of ​​the target object and the area of ​​the tracking object are superimposed on the first display image; The second display image includes an indicator indicating an angle of view of the main camera and an indicator indicating an angle of view of the sub camera. 12. The imaging control device according to item 11, (Item 13) 13. The photographing control device according to item 11 or 12, characterized in that the second display image further includes information regarding a role set for the sub-camera. (Item 14) Item 14. The shooting control device according to item 13, characterized in that the information about the role includes one or more of the name of the role and the control content of the sub-camera according to the role. (Item 15) The shooting control device described in any one of items 12 to 14, characterized in that the second display image includes an area for presenting information about the main camera, which is shown with a rectangular frame of a predetermined color for the target subject of the main camera, and an area for presenting information about the sub-camera, which is shown with a rectangular frame of a predetermined color for the tracking subject of the sub-camera. (Item 16) The plurality of cameras; An imaging control device according to any one of items 1 to 15, An imaging system having the above configuration. (Item 17) 17. The imaging system according to item 16, further comprising a display device that displays the display image output by the imaging control device. (Item 18) A display control device used in an imaging system in which a shooting direction and a field of view of a sub-camera among a plurality of cameras including a main camera and a sub-camera are controlled based on a role set for the sub-camera and a target subject and a field of view of the main camera, a generating means for generating a display image presenting information about the sub-camera; an output means for outputting the display image; having The display control device according to claim 1, wherein the generating means generates a display image that presents at least one of information about a subject being tracked by the sub-camera and information about an angle of view of the sub-camera. (Item 19) A photographing control method executed by a photographing control device, Among a plurality of cameras including a main camera and a sub camera, a shooting direction and a field angle of the sub camera are controlled based on a role set for the sub camera and a target object and a field angle of the main camera; generating a display image presenting information about the main camera and the sub camera; and outputting the display image. The photographing control method, characterized in that the generating step includes generating a display image that presents at least one of information regarding the subject of interest of the main camera and the subject being tracked by the sub camera, and information regarding the angles of view of the main camera and the sub camera. (Item 20) A display control method executed by a display control device used in an imaging system in which a shooting direction and a field of view of a sub-camera among a plurality of cameras including a main camera and a sub-camera are controlled based on a role set for the sub-camera and a target subject and a field of view of the main camera, generating a display image presenting information about the sub-camera; and outputting the display image. A display control method, characterized in that the generating includes generating a display image that presents at least one of information regarding a subject tracked by the sub-camera and information regarding the angle of view of the sub-camera. (Item 21) A program for causing a computer to function as each of the means possessed by the imaging control device according to any one of items 1 to 15. (Item 22) A program for causing a computer to function as each of the means possessed by the display control device described in item 18.

[0302] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0303] 100: Shooting control device, 300: Bird's-eye view camera, 400: Sub-camera, 500: Main camera, 600: Role control device, 101: CPU, 102: RAM, 103: ROM, 104: Inference unit, 105: Network I / F, 106: User input unit

Claims

1. A control means for controlling a shooting direction and a field angle of the sub-camera among a plurality of cameras including a main camera and a sub-camera, based on a role set for the sub-camera and a target object and a field angle of the main camera; a generating means for generating a display image presenting information about the main camera and the sub camera; an output means for outputting the display image; having The image capturing control device is characterized in that the generation means generates a display image that presents at least one of information regarding the subject of interest of the main camera and the subject being tracked by the sub camera, and information regarding the angles of view of the main camera and the sub camera.

2. The shooting control device according to claim 1, characterized in that the generation means generates the display image by superimposing information from the main camera and the sub-camera on an image captured by a camera other than the multiple cameras, which captures the entire shooting range of the multiple cameras.

3. The photographing control device according to claim 2, characterized in that the generation means presents information regarding the subject of interest and the subject to be tracked by superimposing indices associated with the area of ​​the subject of interest and the area of ​​the subject to be tracked in the image photographed by the other camera.

4. 4. The photographing control device according to claim 3, wherein the indices include a rectangular frame-shaped indice including an area of ​​the target subject, and a rectangular frame-shaped indice including an area of ​​the tracking subject.

5. 5. The photographing control device according to claim 4, wherein the color of the rectangular frame-shaped indicator is a color that is determined in advance for each subject.

6. The imaging control device according to claim 4 , wherein the indicator further includes identification information of the main camera and the sub camera.

7. The shooting control device according to claim 2, characterized in that the generation means presents information regarding the angles of view of the main camera and the sub camera by superimposing an indicator indicating the angle of view of the main camera and an indicator indicating the angle of view of the sub camera on the image shot by the other camera.

8. 8. The photographing control device according to claim 7, wherein the index indicating the angle of view indicates the size of the angle of view by a size.

9. 8. The imaging control device according to claim 7, wherein the index indicating the angle of view indicates the size of the angle of view by the size of a range used as the index in a predetermined image.

10. 8. The photographing control device according to claim 7, wherein at least a part of the outline of an area on which an index indicating the angle of view of the main camera is superimposed is presented in a predetermined color with respect to the subject of interest of the main camera.

11. The image capturing control device according to claim 1, characterized in that the generation means generates, as the display images, a first display image based on an image captured by a camera other than the multiple cameras, which captures the entire shooting range of the multiple cameras, and a second display image not based on an image captured by the other camera.

12. indices associated with the area of ​​the target subject and the area of ​​the tracking subject are superimposed on the first display image; the second display image includes an indicator indicating an angle of view of the main camera and an indicator indicating an angle of view of the sub camera; The imaging control device according to claim 11 .

13. The photographing control device according to claim 11 , wherein the second display image further includes information regarding a role set for the sub-camera.

14. 14. The imaging control device according to claim 13, wherein the information about the role includes at least one of a name of the role and a control content of the sub-camera according to the role.

15. The photographing control device according to claim 12, characterized in that the second display image includes an area for presenting information about the main camera, which is shown with a rectangular frame of a predetermined color for the subject of interest of the main camera, and an area for presenting information about the sub-camera, which is shown with a rectangular frame of a predetermined color for the subject to be tracked of the sub-camera.

16. The plurality of cameras; The imaging control device according to any one of claims 1 to 15, An imaging system having the above configuration.

17. 17. The imaging system according to claim 16, further comprising a display device that displays the display image output by the imaging control device.

18. A display control device used in an imaging system in which a shooting direction and a field of view of a sub-camera among a plurality of cameras including a main camera and a sub-camera are controlled based on a role set for the sub-camera and a target subject and a field of view of the main camera, a generating means for generating a display image presenting information about the sub-camera; an output means for outputting the display image; having The display control device according to claim 1, wherein the generating means generates a display image that presents at least one of information about a subject being tracked by the sub-camera and information about an angle of view of the sub-camera.

19. A photographing control method executed by a photographing control device, Among a plurality of cameras including a main camera and a sub camera, a shooting direction and a field angle of the sub camera are controlled based on a role set for the sub camera and a target object and a field angle of the main camera; generating a display image presenting information about the main camera and the sub camera; and outputting the display image. The photographing control method, characterized in that the generating step includes generating a display image that presents at least one of information regarding the subject of interest of the main camera and the subject being tracked by the sub camera, and information regarding the angles of view of the main camera and the sub camera.

20. A display control method executed by a display control device used in an imaging system in which a shooting direction and a field of view of a sub-camera among a plurality of cameras including a main camera and a sub-camera are controlled based on a role set for the sub-camera and a target subject and a field of view of the main camera, generating a display image presenting information about the sub-camera; and outputting the display image. A display control method, characterized in that the generating includes generating a display image that presents at least one of information regarding a subject tracked by the sub-camera and information regarding the angle of view of the sub-camera.

21. A program for causing a computer to function as each of the means included in the imaging control device according to any one of claims 1 to 15.

22. A program for causing a computer to function as each of the means included in the display control device according to claim 18.

Images