Display control device, display control method, image pickup system, computer-readable medium, and computer program product

Abstract

The invention provides a display control apparatus, a display control method, an imaging system, a computer readable medium, and a computer program product. Disclosed is a display control apparatus for use in an imaging system including a first imaging apparatus and a second imaging apparatus. In the image pickup system, a subject, which is a tracking target of the second image pickup apparatus, and a viewing angle of the second image pickup apparatus are controlled based on information related to the first image pickup apparatus. The display control device: generates a display image different from an image captured by the first imaging device and an image captured by the second imaging device; and outputting the display image. The display control device generates, as the display image, an image indicating a subject, which is a tracking target of the second imaging device, and an angle of view of the second imaging device.

Description

Technical Field

[0001] The present invention relates to a display control device, a display control method, a camera system, a computer-readable medium and a computer program product, and in particular to a technology for controlling multiple camera devices. Background Art

[0002] Japanese Patent Laid-Open No. 2020-25248 describes a camera system in which a plurality of cameras are classified into a main camera and a sub-camera, and the sub-camera is controlled to shoot the same subject as the main camera. In addition, the camera system described in Japanese Patent Laid-Open No. 2020-25248 allows the shooting direction and viewing angle of each camera to be checked on a remote control screen.

[0003] According to the imaging system described in Japanese Patent Application Laid-Open No. 2020-25248, imaging by the sub-camera can be automatically controlled, thereby saving manpower. However, in order to check the subject photographed by each camera, it is necessary to operate the remote control screen to display the video currently photographed by each camera. Summary of the invention

[0004] In view of this, an aspect of the present invention provides a display control apparatus and a display control method capable of presenting information for improving the convenience of an automatic imaging system using a plurality of cameras.

[0005] According to one aspect of the present invention, a display control device used in a camera system is provided, the camera system comprising a first camera device and a second camera device, wherein in the camera system, a subject serving as a tracking target of the second camera device and a viewing angle of the second camera device are controlled based on information related to the first camera device, and the display control device comprises: a generation unit for generating a display image different from an image captured by the first camera device and an image captured by the second camera device; and an output unit for outputting the display image, wherein the generation unit generates an image indicating the subject serving as a tracking target of the second camera device and the viewing angle of the second camera device as the display image.

[0006] According to another aspect of the present invention, a camera system is provided, which includes: a first camera device; a second camera device; a shooting control device, which controls a subject that is a tracking target of the second camera device and a viewing angle of the second camera device based on information related to the first camera device; and a display control device according to the present invention.

[0007] According to another aspect of the present invention, there is provided a display control method implemented in a camera system, the camera system comprising a first camera device and a second camera device, wherein in the camera system, a subject serving as a tracking target of the second camera device and a viewing angle of the second camera device are controlled based on information related to the first camera device, the display control method comprising: generating a display image that is different from an image captured by the first camera device and an image captured by the second camera device; and outputting the display image, wherein the generating comprises: generating an image indicating the subject serving as a tracking target of the second camera device and the viewing angle of the second camera device as the display image.

[0008] According to another aspect of the present invention, a computer-readable medium is provided, which stores a program having instructions, wherein the instructions, when executed by one or more processors in a camera system, cause the one or more processors to perform the display control method according to the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, which stores a program having instructions, wherein when the instructions are executed by one or more processors in a camera system, the one or more processors perform the display control method according to the present invention.

[0010] Further features of the present invention will become apparent from the following description of embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic diagram of an imaging system according to the first embodiment.

[0012] Figure 2 is a block diagram showing an exemplary functional configuration of each device in the imaging system according to the first embodiment.

[0013] Figure 3 is a diagram showing a photographing control device according to an embodiment, focusing on main operations and signal flows.

[0014] Figure 4 is a diagram showing an example of roles and control contents that can be set for a sub-camera according to an embodiment.

[0015] Figure 5 is a flowchart related to the role determination process according to the first embodiment.

[0016] FIG. 6A to FIG. 6D is a flowchart related to the operation of each device in the camera system according to the embodiment.

[0017] Fig. 7A and Figure 7Bis a diagram showing coordinate transformation according to the embodiment.

[0018] Fig. 8A and Figure 8B is a diagram related to object detection and coordinate transformation according to an embodiment.

[0019] 9A to 9C is a schematic diagram of operation control of the sub-camera according to the first embodiment.

[0020] FIG. 10A to FIG. 10C is a schematic diagram of another operation control of the sub-camera according to the first embodiment.

[0021] Fig.11 is a diagram illustrating calculation of a pan value according to an embodiment.

[0022] Fig.12 is a diagram illustrating calculation of a pitch value according to an embodiment.

[0023] Fig.13 : is a diagram showing an example of mapping of zoom values ​​of the main camera and the sub-camera according to the first embodiment.

[0024] Fig.14 is a flowchart related to processing for determining control content corresponding to the role of the sub-camera according to the first embodiment.

[0025] Fig.15 is a schematic diagram of control corresponding to the role of the sub-camera according to the first embodiment.

[0026] Fig.16 is a schematic diagram of control corresponding to the role of the sub-camera according to the first embodiment.

[0027] Fig.17 is a schematic diagram of an operation of tracking a plurality of subjects according to a modification of the first embodiment.

[0028] Fig.18 is a schematic diagram of an imaging system according to a second embodiment.

[0029] Fig.19 : is a diagram showing an example of settable roles and control contents in the second embodiment.

[0030] Fig. 20A and Fig. 20B : is a schematic diagram showing an example of control related to tracking target subject of the sub-camera according to the second embodiment.

[0031] Fig.21 is a schematic diagram of an imaging system according to a third embodiment.

[0032] Fig. 22is a block diagram showing an exemplary functional configuration of each device in the imaging system according to the third embodiment.

[0033] Fig.23 is a diagram showing a photographing control device according to a third embodiment, focusing on main operations and signal flows.

[0034] Fig.24A and Fig. 24B is a flowchart related to the operation of the photographing control device according to the third embodiment.

[0035] Fig.25 1 is a diagram showing information associated with identification IDs of subjects in the third and fourth embodiments.

[0036] Fig.26 is a diagram showing an example of an overhead image according to the third embodiment.

[0037] FIG. 27A to FIG. 27F : is a diagram showing an example of an overhead image generated in the third embodiment.

[0038] FIG. 28A to FIG. 28E : is a diagram showing an example of an overhead image generated in the third embodiment.

[0039] Fig.29 is a diagram showing a photographing control device according to a fourth embodiment, focusing on main operations and signal flows.

[0040] Fig.30 is a flowchart related to the operation of the photographing control device according to the fourth embodiment.

[0041] FIG. 31A to FIG. 31F is a diagram showing an example of an image generated in the fourth embodiment. DETAILED DESCRIPTION

[0042] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but the invention is not limited to all of these features, and these features can be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar configurations, and their redundant descriptions will be omitted.

[0043] ·<First Embodiment>

[0044] (Overview of Multi-Camera Imaging System)

[0045] Figure 11 is a schematic diagram showing an exemplary configuration of a multi-camera imaging system 10 (hereinafter referred to as an imaging system) according to the present embodiment. The imaging system 10 includes a plurality of cameras 300, 400, and 500, a shooting control device 100, and a character control device 600. The plurality of cameras 300, 400, and 500, the shooting control device 100, and the character control device 600 are connected so that they can communicate via a communication network 700.

[0046] The communication network 700 conforms to known wired or wireless communication standards such as IEEE 802.3 series and IEEE 802.11 series. In addition, each of the plurality of cameras 300 , 400 , and 500 , the photographing control device 100 , and the character control device 600 includes a communication interface conforming to the standard of the communication network 700 .

[0047] Among the plurality of cameras 300, 400, and 500, the camera 300 captures the entire predetermined capture area. The capture area is set to, for example, an area in a studio where there may be a subject to be captured. Therefore, all subjects within the capture area are captured in the video of the camera 300.

[0048] The purpose of the camera 300 is to capture images for detecting the subject to be captured that exists in the shooting area. Therefore, the shooting direction and viewing angle of the camera 300 are determined according to the position of the camera 300 and the shooting area, and are basically fixed during the shooting. In addition, in order to capture a pan-focus image of the shooting area, the focusing distance of the camera 300 can also be basically fixed. In addition, it is expected that the camera 300 captures the entire shooting area, and objects outside the shooting area will not block the shooting area. For this reason, the camera 300 is arranged here at a position where a top view of the entire shooting area is obtained. In the following, the camera 300 will be referred to as a bird's-eye view camera to distinguish it from other cameras 400 and 500 whose shooting directions and viewing angles are basically not fixed during the shooting. However, the arrangement position of the camera 300 is not limited to the position where a bird's-eye view of the shooting area is obtained. The operation of the bird's-eye view camera 300 can be controlled from the shooting control device 100.

[0049] Cameras 400 and 500 are, for example, PTZ cameras, and their operations, including shooting directions (pan angles and pitch angles) and viewing angles (zoom) can be controlled from an external device. It is assumed here that the user of the camera system controls the operation of camera 500, and the shooting control device 100 controls the operation of camera 400. Since the shooting control device 100 controls the operation of camera 400 based on the state of camera 500, camera 500 and camera 400 will be referred to as a main camera and a sub-camera, respectively, hereinafter. Although only one sub-camera 400 is shown for ease of explanation and understanding, multiple sub-cameras may also be provided. Note that the user may directly operate the main camera 500. In addition, cameras 400 and 500 may be configured so that their shooting directions (pan angles and pitch angles) can be controlled by attaching the camera body to a gimbal. In addition, cameras 400 and 500 may be configured so that an interchangeable lens capable of zooming is mounted on the camera body.

[0050] Here, in this embodiment, it is assumed that there is an operator for the character control device 600. In addition, there may be an operator (user) for the shooting control device 100, but the operator is not indispensable. The operator of the character control device 600 may be the user of the shooting control device 100. No photographer is required for the overhead camera 300 and the auxiliary camera 400 because their shooting is controlled by the shooting control device 100. It is assumed that there is an operator or a photographer for the main camera 500. Since some devices do not require the aforementioned configuration of an operator or a photographer, manpower can be saved.

[0051] although Figure 1 The illustration of indicates that all signals are communicated via the communication network 700, but, for example, video signals and control signals may be communicated using different methods. For example, each of the plurality of cameras 300, 400, and 500 may directly supply a video signal to the photographing control device 100 via a cable. The cameras 300, 400, and 500 and the photographing control device 100 include a communication circuit corresponding to a standard of the video signal. The standard of the video signal is, for example, a serial digital interface (SDI) standard, a high-definition multimedia interface Standards, etc., but not limited to these.

[0052] The photographing control device 100 detects a subject from a video signal received from the overhead camera 300. The photographing control device 100 determines a photographing direction and a viewing angle of the sub-camera based on the detection result of the subject, the state of the main camera 500, and the role set for the sub-camera. The photographing control device 100 sends a control command including the determined photographing direction and viewing angle to the sub-camera 400. Changing the role setting can change the method of determining the photographing direction and viewing angle of the sub-camera 400, thereby increasing the degree of freedom of the operation of controlling the sub-camera 400.

[0053] (Exemplary Functional Configuration of Each Device)

[0054] Figure 2 It shows the composition Figure 1 1 is a block diagram of an exemplary functional configuration of each device of the multi-camera imaging system 10 shown in FIG. Note that the constituent elements represented as functional blocks in the figure can be implemented by an integrated circuit (such as an ASIC and an FPGA), a discrete circuit, or a combination of a memory and a processor that executes a program stored in the memory. In addition, one functional block can be implemented by multiple integrated circuit packages, or multiple functional blocks can be implemented by one integrated circuit package. In addition, the same functional block can be implemented as different constituent elements depending on the operating environment, required capabilities, etc.

[0055] (Photography Control Device 100)

[0056] First, an exemplary functional configuration of the photographing control device 100 will be described. For example, the photographing control device 100 may be a general-purpose computer device such as a personal computer and a workstation. The photographing control device 100 is configured such that a CPU 101, a RAM 102, a ROM 103, an inference unit 104, a network interface (I / F) 105, a user input unit 106, and a display unit 108 are connected to each other via an internal bus 110.

[0057] The CPU 101 is a microprocessor capable of executing programmed instructions. For example, the CPU 101 implements the functions of the photographing control device 100 described later by reading a program stored in the ROM 103 into the RAM 102 and executing the program. For example, the CPU 101 can implement the functions of the photographing control device 100 by executing a photographing control application operating on the basic software (OS).

[0058] The RAM 102 is used to load programs executed by the CPU 101 and temporarily store data processed by the CPU 101 , currently processed data, etc. In addition, a part of the RAM 102 can be used as a video memory of the display unit 108 .

[0059] The ROM 103 is a nonvolatile rewritable memory, and stores programs (OS and applications) executed by the CPU 101 , user data, and the like.

[0060] The inference unit 104 performs a process of detecting a subject area using a machine learning model for the video of the overhead camera 300. For example, the inference unit 104 can be implemented using a hardware circuit that can perform calculations of a machine learning model at high speed, such as a graphics processing unit (GPU) and a neural network processing unit (NPU). Alternatively, the inference unit 104 can be implemented using a reconfigurable logic circuit such as a field programmable gate array (FPGA). The CPU 101 can implement the functions of the inference unit 104 by executing a program.

[0061] The machine learning model may be a convolutional neural network (CNN) that has been trained according to the type of subject to be detected. It is assumed here that the inference unit 104 detects a human body region or a face region as a subject region from the input image. In addition, it is assumed that the inference unit 104 outputs the position and size of a rectangular region inscribed in the subject region, as well as the detection reliability for each detected subject region. Note that multiple types of machine learning models may be used to perform processing for detecting different types of subject regions on the same input image. Note that the inference unit 104 may perform processing for detecting subject regions using a known method that does not use a machine learning model. For example, the inference unit 104 may detect subject regions using a method using local feature quantities (such as SIFT and SURF, etc.), a method using pattern matching, and the like.

[0062] The network I / F 105 is an interface for connecting the photographing control device 100 to the communication network 700. The photographing control device 100 (CPU 101) can communicate with external devices (such as the overhead camera 300, the sub-camera 400, the main camera 500, and the character control device 600) in the communication network 700 via the network I / F 105. Note that the photographing control device 100 can communicate with the external devices (such as the overhead camera 300, the sub-camera 400, the main camera 500, and the character control device 600) in the communication network 700 via other communication interfaces (USB, (Bluetooth, etc.) to communicate with external devices.

[0063] In order to communicate with each device (the overhead camera 300, the sub-camera 400, the main camera 500, and the character control device 600) in the communication network 700, the CPU 101 obtains the network address of each device at an arbitrary timing and stores it in the RAM 102. In addition, the CPU 101 also obtains the information of each device (the type of device, model name, etc.) at an arbitrary timing (for example, at the time of the first communication) and stores it in the RAM 102. Assume that, as described above, the CPU 101 knows at least the identification information and the type of device for the overhead camera 300, the sub-camera 400, the main camera 500, and the character control device 600. Note that the user may be allowed to give an arbitrary name to each individual device.

[0064] The user input unit 106 is an input device such as a mouse, a keyboard, and a touch panel, etc. The photographing control apparatus 100 accepts user instructions via the user input unit 106 .

[0065] The display unit 108 is a display device such as a liquid crystal display (LCD), etc. The display unit 108 displays a GUI screen provided by the OS, a shooting control application, or the like.

[0066] (Overhead camera 300)

[0067] Next, an exemplary functional configuration of the overhead camera 300 will be described.

[0068] The CPU 301 is a microprocessor capable of executing programmed instructions. For example, the CPU 301 controls the operation of each functional block and realizes the functions of the overhead camera 300 described later by reading a program stored in the ROM 303 into the RAM 302 and executing the program.

[0069] The RAM 302 is used to load programs executed by the CPU 301 and temporarily store data processed by the CPU 301, currently processed data, etc. In addition, the RAM 302 can be used as a buffer for a video signal obtained by imaging.

[0070] The ROM 308 is a nonvolatile rewritable memory. The ROM 308 stores programs executed by the CPU 301, setting values ​​of the overhead camera 300, user data, and the like. Note that the ROM 308 can also be used as a recording destination of a video signal. The ROM 308 may include a built-in memory and an attachable / removable memory card.

[0071] The image sensor 307 includes an imaging optical system and an image sensor. The image sensor may be, for example, a known CCD or CMOS color image sensor including a color filter based on a primary color Bayer arrangement. The image sensor includes a pixel array of a plurality of pixels in a two-dimensional array, and a peripheral circuit for reading out a signal from each pixel. Each pixel accumulates a charge corresponding to the amount of incident light by photoelectric conversion. A signal having a voltage corresponding to the amount of charge accumulated during the exposure period is read out from each pixel; as a result, a set of pixel signals (analog image signals) representing an image of a subject formed on the imaging surface is obtained.

[0072] The image processing unit 306 applies predetermined signal processing and image processing to the analog image signal output from the image sensor 307 , thereby generating a signal and image data suitable for the intended purpose, and obtains and / or generates various types of information.

[0073] The processing applied by the image processing unit 306 may include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data editing processing, evaluation value calculation processing, special effect processing, and the like.

[0074] Preprocessing may include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, etc.

[0075] The color interpolation process is a process performed when the image sensor 307 includes a color filter, in which the values ​​of color components not included in each pixel data constituting the image data are interpolated. The color interpolation process is also called a demosaic process.

[0076] The correction processing may include processing such as white balance adjustment, chromaticity correction, correction of image degradation caused by optical aberration of the imaging optical system (image restoration), correction of the influence of vignetting of the imaging optical system, and color correction.

[0077] The data editing process may include processing such as cutting (cropping), synthesis, scaling, encoding and decoding of an area, and generation of header information (generation of a data file), etc. The data editing process also includes generation of a video signal to be output to the outside and video data to be recorded in the ROM 308.

[0078] The evaluation value calculation process may include processes such as generation of a signal and an evaluation value used in automatic focus detection (AF) and generation of an evaluation value used in automatic exposure control (AE). The CPU 301 performs AF and AE.

[0079] Special effects processing may include processing such as the addition of blur effects, changes in color tone, relighting, and the like.

[0080] Note that these are examples of processing that the image processing unit 306 can apply, and are not intended to limit the processing applied by the image processing unit 306 .

[0081] The image processing unit 306 outputs the obtained or generated information and data to the CPU 301 , the RAM 302 , and the like according to the intended use.

[0082] Note that the type and settings of processing applied by the image processing unit 306 can be controlled by sending commands from the photography control device 100 to the overhead camera 300 .

[0083] The network I / F 305 is an interface for connecting the overhead camera 300 to the communication network 700. The overhead camera 300 (CPU 301) can communicate with external devices (such as the shooting control device 100, the sub-camera 400, the main camera 500, and the character control device 600) in the communication network 700 via the network I / F 305. Note that the overhead camera 300 can communicate with external devices via other communication interfaces (USB, Bluetooth, etc.) not shown.

[0084] (Sub-camera 400)

[0085] Next, description will be given of an exemplary functional configuration of the sub-camera 400. Functional blocks having the same names of the sub-camera 400 and the overhead camera 300 are assumed to have the same functions, and description thereof is omitted.

[0086] As described above, the sub-camera 400 is a PTZ camera, and its shooting direction and viewing angle can be controlled from the outside. To this end, the sub-camera 400 includes a drive unit 409 capable of panning and tilting operations and zooming operations, and a drive I / F 408. The drive I / F 408 is a communication interface between the drive unit 409 and the CPU 401.

[0087] The driving unit 409 includes a pan / tilt mechanism for supporting the sub-camera 400 so that it can pan and tilt, a zoom mechanism for changing the angle of view of the imaging optical system, a motor for driving these mechanisms, and the like. The zoom mechanism can be used for enlarging and reducing an image by the image processing unit 406. The driving unit 409 drives the motor according to a command received from the CPU 401 via the driving I / F 408, and adjusts the optical axis direction and the angle of view of the imaging optical system.

[0088] (Main camera 500)

[0089] Next, an exemplary functional configuration of the main camera 500 will be described. It is assumed that the functional blocks having the same name of the main camera 500 and the sub-camera 400 have the same functions, and their description is omitted. The main camera 500 is operated by a user. It is assumed here that the user remotely operates the main camera 500 by sending a command via the communication network 700. However, for example, in the case where the main camera 500 is not a PTZ camera, the user can directly operate the main camera 500.

[0090] The photographing control device 100 (CPU 101) can obtain information on the photographing direction and the viewing angle of the sub-camera 400 and the main camera 500 from the sub-camera 400 and the main camera 500 via the network I / F 505. Note that the photographing direction may be the pan angle and the pitch angle of the driving units 409 and 509 based on a predetermined reference direction of 0°. The reference direction may be a direction directly facing the photographing area.

[0091] (Character Control Device 600)

[0092] Next, an exemplary functional configuration of the character control device 600 will be described.

[0093] The CPU 601 is a microprocessor capable of executing programmed instructions. For example, the CPU 601 controls the operation of each functional block and realizes the function of the character control device 600 by reading a character setting program stored in the ROM 603 into the RAM 602 and executing the character setting program.

[0094] The RAM 602 is used to load programs executed by the CPU 601 and temporarily store data processed by the CPU 601 , currently processed data, etc. In addition, a part of the RAM 602 can be used as a video memory for the display unit 608 .

[0095] The ROM 603 is a nonvolatile rewritable memory, and stores programs executed by the CPU 601 , setting values ​​of the character control device 600 , user data, and the like.

[0096] For example, the user input unit 611 is an input device such as a button, a dial, a joystick, and a touch panel, etc. The character control apparatus 600 accepts a user instruction related to character setting for the sub-camera 400 via the user input unit 611 .

[0097] The network I / F 605 is an interface for connecting the character control device 600 to the communication network 700. The character control device 600 (CPU 601) can communicate with external devices (such as the overhead camera 300, the sub-camera 400, the main camera 500, and the photography control device 100) in the communication network 700 via the network I / F 605. Note that the character control device 600 can communicate with external devices via other communication interfaces (USB, Bluetooth, etc.) not shown.

[0098] The display unit 608 is a display device such as a liquid crystal display (LCD), etc. The display unit 608 displays a GUI screen provided by the OS, a character setting application, or the like.

[0099] For example, the character control device 600 stores the character setting information in the ROM 603. The character setting information is information that associates the identification information of the sub-camera 400 and the information indicating the role that has been set. The CPU 601 displays the character setting screen on the display unit 608 by executing the character setting application. For example, the character setting screen displays the identification information (network address, name set by the user, etc.) of the sub-camera 400 and the name of the currently set role in association with each other. The initial value of the currently set role may be a default role that has been set in advance. The user can change the current role displayed in association with the desired sub-camera 400 by operating the user input unit 611.

[0100] Upon detecting a user operation indicating the end of the setting operation (such as an operation of an OK button included in the character setting screen), the CPU 601 updates the character setting information stored in the ROM 103 according to the content of the character setting screen.

[0101] When receiving a character obtaining command via the network I / F 605, the CPU 601 reads out the character setting information stored in the ROM 103, and transmits the character setting information to the transmission source of the character obtaining command.

[0102] Note that although Figure 1 and Figure 2 The character control device 600 is shown as an independent device, but for example, a shooting control application executed on the shooting control device 100 can provide functions similar to those of the character control device 600. In addition, the role can be directly set to the sub-camera 400, and the shooting control device 100 can obtain the role assigned to the sub-camera 400 from the sub-camera 400.

[0103] How to use information obtained from the main camera 500 when controlling the operation of the sub camera 400 is predetermined in relation to the role settable for the sub camera 400. As an example, it is assumed here that information of the main camera is used when controlling tracking target objects and zoom operations of the sub camera 400.

[0104] Figure 4 4 shows examples of the types of roles that can be set for the secondary camera 400 and the control contents associated with the roles. For example, the control contents of each role can be Figure 4 The table format shown is stored in the ROM 603 of the character control device 600 and the ROM 103 of the shooting control device 100. It is assumed here that one of "main follow (MF)", "main counterweight (MC)", "auxiliary follow (AF)" and "auxiliary counterweight (AC)" can be set as a role (ROLE). In the case where there are a plurality of sub-cameras 400, a role can be set for each sub-camera.

[0105] For the sub-camera 400 whose role (ROLE) is "main follow", the shooting control device 100 (CPU 101) sets the same tracking target subject as the main camera 500, and also performs in-phase zoom control on the sub-camera 400 when the zoom operation is performed on the main camera 500. In-phase here means that the zoom direction (telephoto direction or wide-angle direction) is the same, that is, the direction of change of the angle of view is the same. On the other hand, inverse phase means that the zoom direction (telephoto direction or wide-angle direction) is the opposite direction, that is, the direction of change of the angle of view is the opposite direction. Note that even if the zoom direction is in phase, the angle of view may not be the same as that of the main camera 500, and regardless of whether the zoom direction is in phase or inverse phase, the degree of change of the zoom (change speed, change rate, etc.) may not be the same as that of the main camera 500.

[0106] For the sub-camera 400 whose role (ROLE) is "main counterweight", the shooting control device 100 (CPU 101) sets the same tracking target subject as the main camera 500, and also performs reverse zoom control on the sub-camera 400 when the zoom operation is performed on the main camera 500. Therefore, in the case where the zoom-in operation is performed on the main camera 500, the shooting control device 100 (CPU 101) controls the sub-camera 400 of the role to zoom out. Note that zoom-in refers to zooming switched to the telephoto direction (direction toward the telephoto end), and zooming out refers to zooming switched to the wide-angle direction (direction toward the wide-angle end). In the case of zoom control by the image processing unit 406, zoom-in refers to reducing the area to be cut out from the image and increasing the magnification of the cut-out area compared to before the area is changed. On the other hand, zooming out refers to increasing the area to be cut out from the image and reducing the magnification of the cut-out area compared to before the area is changed.

[0107] For the sub-camera 400 whose role (ROLE) is “Sub-Follow”, the photographing control device 100 (CPU 101) sets a tracking target subject different from that of the main camera 500, and also performs in-phase zoom control on the sub-camera 400 when the main camera 500 is zoomed.

[0108] For the sub-camera 400 whose role (ROLE) is "auxiliary counterweight", the imaging control device 100 (CPU 101) sets a different tracking target subject from that of the main camera 500. In addition, when the main camera 500 is zoomed, the imaging control device 100 performs reverse zoom control on the sub-camera 400.

[0109] Here, for the sub-camera 400 whose role (ROLE) is "Sub-Follow" or "Sub-Rival", the subject located on the left side (leftmost) among the subjects other than the subject of interest of the main camera 500 in the image is set as the tracking target subject of the sub-camera 400. Note that the tracking target subject of the sub-camera 400 may be set according to other conditions. For example, the subject located on the right side (rightmost), upper side (top), or lower side (bottom) among the subjects other than the subject of interest of the main camera 500 in the image may be set as the tracking target subject of the sub-camera. Alternatively, the subject closest to the front or rear among the subjects other than the subject of interest of the main camera 500 may be set as the tracking target subject of the sub-camera 400.

[0110] Furthermore, only one of the setting of the tracking target subject and the zoom control may be performed, and the other control items may also be added.

[0111] In the character setting information stored in the ROM 603 by the character control device 600, information indicating the role (ROLE) (for example, the name of the above-mentioned type or a number assigned to the type) is associated with the identification information of the sub-camera 400. The CPU 101 of the shooting control device 100 obtains the character setting information from the character control device 600, and performs control on the operation of the sub-camera 400 according to the type of the role (ROLE) set for the sub-camera 400.

[0112] Note that, in the event that the role setting for the sub-camera 400 has been changed, the role control device 600 may notify the external device (eg, the photographing control device 100) of the change. In this way, the change in the role setting can be immediately reflected in the control of the operation of the sub-camera 400.

[0113] <Description of Operation of Each Device>

[0114] Here, it is assumed that the shooting control device 100 automatically controls the shooting operation of the sub-camera 400 based on the video of the overhead camera 300, the information obtained from the main camera 500, and the role set for the sub-camera 400.

[0115] Figure 3 4 is a diagram showing a processing sequence performed when the photographing control device 100 controls the operation of the sub-camera 400, focusing on the main operations and signal flows. The functional blocks shown within the photographing control device 100 schematically indicate the main operations, and correspond to the main functions provided by the photographing control application. Figure 3 The functional blocks are composed of the CPU 101 executing the shooting control application and Figure 2 The method may be implemented by combining one or more functional blocks of the photographing control device 100 shown in FIG.

[0116] Figure 5 is a flowchart showing the operation of the CPU 101 as the role determination unit 120. In addition, FIG. 6A to FIG. 6D These are flowcharts related to the operations of the photographing control device 100 , the overhead camera 300 , the main camera 500 , and the sub-camera 400 , respectively.

[0117] In the following description, it is assumed that the photographing control device 100 knows the three-dimensional coordinate values ​​of the viewpoint position of the overhead camera 300 and its photographing direction (optical axis direction). In addition, it is assumed that known position information (such as the three-dimensional coordinate values ​​of the viewpoint positions of the sub-camera 400 and the main camera 500, and the coordinate values ​​of the markers arranged in the photographing area) is pre-stored in the ROM 103 as predetermined position information REF_POSI. Note that it is assumed that the coordinate system of the position is predetermined according to the type of the position.

[0118] (Operation of Role Determining Unit 120)

[0119] First, refer to Figure 5 The flowchart shown describes Figure 3 The operations described below are implemented by the CPU 101 executing the shooting control application.

[0120] Note that although the beginning Figure 5 The timing of the operations shown in the flowchart is not particularly limited, but these operations are performed at least before starting the shooting operation of controlling the sub-camera 400. In addition, it is assumed that these operations are also performed when a notification indicating that the character setting of the sub-camera 400 has been changed is received from the character control device 600 via the network I / F 105.

[0121] In step S101, the CPU 101 as the role determination unit 120 obtains the role (ROLE) (role setting information) corresponding to the sub-camera 400 from the character control device 600. The CPU 101 can obtain the above-mentioned role setting information from the character control device 600 by, for example, sending a role obtaining command to the character control device 600 via the network I / F 105. The CPU 101 stores the obtained role setting information in the RAM 102.

[0122] In step S103, the CPU 101 refers to the role setting information stored in the RAM 102 based on the identification information of the sub-camera 400, and obtains the content of the operation control on the sub-camera 400. Then, the CPU 101 as the role determination unit 120 sends the obtained content of the operation control (CAMERA_ROLE) to the tracking target subject determination unit 123. In practice, the CPU 101 stores the content of the operation control into a specific area of ​​the RAM 102, and refers to the content when functioning as the tracking target subject determination unit 123.

[0123] In step S104, the CPU 101 as the role determination unit 120 sends the obtained content of the operation control (CAMERA_ROLE) to the zoom value calculation unit 125. Actually, the CPU 101 stores the content of the operation control into a specific area of ​​the RAM 102 and refers to the content when functioning as the zoom value calculation unit 125.

[0124] (Operation of Photographing Control Device 100)

[0125] Next, we will refer to Figure 3 and Fig. 6A The operation of the photography control device 100 to control the photography by the sub-camera 400 is described. The operation described below is equivalent to the operation of Figure 3 The operations of the CPU 101 of the recognition unit 121, the focus subject determination unit 122, the tracking target subject determination unit 123, the pan / tilt value calculation unit 124, and the zoom value calculation unit 125 are described below. Note that the operations described below are realized by the CPU 101 executing a shooting control application.

[0126] In step S201, the CPU 101 sends a shooting instruction command to the overhead camera 300 via the network I / F 105 using a predetermined protocol. In response to the command, the overhead camera 300 starts supplying a video signal (moving image data) IMG to the video input unit 107. The CPU 101 starts storing the video signal received by the video input unit 107 in the RAM 102, and then executes step S202.

[0127] In step S202, the CPU 101 obtains information ANGLE indicating a shooting direction from the main camera 500. Specifically, the CPU 101 sends a shooting direction obtaining command to the main camera 500 via the network I / F 105 using a predetermined protocol. In response to the shooting direction obtaining command, the CPU 501 of the main camera 500 sends information ANGLE indicating a current shooting direction of the main camera 500 to the shooting control device 100. For example, the information ANGLE may be a pan angle and a pitch angle of the drive unit 509. The CPU 101 stores the obtained information ANGLE in the RAM 102.

[0128] In step S203 , the identification unit 121 performs the following processing.

[0129] (1) A process of detecting the subject area is applied to the input frame image, and the detection result is stored.

[0130] (2) For each detected subject area, coordinate transformation is applied to the position information (image coordinates).

[0131] (3) For each detected subject area, recognition processing is applied and identification information is assigned (in the case of a new subject, information used for recognition processing is added).

[0132] (4) For each detected subject area, identification information ID[n] and position information POSITION[n] are stored in association with each other.

[0133] The recognition unit 121 is mainly realized by the CPU 101 and the inference unit 104. The CPU 101 reads out one frame of a video received from the overhead camera 300 from the RAM 102, and inputs the frame to the inference unit 104.

[0134] The operation of the identification unit 121 is described below in sequence.

[0135] (1) First, the inference unit 104 inputs the frame image into the machine learning model and detects the subject region. The inference unit 104 stores the position and size of each detected subject region and its detection reliability output by the machine learning model as the detection result into the RAM 102. The position and size of the subject region may be any information that allows the position and size of a rectangular region inscribed in the subject region to be specified. Here, the center coordinates of the lower edge of the rectangular region and the width and height of the rectangular region are used as the position and size of the subject region.

[0136] In addition, the inference unit 104 stores the detection result of the first frame image in association with the identification information ID[n] of the subject in the RAM 102. Here, n is the subject number, which is an integer ranging from 1 to the total number of detected subject regions. In addition, the inference unit 104 stores the subject region detected from the first frame image as a template for identifying a single subject in association with the identification information ID[n] of the subject in the RAM 102. In the case where template matching is not used when identifying a subject, the template may not be stored.

[0137] Fig. 8A The inference unit 104 shows Fig. 7A Here, the region of human subjects A to C existing in the photographing area 20 is detected, and the coordinates (foot coordinates) of the center of the lower edge of the rectangular region inscribed in the subject region are output as positions.

[0138] Note that, for example, Figure 7B In the case where marks (Marks) are arranged at known positions within the photographing area 20 to perform coordinate transformation described later, the CPU 101 detects the coordinates included in the frame image ( Fig. 7A ) and stores its position in RAM 102. Inference unit 104 may also be configured to perform detection of the labeled image. Detection of the labeled image may be performed using any known method, such as pattern matching using a labeled template. The labeled image may be detected using a pre-stored machine learning model for label detection.

[0139] (2) Next, the coordinate transformation performed by the inference unit 104 will be described. Fig. 7A The video of the overhead camera 300 is schematically shown. Figure 7B The state of viewing the imaging region 20 from directly above the center of the imaging region 20 is schematically shown. The estimation unit 104 transforms the coordinates of the position of the imaging region in the coordinate system of the overhead camera into values ​​of the coordinate system (plane coordinate system) when viewing the imaging region 20 from directly above the center of the imaging region 20.

[0140] Here, the reason for transforming the coordinates into values ​​of a plane coordinate system is that the coordinate transformation facilitates calculation of a pan value (movement angle on a horizontal plane) for causing the sub-camera 400 to capture a specific subject. Note that the present description is based on the premise that the sub-camera 400 is arranged so that the drive unit 409 performs a pan operation on a horizontal plane parallel to the ground of the photographing area 20.

[0141] Coordinate transformation can be performed using various methods; here, markers are arranged at a plurality of known positions on the ground of the shooting area 20, and the coordinates of the overhead camera coordinate system are transformed into coordinates of the plane coordinate system based on the marker positions within the video obtained from the overhead camera 300. Note that the coordinate transformation can be performed using, for example, the viewpoint position and shooting direction of the overhead camera 300 without using markers.

[0142] Coordinate transformation may be performed using a homography transformation matrix H according to the following Formula 1.

[0143]

[0144] In Formula 1, x and y on the right side are the horizontal coordinate and vertical coordinate in the overhead camera coordinate system, and X and Y on the left side are the horizontal coordinate and vertical coordinate in the plane coordinate system.

[0145] The homography transformation matrix can be calculated by substituting the coordinates of the four markers detected from the video and the (known) coordinates of the four markers arranged in the shooting area 20 into Formula 1 to solve the simultaneous equations. When the positional relationship between the shooting area 20 and the overhead camera 300 is fixed, the homography transformation matrix H can be calculated in advance and stored in the ROM 103, for example, during test shooting.

[0146] The CPU 101 sequentially reads the position of the subject area from the RAM 102 and converts the coordinates thereof into values ​​of a plane coordinate system. Figure 8B The following state is schematically shown: Using formula 1 and the homography transformation matrix H stored in ROM 103, Fig. 8A The foot coordinates (x, y) of each subject area detected in the video of the overhead camera 300 shown in FIG. 1 are converted into coordinate values ​​(X, Y) of the plane coordinate system. The CPU 101 stores the foot coordinates obtained by the coordinate conversion in the RAM 102 as POSITION[n].

[0147] (3) Next, the operation of the inference unit 104 to specify the identification information ID[n] of the object is described. It is assumed here that template matching is used to identify the object. The object recognition is performed for the result of the object detection process performed for the second time or later. For the result of the process performed for the first time, the identification information ID[n] is newly assigned to the object area.

[0148] The inference unit 104 specifies the identification information ID[n] of the detected subject area by template matching using the template stored in the RAM 102. As a result, the subject within the photographing area is identified. For example, for each detected subject area, the inference unit 104 calculates an evaluation value indicating the correlation with each template. Then, the inference unit 104 specifies the identification information ID[n] corresponding to the template having a correlation equal to or greater than a certain level with the subject area and having the highest correlation, as the identification information ID[n] of the subject area. For example, a known value such as the sum of absolute differences between pixel values ​​can be used as the evaluation value.

[0149] Note that, for a subject region that does not have a correlation equal to or greater than a certain level with any template, the inference unit 104 assigns new identification information ID[n], and adds the image of the subject region as a template.

[0150] In addition, the inference unit 104 can update the existing template using the subject area detected in the latest frame image, and can delete the template when the subject area having a correlation equal to or greater than a certain level with the template does not exist for a certain period of time. In addition, the inference unit 104 can store the template corresponding to the identification information ID[n] that appears frequently in the ROM 103.

[0151] Note that a method other than template matching can be used to identify the subject. For example, the same identification information ID[n] can be assigned to a subject area that is closest to the immediately preceding subject area in at least one of the detected position and size. In addition, a Kalman filter or the like can be used to predict the position in the current frame image based on position transitions in a plurality of past detection results associated with the same identification information, and the same identification information ID can be assigned to the subject area that is closest to the predicted position. Furthermore, these methods can be combined. When template matching is not used, the recognition accuracy of different subjects with similar appearances can be improved.

[0152] (4) The inference unit 104 stores the designated identification information ID[n] and the position (plane coordinate system) POSITION[n] of the corresponding object region in the RAM 102 in association with each other.

[0153] Note that in the processes of (1) to (4), the CPU 101 may execute processes other than subject detection instead of the inference unit 104 .

[0154] Here, the video of the overhead camera 300 is used to obtain the identification information ID[n] and the position POSITION[n] related to the object in the photographing area 20. However, the video of the sub-camera 400 may be used. In the case where there are a plurality of sub-cameras 400, the CPU 101 executes the image processing for each sub-camera 400. Fig. 6A The position of the subject area is output as a value of the coordinate system of each sub-camera 400. In this way, the overhead camera 300 is not indispensable, but it is considered that the accuracy of subject detection is higher when the overhead camera 300 is used.

[0155] Back to Fig. 6A In step S204, as Figure 3 The CPU 101 of the attention subject determination unit 122 determines the attention subject as the tracking target subject of the main camera 500. The CPU 101 may determine the attention subject of the main camera 500 among the subjects detected in step S203 based on the shooting direction of the main camera 500 obtained in step S202. The CPU 101 stores the identification information ID[n] corresponding to the subject area determined as the attention subject of the main camera 500 in the RAM 102 as the identification information MAIN_SUBJECT of the attention subject.

[0156] For example, the CPU 101 may determine the subject closest to the shooting direction of the main camera 500 in the plane coordinate system as the focus subject of the main camera 500. Note that, in the case where there are multiple subjects whose distances from the shooting direction of the main camera 500 are equal to or less than a threshold, the user can select the focus subject from these subjects.

[0157] When the user selects the focus object, the CPU 101 causes the display unit 108 or an external display device to display the frame image to which the subject detection process is applied in step S202 together with an indicator indicating the shooting direction and an indicator indicating a subject area that is a candidate for the focus object. The indicator of the subject area may be, for example, an indicator indicating Fig. 8A The CPU 101 may be a rectangular frame at the outer edge of the subject area shown in , or may be other indicators. In addition, the CPU 101 may also cause the display unit 108 to display, for example, a message for prompting selection of a subject of interest within the image.

[0158] The user can select a subject region corresponding to a desired subject of interest by operating the user input unit 106 (input device). The selection method is not particularly limited, but may be an operation of designating a desired subject region by operating a mouse or a keyboard.

[0159] Upon detecting a user operation of designating a subject region, the CPU 101 stores identification information ID[n] corresponding to the designated subject region in the RAM 102 as identification information MAIN_SUBJECT of a subject of interest.

[0160] Next, in step S205, as Figure 3 The CPU 101 of the tracking target subject determination unit 123 obtains the control content CAMERA_ROLE corresponding to the role set for the sub-camera 400. Specifically, the CPU 101 reads out the role of the sub-camera 400 that has been used. Figure 5 The control content CAMERA_ROLE obtained in the role determination processing described above and stored in the RAM 102. Note that in the case where there are a plurality of sub-cameras 400, the CPU 101 executes the processing of steps S205 to S207 for each sub-camera.

[0161] In step S206, the CPU 101 as the tracking target subject determination unit 123 determines the subject to be tracked and photographed by the sub-camera 400 according to the control content CAMERA_ROLE. The CPU 101 determines the subject to be tracked and photographed by the sub-camera 400 according to the provisions ( Figure 4 ) to determine the tracking target object of the sub-camera 400.

[0162] When the tracking target subject of the sub-camera 400 is the same as the attention subject of the main camera 500 , the CPU 101 sets the identification information MAIN_SUBJECT of the attention subject determined in step S203 as identification information SUBJECT_ID of the tracking target subject of the sub-camera 400 .

[0163] When the subject located on the left side among the subjects other than the focused subject of the main camera 500 is set as the tracking target subject of the sub-camera 400, the CPU 101 detects the leftmost subject region among the subject regions other than the focused subject, among the subject regions detected in step S203. Then, the CPU 101 sets the identification information ID[n] corresponding to the detected subject region as the identification information SUBJECT_ID of the tracking target subject of the sub-camera 400.

[0164] The CPU 101 writes the identification information SUBJECT_ID of the determined tracking target subject into the RAM 102. In the case where the tracking target subject may be different between the sub-cameras, the CPU 101 stores the identification information SUBJECT_ID of the tracking target subject in association with the identification information of the sub-camera. Note that, in the case where the tracking target subject changes, the CPU 101 maintains the information of the previous tracking target subject in the RAM 102 without deleting the information.

[0165] The following uses 9A to 9C The operation in the case where the role set for the sub-camera 400 is “main follow” will be described. For the sub-camera 400 to which the role “main follow” is set, the photographing control device 100 controls to track the object of interest of the main camera 500.

[0166] Therefore, if Fig. 9A As shown in FIG. 4 , when the subject of interest of the main camera 500 is determined to be subject B, the CPU 101 determines subject B as the tracking target subject of the sub-camera 400. Fig. 9B As shown in FIG. 4 , in the case where it is determined that the subject of interest of the main camera 500 has been changed to subject A, the CPU 101 changes the tracking target subject of the sub-camera 400 to subject A. Similarly, Fig. 9C As shown, in the case where it is determined that the subject of interest of the main camera 500 has been changed to the subject C, the CPU 101 changes the tracking target subject of the sub-camera 400 to the subject C.

[0167] The following uses FIG. 10A to FIG. 10C The operation when the role set for the sub-camera 400 is "Sub-Follow" will be described. For the sub-camera 400 set with the role "Sub-Follow", the shooting control device 100 controls to track the subject located on the left side of the subjects other than the focus subject of the main camera 500.

[0168] Therefore, if Fig. 10A As shown in FIG. 4 , when the subject of interest of the main camera 500 is determined to be subject B, the CPU 101 determines the left subject A of the subjects A and C as the tracking target subject of the sub-camera 400. Fig. 10B As shown in FIG. 4 , in the case where it is determined that the subject of interest of the main camera 500 has been changed to subject A, the CPU 101 changes the tracking target subject of the sub-camera 400 to the left subject B among subjects B and C. In addition, as shown in FIG. Fig. 10CAs shown, upon determining that the focus subject of the main camera 500 has been changed to the subject C, the CPU 101 changes the tracking target subject of the sub-camera 400 to the left subject A of the subjects A and B.

[0169] By dynamically changing the role set to the sub-camera 400 using the character control device 600, the tracking target subject of the sub-camera 400 can be changed, and automatic imaging can be flexibly performed.

[0170] Back to Fig. 6A , in step S207, the CPU 101 as the pan / tilt value calculation unit 124 calculates the amount of change in the pan angle and the pitch angle required for the sub-camera 400 to track and photograph the tracking target subject determined in step S206. In addition, the CPU 101 as the zoom value calculation unit 125 calculates the zoom value of the sub-camera 400 corresponding to the change in the angle of view of the main camera 500. The following describes a case where there is one sub-camera 400; however, in the case where there are a plurality of sub-cameras 400, the amount of change in the pan angle and the pitch angle and the zoom value are calculated for each sub-camera.

[0171] First, description will be given of the operation of the CPU 101 as the pan / tilt value calculation unit 124. It is assumed here that the following information is stored in advance in the ROM 103 as predetermined position information REF_POSI for each sub-camera 400.

[0172] ·Three-dimensional coordinates of the layout position (values ​​in the plane coordinate system)

[0173] · Shooting direction corresponding to the initial values ​​of the pan and tilt angles of the drive unit

[0174] ·Controllable pan and tilt angle range

[0175] The CPU 101 reads out the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking target subject of the sub-camera 400 from the RAM 102. Then, the CPU 101 first determines the pan angle based on the position information POSITION_OH and the arrangement position of the sub-camera 400.

[0176] Fig.11 θ is a diagram showing an example of the positional relationship between the sub-camera 400 and the tracking target object in the plane coordinate system. It is assumed here that a pan angle θ for directing the optical axis direction of the sub-camera 400 toward the object position is determined. The CPU 101 calculates the pan angle θ using the following formula 2.

[0177]

[0178] In Formula 2, px and py are the horizontal and vertical coordinates of the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking target subject. In addition, subx and suby are the horizontal and vertical coordinates of the arrangement position of the sub-camera. Here, it is assumed that the current pan angle is the initial value of 0°, and the optical axis direction is the vertical direction (Y-axis direction). In the case where the current optical axis direction is not the vertical direction, the angle difference between the current optical axis direction and the vertical direction is reflected in the angle obtained from Formula 2. In addition, if subx > px, the pan direction is counterclockwise, and if subx < px, the pan direction is clockwise.

[0179] Next, the method for determining the pitch angle will be described using Fig.12 Figure 5. Fig.12 FIG. 5 shows a state of viewing the sub-camera and the tracking target subject from the side. It is assumed that the current optical axis of the sub-camera 400 extends in the horizontal direction, its height is h1, and the height of the face of the tracking target subject to which the optical axis is to be directed is h2. It is assumed that the angle difference (pitch angle) in the height direction between the current optical axis direction and the target optical axis direction is ρ. The CPU 101 calculates the pitch angle ρ using the following Formula 3 and Formula 4.

[0180]

[0181]

[0182] The coordinate values used in Formula 4 are the same as the coordinate values used in Formula 2. It is assumed that h1 and h2 are input to the shooting control application in advance and stored in the RAM 102. In this case, the identification number associated with h2 of each subject is set to be the same as the identification number assigned in the subject detection process. Alternatively, the value measured in real time by a sensor (not shown) can be used as h2.

[0183] Here, it is assumed that the current pitch angle is the initial value of 0°, and the optical axis direction is the horizontal direction (constant height). In the case where the current optical axis direction is not the horizontal direction, the angle difference between the current optical axis direction and the horizontal direction is reflected in the angle obtained from Formula 4. In addition, if h1 > h2, the pitch direction is the downward direction, and if h1 < h2, the pitch direction is the upward direction.

[0184] The CPU 101 periodically communicates with the sub-camera 400 via the communication network 700, obtains the current optical axis direction (the pan angle and the pitch angle of the drive unit), and stores it in the RAM 102. Note that, for example, the communication cycle may be equal to or less than the inverse of the frame rate. Alternatively, the CPU 101 may hold the value of the sum of the pan angle and the pitch angle controlled for the sub-camera 400 from the initial state in the RAM 102, and use the value as the current optical axis direction.

[0185] The CPU 101 calculates the amount of change in the pan and pitch angles of the sub-camera 400 in the above manner, and stores them in the RAM 102. Note that, in the case where there are a plurality of sub-cameras 400, the CPU 101 calculates the amount of change in the pan and pitch angles for each sub-camera.

[0186] The amount of change in the pan angle and the pitch angle may be an angular velocity at which the sub-camera 400 is turned in a direction to track the target subject. For example, the CPU 101 obtains the current pan angle and pitch angle from the sub-camera 400 via the communication network 700. Then, the CPU 101 obtains a pan angular velocity proportional to the difference between the pan angle θ read out from the RAM 102 and the current pan angle. In addition, the CPU 101 obtains a pitch angular velocity proportional to the difference between the pitch angle ρ read out from the RAM 102 and the current pitch angle. The CPU 101 stores the angular velocity calculated in the above manner in the RAM 102.

[0187] Note that the amount of change in the pan angle and the pitch angle may be calculated using the video of the sub-camera 400 instead of the video of the overhead camera 300. In this case, the CPU 101 may calculate the amount of change in the pan angle based on the difference in the horizontal direction between the current optical axis direction and the direction of the tracking target subject in the coordinate system of the sub-camera 400, and calculate the amount of change in the pitch angle based on the difference in the vertical direction between the two. Furthermore, in an imaging system, the change in the shooting direction for tracking and shooting the tracking target subject may be performed in only one of the pan direction and the pitch direction; in such an imaging system, the amount of change in only one of the pan angle and the pitch direction may be calculated.

[0188] Next, a description will be given of the operation of the CPU 101 as the zoom value calculation unit 125. The CPU 101 as the zoom value calculation unit 125 periodically obtains information MAIN_ZOOM indicating the angle of view of the main camera 500 and stores it into the RAM 102. Then, in the case where the information MAIN_ZOOM has changed, the CPU 101 calculates the zoom value Z_VALUE of the sub-camera 400 based on the control content CAMERA_ROLE corresponding to the role set for the sub-camera 400.

[0189] Note that the CPU 101 can determine the zoom operation of the main camera 500 and its phase by, for example, detecting a change in the angle of view of the video of the main camera 500. For example, the change in angle of view can be detected based on the size of the object area, the time change of the interval therebetween, and the like.

[0190] Fig.13 An example of mapping of zoom values ​​of the main camera and the sub-camera is shown. It is assumed here that the main camera 500 and the sub-camera 400 optically change the viewing angle (their imaging optical systems have a zoom function). However, a similar function can be realized by using digital zoom of the image processing units 406 and 506.

[0191] Note that the zoom value is a parameter having a value corresponding to the angle of view; in the present embodiment, the zoom value decreases as the angle of view decreases (narrows), and the zoom value on the telephoto side is smaller than the zoom value on the wide-angle side. It is assumed that the sub-camera 400 and the main camera 500 can control their imaging optical systems to have an angle of view corresponding to the zoom value by sending a command to assign the zoom value to their imaging optical systems. That is, the zoom value is information related to the angle of view, and is information indicating the zoom state. For example, the zoom value may be the focal length (mm) of the imaging optical system corresponding to a full-size 35 mm image sensor; in this case, the zoom value on the telephoto side is larger than the zoom value on the wide-angle side.

[0192] exist Fig.13 , the zoom value MAIN_ZOOM of the main camera 500 is within the range of main_min to main_max. Meanwhile, the sub-camera 400 has a zoom range of sub_min to sub_max. Main_min and sub_min are zoom values ​​corresponding to the telephoto ends of the main camera 500 and the sub-camera 400, respectively, while main_max and sub_max are zoom values ​​corresponding to the wide-angle ends of the main camera 500 and the sub-camera 400, respectively. Fig.13 An example is shown in which the range of zoom values ​​of the main camera 500 is wider than the range of zoom values ​​of the sub-camera 400 at both the telephoto end and the wide-angle end.

[0193] In the case where the zoom value SUB_ZOOM of the sub camera 400 is controlled in phase with the zoom value MAIN_ZOOM of the main camera 500 , the CPU 101 calculates SUB_ZOOM corresponding to the current MAIN_ZOOM using Formula 5 below.

[0194]

[0195] On the other hand, in the case where the zoom value SUB_ZOOM of the sub-camera 400 is controlled to be inversely phased with respect to the zoom value MAIN_ZOOM of the main camera 500, SUB_ZOOM corresponding to the current MAIN_ZOOM is calculated using the following Formula 6. Specifically, the CPU 101 calculates the SUB_ZOOM corresponding to the current MAIN_ZOOM by substituting the SUB_ZOOM calculated from Formula 5 into the right side of the following Formula 6.

[0196] SUB_ZOOM=sub_max-(SUB_ZOOM-sub_min)…(Formula 6)

[0197] In the case where the main camera 500 performs digital zooming and controls the viewing angle by cropping, the CPU 101 may determine the zoom value SUB_ZOOM of the sub-camera 400 according to the size of the cropped range of the main camera 500. Specifically, the CPU 101 sets the zoom value SUB_ZOOM so that it decreases as the size of the cropped range of the main camera 500 decreases (to achieve a higher magnification), and sets the zoom value SUB_ZOOM so that it increases as the size of the cropped range of the main camera 500 increases (to achieve a lower magnification).

[0198] In addition, the content of the zoom control associated with the role of the sub-camera 400 is not limited to the control of being in phase or inverse phase with respect to the main camera 500. For example, a zoom operation that is independent of the change in the angle of view of the main camera 500 may be associated with a role. For example, an automatic zoom operation that keeps the tracking target subject at a constant size may be associated with a role. In addition, the angle of view of the sub-camera 400 may be fixed at a specific angle of view. By adding a role associated with the aforementioned zoom control to the main camera 500, Figure 4 In the control content corresponding to each role shown in, or by changing the Figure 4 The content of the zoom control corresponding to the character shown in , various zoom controls can be performed on the sub-camera 400.

[0199] Back to Fig. 6AIn step S207, the CPU 101 reads out the amount of change of the pan angle and the pitch angle and the zoom value calculated in step S206 from the RAM 102. Then, the CPU 101 generates a control command PT_VALUE for instructing the sub-camera 400 to make a change in the pan angle and the pitch angle equivalent to these amounts of change. In addition, the CPU 101 generates a control command Z_VALUE for instructing the sub-camera 400 to make a change in the angle of view equivalent to the zoom value. It is assumed that the format of the control command has been determined in advance. The CPU 101 stores the generated control commands PT_VALUE and Z_VALUE in the RAM 102. Note that in the case where there is no need to generate a control command, such as when the tracking target subject is stationary and the angle of view of the main camera 500 is not changed, step S207 can be skipped.

[0200] Then, the CPU 101 reads out the control commands PT_VALUE and Z_VALUE from the RAM 102, and transmits them to the communication network 700 via the network I / F 105. The sub-camera 400 receives the control commands PT_VALUE and Z_VALUE via the network I / F 405.

[0201] The CPU 101 executes the processing from step S201 on the next frame image in the video of the overhead camera 300. Note that Fig. 6A The processing shown in does not have to be performed on a frame-by-frame basis.

[0202] (Operation of the Overhead Camera 300)

[0203] Next, we will refer to Figure 6B The operation of the overhead camera 300 is described. The operation described below is realized by the CPU 301 executing a program.

[0204] When the overhead camera 300 is powered on, the CPU 301 initializes each functional block and then starts a shooting standby state. In the shooting standby state, the CPU 301 can start a moving image shooting process for live view display and output the display image data generated by the image processing unit 306 to the shooting control device 100 via the network I / F 305.

[0205] In the shooting standby state, the CPU 301 waits for reception of a control command via the network I / F 305. Upon receiving the control command, the CPU 301 performs an operation corresponding to the control command. The following describes an operation in the case where a shooting command is received from the shooting control apparatus 100 as a control command.

[0206] In step S301 , the CPU 301 receives a shooting command from the shooting control apparatus 100 via the network I / F 305 .

[0207] Note that in the shooting command, shooting parameters such as frame rate and resolution etc. may be specified. Furthermore, the shooting command may include settings related to the processing applied by the image processing unit 306 .

[0208] In step S302, in response to the reception of the shooting command, the CPU 301 starts a process of shooting a moving image to be supplied to the shooting control device 100. In this moving image shooting process, the shooting image quality is higher than the moving image for live view display in the moving image shooting process. For example, the captured moving image is higher than the moving image for live view display in at least one of the moving image resolution and the shooting frame rate. The image processing unit 306 applies processing to the image based on the setting of the moving image to be supplied to the shooting control device 100. The image processing unit 306 sequentially stores the generated moving image data in the RAM 302.

[0209] In step S303, the CPU 101 reads out the moving image data from the RAM 302, and transmits it to the shooting control apparatus 100 via the network I / F 305. From then on, the process from imaging to supply of moving image data continues until a control command to stop imaging is received.

[0210] (Operation of Main Camera 500)

[0211] Next, we will refer to Figure 6C The operation of the main camera 500 is described. The operation described below is realized by the CPU 501 executing a program.

[0212] When the power of the main camera 500 is turned on, the CPU 501 initializes each functional block and then starts processing for capturing a moving image to be supplied to the shooting control apparatus 100. The image processing unit 506 applies processing to an analog image signal obtained from the image sensor 507 based on the setting of the moving image to be supplied to the shooting control apparatus 100. The image processing unit 506 sequentially stores the generated moving image data into the RAM 502. The CPU 501 reads out the moving image data from the RAM 502 and supplies it to the shooting control apparatus 100 via the network I / F 505.

[0213] While supplying the moving image data to the shooting control device 100, the CPU 501 waits for receiving a control command via the network I / F 305. Upon receiving the control command, the CPU 501 performs an operation corresponding to the control command. The following describes the operation in the case where a shooting direction acquisition command is received. Note that in the case where a pan / tilt control command PT_VALUE or a zoom control command Z_VALUE is received, the CPU 501 drives the drive unit 509 according to the command.

[0214] In step S501, the CPU 501 receives a photographing direction obtaining command via the network I / F 505. The CPU 501 stores the received photographing direction obtaining command in the RAM 502.

[0215] In step S502 , the CPU 501 obtains the current pan angle and pitch angle from the drive unit 509 via the drive I / F 508 in response to reception of the shooting direction obtaining command, and stores them in the RAM 502 .

[0216] In step S503 , the CPU 501 reads out the current pan angle and pitch angle from the RAM 502 , and transmits them as information ANGLE of the photographing direction to the photographing control device 100 via the network I / F 305 .

[0217] (Operation of Sub-Camera 400)

[0218] Next, we will refer to Fig.6D The operation of the sub-camera 400 is described. The operation described below is realized by the CPU 401 executing a program.

[0219] When the power of the sub-camera 400 is turned on, the CPU 401 initializes each functional block and then starts processing for capturing a moving image to be supplied to the shooting control apparatus 100. The image processing unit 406 applies processing to an analog image signal obtained from the image sensor 407 based on the setting of the moving image to be supplied to the shooting control apparatus 100. The image processing unit 406 sequentially stores the generated moving image data into the RAM 402. The CPU 401 reads out the moving image data from the RAM 402 and supplies it to the shooting control apparatus 100 via the network I / F 405.

[0220] While supplying moving image data to the shooting control device 100, the CPU 401 waits for reception of a control command via the network I / F 305. Upon receiving the control command, the CPU 401 performs an operation corresponding to the control command. The following describes an operation in the case where a pan / tilt control command PT_VALUE and a zoom control command Z_VALUE are received from the shooting control device 100.

[0221] In step S401, the CPU 401 receives at least one of the pan / tilt control command PT_VALUE and the zoom control command Z_VALUE from the photographing control apparatus 100 via the network I / F 405. The CPU 401 stores the received control command into the RAM 402.

[0222] In step S402, the CPU 401 reads out the operation direction and the corresponding operation amount (vector amount) from the control command stored in the RAM 402, and stores them in the RAM 402. Here, in the case of the pan / tilt control command PT_VALUE, the operation direction is the direction of pan and / or tilt, and the operation amount is the target angle. Meanwhile, in the case of the zoom control command Z_VALUE, the operation amount is the zoom value, and since the operation direction can be specified from the zoom value, it is not necessary to read out and store the operation direction.

[0223] In step S403, the CPU 401 generates a driving parameter of the driving unit 409 based on the operation direction and the operation amount read out in step S402. For example, the CPU 401 can obtain a driving parameter corresponding to a combination of the operation direction and the operation amount using a table that has been previously stored in the ROM 403. Note that, in the case where the operation amount is provided in the form of a target value (target angle or zoom value), the CPU 401 obtains the driving parameter based on the difference from the current value.

[0224] In step S404, the CPU 401 drives the driving unit 409 via the driving I / F 408 based on the driving parameters obtained in step S403. Accordingly, the driving unit 409 changes the shooting direction of the sub-camera 400 to the operation direction and angle assigned by the pan / tilt control command PT_VALUE. In addition, the driving unit 409 changes the angle of view of the imaging optical system to the zoom value assigned by the zoom control command Z_VALUE.

[0225] Next, we will use Fig.14 The illustrated flowchart describes in more detail the operation of the photographing control apparatus 100 to control the photographing direction (pan and tilt) and the angle of view (zoom value) of the sub-camera 400 according to the role set to the sub-camera 400 . Fig.14 The operations shown in the flowchart are as Fig. 6A A portion of the operations of steps S205 to S207 in are performed.

[0226] In step S601 equivalent to step S205, the CPU 101 reads Figure 5 The control content CAMERA_ROLE is stored in RAM 102 in step S103.

[0227] Steps S602 to S607 are performed in, for example, step S206.

[0228] In step S602, the CPU 101 determines whether the provision related to the tracking target subject of the sub-camera 400 included in the control content CAMERA_ROLE indicates the tracking target subject (focus subject) of the main camera 500. For example, in the case where the provision related to the tracking target subject of the sub-camera 400 has a value indicating "same as main camera", the CPU 101 determines that the provision related to the tracking target subject of the sub-camera 400 indicates the tracking target subject of the main camera 500, and executes step S603. On the other hand, in the case where the provision related to the tracking target subject of the sub-camera 400 has a value indicating "different from main camera (left side)", the CPU 101 determines that the provision related to the tracking target subject of the sub-camera 400 does not indicate the tracking target subject of the main camera 500, and executes step S604.

[0229] In step S603 , the CPU 101 determines to control the photographing direction of the sub-camera 400 so as to track the tracking target object (attention object) of the main camera 500 .

[0230] In step S604 , the CPU 101 determines to control the photographing direction of the sub-camera 400 so as to track a subject located on the left side among subjects other than the subject of interest of the main camera 500 .

[0231] In step S605, the CPU 101 determines whether the provision related to the zoom control of the sub-camera 400 included in the control content CAMERA_ROLE indicates control in phase with the main camera 500. For example, in the case where the provision related to the zoom control of the sub-camera 400 has a value indicating "in phase with the main camera", the CPU 101 determines that the provision related to the zoom control of the sub-camera 400 indicates control in phase with the main camera 500, and executes step S606. On the other hand, in the case where the provision related to the zoom control of the sub-camera 400 has a value indicating "inverted relative to the main", the CPU 101 determines that the provision related to the zoom control of the sub-camera 400 does not indicate control in phase with the main camera 500, and executes step S607.

[0232] In step S606 , the CPU 101 determines to control the zoom value (angle of view) of the sub-camera 400 in phase with the change in the zoom value of the main camera 500 .

[0233] In step S607 , the CPU 101 determines to control the zoom value (angle of view) of the sub-camera 400 so that the control is inversely phased with respect to the change in the zoom value of the main camera 500 .

[0234] Use below Fig.15An example of control of the sub camera in a case where the role set to the sub camera 400 is “Master Follow” is described. Fig.15 The figure schematically shows how the shooting control device 100 controls the shooting direction and the angle of view of the sub-camera 400 when the subject of interest and the angle of view of the main camera 500 change over time during shooting. In the figure, time passes from the left to the right. Note that Fig.15 In the , the zoom status is indicated in three stages: "Telephoto end", "Middle end" and "Wide end". This is because Fig.13 As shown, the range of zoom values ​​may be different between the sub-camera and the main camera. "Telephoto end" corresponds to a state where the camera zooms in to the telephoto end, "wide-angle end" corresponds to a state where the camera zooms out to the wide-angle end, and "middle" corresponds to a zoom state between the "telephoto end" and the "wide-angle end"; however, the actual zoom value may be different between the sub-camera and the main camera. For example, when the zoom state is the "telephoto end", the zoom value of the main camera is main_min, and the zoom value of the sub-camera is sub_min.

[0235] Initially, the subject of interest (tracking target subject) of the main camera 500 is subject B, and its zoom state is "middle". Therefore, the CPU 101 determines subject B as the tracking target subject of the sub-camera 400, and controls the shooting direction so that the sub-camera 400 tracks subject B. In addition, the CPU 101 controls the zoom state of the sub-camera 400 to "middle".

[0236] Thereafter, the subject of interest of the main camera 500 changes from subject B to subject A, and its zoom state changes to the “telephoto end.” In response, the CPU 101 changes the tracking target subject of the sub-camera 400 from subject B to subject A, and controls the shooting direction so that the sub-camera 400 tracks subject A. In addition, the CPU 101 controls the zoom state of the sub-camera 400 to the “telephoto end.”

[0237] Thereafter, the subject of interest of the main camera 500 changes from subject A to subject C, and its zoom state changes to the “wide-angle end.” In response, the CPU 101 changes the tracking target subject of the sub-camera 400 from subject A to subject C, and controls the shooting direction so that the sub-camera 400 tracks subject C. In addition, the CPU 101 controls the zoom state of the sub-camera 400 to the “wide-angle end.”

[0238] As described above, when the role of the sub-camera 400 is "main follow", the CPU 101 automatically changes the tracking target subject and zoom value of the sub-camera 400 so as to follow the changes in the focus subject and the angle of view (zoom value) of the main camera 500. Fig.15 In the example shown, the degree of change of the zoom state of the sub-camera 400 is controlled to be the same as the degree of change of the zoom state of the main camera 500, but the actual zoom value may be different as long as the directions of change of the zoom values ​​are in phase with each other. For example, when the zoom state of the main camera 500 is at the telephoto end, the zoom state of the sub-camera 400 may not be at the telephoto end. Whether to make the zoom value of the sub-camera 400 consistent with the zoom value of the main camera 500 can be set using the role setting information.

[0239] Next, we will use Fig.15 akin Fig.16 An example of control of the sub-camera in a case where the role set to the sub-camera 400 is “sub-counterweight” will be described.

[0240] Initially, the subject of interest (tracking target subject) of the main camera 500 is subject B, and its zoom state is at the "telephoto end". Therefore, the CPU 101 determines the left-side subject A of subjects A and C other than subject B as the tracking target subject of the sub-camera 400, and controls the shooting direction so that the sub-camera 400 tracks subject A. In addition, the CPU 101 controls the zoom state of the sub-camera 400 to the "wide-angle end" which is the opposite phase to that of the main camera 500.

[0241] Thereafter, the subject of interest of the main camera 500 changes from subject B to subject A, and its zoom state changes to "middle". In response, the CPU 101 changes the tracking target subject of the sub-camera 400 to the left subject B among subjects B and C other than subject A, and controls the shooting direction so that the sub-camera 400 tracks subject B. In addition, since the zoom state of the main camera 500 has changed from "telephoto end" to "middle", the CPU 101 controls the zoom state of the sub-camera 400 to change from "wide-angle end" to "middle" (inverted).

[0242] Thereafter, the subject of interest of the main camera 500 changes from subject A to subject C, and its zoom state changes to the "wide end". In response, the CPU 101 changes the tracking target subject of the sub-camera 400 to the left subject A of subjects A and B other than subject C, and controls the shooting direction so that the sub-camera 400 tracks subject A. In addition, since the zoom state of the main camera 500 changes from "middle" to "wide end", the CPU 101 controls the zoom state of the sub-camera 400 to change from "middle" to "telephoto end" (inverted).

[0243] As described above, in the case where the role of the sub-camera 400 is “auxiliary counterweight”, the CPU 101 automatically changes the tracking target subject of the sub-camera 400 to one of the subjects other than the subject of interest of the main camera 500 in response to the change of the subject of interest of the main camera 500. In addition, the CPU 101 automatically changes the zoom value of the sub-camera 400 in the direction opposite to the change of the angle of view (zoom value) of the main camera 500.

[0244] Note that although Fig.16 In the example shown, the zoom state of the sub-camera 400 is controlled so that the degree of change therein is the same as the degree of change of the zoom state of the main camera 500, but the amount of change of the zoom value may be different as long as the direction of change of the zoom value is reversed. For example, when the zoom state of the main camera 500 is at the telephoto end, the zoom state of the sub-camera 400 may not be at the wide-angle end. The degree of change of the zoom state of the sub-camera 400 relative to the degree of change of the zoom state of the main camera 500 may be set using the role setting information.

[0245] (Variation Example)

[0246] An example of automatically controlling the tracking target subject and zoom value of the sub-camera 400 based on the focus subject and zoom value of the main camera 500 is described above. In the above example, automatic control is performed so that the sub-camera tracks a single subject; however, automatic control may also be performed so that the sub-camera tracks multiple subjects within a photographing area.

[0247] Now use Fig.17 Describes the case where the sub-camera 400 tracks a single subject when the role set to the sub-camera 400 is "auxiliary follow" and an example of control in the case where the sub-camera 400 tracks multiple subjects. For ease of understanding and explanation, the following describes a case where the angle of view of the main camera 500 does not change and only the control of tracking the target subject is performed. In addition, it is assumed that the angle of view of the sub-camera 400 allows all subjects within the shooting area 20 to be photographed at any time regardless of the shooting direction. Note that Fig.17 The photographing direction of the sub-camera 400 shown in the topmost layer of FIG. 1 indicates the photographing direction in the case of tracking a single subject.

[0248] Similar to Fig.16, initially, the subject of interest (tracking target subject) of the main camera 500 is subject B. Therefore, the CPU 101 determines the left subject A of subject A and subject C other than subject B as the tracking target subject of the sub-camera 400, and controls the shooting direction so that the sub-camera 400 tracks subject A. If the shooting direction is controlled so that the tracking target subject is located at the center of the screen, the subjects A to C in the video of the sub-camera 400 are squeezed together on the right side, and thus the balance is poor, as shown in the second-to-last layer. In view of this, in the case where the video of the sub-camera 400 includes a plurality of subjects including the tracking target subject, the shooting direction may be controlled so as to track these plurality of subjects. For example, the CPU 101 may control the shooting direction so as to track the centroid of the positions of the plurality of subjects A to C included in the video of the sub-camera 400. As a result, the sub-camera 400 shoots the video shown in the bottom layer.

[0249] according to Fig.17 In the example shown, all of the subjects A to C are photographed regardless of which one of the subjects A to C is tracked by the sub-camera 400 , so the photographing direction of the sub-camera 400 is substantially constant even if the focus subject of the main camera 500 has changed.

[0250] (Second Modification)

[0251] In addition, after determining the tracking target subject of the sub-camera 400, the CPU 101 can control the sub-camera 400 to focus on the tracking target subject. Although the CPU 401 basically continuously controls the focus distance in the sub-camera 400 so as to focus on the assigned tracking target subject, the shooting control device 100 can set the AF frame of the sub-camera 400 at the position of the tracking target subject. This allows the sub-camera 400 to focus on the tracking target subject quickly and reliably. Note that setting the AF frame when the pan speed becomes slower (equal to or less than the threshold) is likely to shorten the time period required to achieve focusing because the tracking target is likely to be located in the center of the screen.

[0252] (Third Modification)

[0253] In addition, the sub-camera 400 may be controlled without using the overhead camera 300. In this case, the shooting direction of the sub-camera 400 may be determined based on the arrangement positions of the main camera 500 and the sub-camera 400 and the shooting direction of the main camera 500 (the direction in which the main camera 500 faces the tracking target subject). The main camera 500 performs a subject detection process, and the CPU 101 obtains and uses the result of the subject detection process; as a result, the sub-camera 400 may be controlled. For example, the CPU 101 obtains an image of a subject area from the main camera 500 as a result of the subject detection process. Then, the CPU 101 may control the sub-camera 400 to perform a subject tracking process using the obtained image as a template. Alternatively, the CPU 101 may use the obtained image as a template and control the sub-camera 400 to track a subject area having a low correlation with the template.

[0254] (Fourth Modification)

[0255] Although the shooting control device 100 and the character control device 600 have been described as devices independent of the main camera 500, the functions of the shooting control device 100 and the character control device 600 may be embedded in the main camera 500. In this case, the video of the overhead camera 300 is supplied to the main camera 500. This configuration can reduce the number of devices required to implement a multi-camera imaging system.

[0256] As described above, according to this embodiment, when the operation of the sub-camera is automatically controlled based on the state and video of the main camera, automatic control corresponding to the role set for the sub-camera is performed. Therefore, the shooting control device of this embodiment can realize automatic shooting control with higher degree of freedom while saving manpower.

[0257] <Second embodiment>

[0258] Next, a second embodiment of the present invention will be described. In this embodiment, in addition to the state or video of the main camera and information of the role set to the sub-camera, the video of the sub-camera is also considered to determine the tracking target subject of the sub-camera.

[0259] Fig.18 is a schematic diagram showing an exemplary configuration of an imaging system 10' according to the present embodiment. Fig.18 In the embodiment, constituent elements similar to those of the imaging system 10 according to the first embodiment are assigned the same Figure 1The same reference numerals are used for the image pickup system 10' according to the present embodiment, and the description thereof is omitted. The image pickup system 10' according to the present embodiment includes two sub-cameras, namely, the sub-camera A 8000 and the sub-camera B 900. Since the functional configurations of the sub-camera A 8000 and the sub-camera B 900 are similar to those of the sub-camera 400 described in the first embodiment, the description thereof is omitted.

[0260] In this embodiment, the character control device 600 sets a character to the photographing control device 100, and the character setting information includes control contents that differ between different sub-cameras. The CPU 101 controls the operation of each sub-camera according to the character set to the photographing control device 100.

[0261] Fig.19 1 is a diagram showing an example of role setting information according to the present embodiment. Here, as an example, only the role "auxiliary follower" and the corresponding control content of each auxiliary camera are shown. However, the types of roles that can be set for the shooting control device 100 and the control content corresponding to the types of roles are not limited to Fig.19 The example shown in .

[0262] Here, the provisions related to the sub-camera A 8000 indicate that the left-side subject among the subjects different from the tracking target subject of the main camera is set as the tracking target subject, and control is performed to focus on the tracking target subject. At the same time, the provisions related to the sub-camera B 900 indicate that the right-side subject among the subjects different from the tracking target subject of the main camera is set as the tracking target subject, and control is performed to focus on the tracking target subject. Here, since the total number of subjects is known in advance to be three, the provisions indicate the left or right person among two people; however, the provisions may indicate only the left or right side. Note that although the provisions related to the zoom control can be provided similarly to the first embodiment, the zoom control is omitted for the sake of simplicity of description.

[0263] Fig.19 The character setting information shown in is stored in advance in the ROM 103 of the shooting control apparatus 100. Alternatively, the character setting information may be supplied from the character control apparatus 600 to the shooting control apparatus 100 and stored in the RAM 102 by the CPU 101.

[0264] Use below Fig. 20A and Fig. 20B How the photographing control apparatus 100 controls the sub-camera A 8000 and the sub-camera B 900 based on the character setting information is described.

[0265] Fig. 20AThe positional relationship among the imaging area 20 , the main camera 500 , the sub-camera A 8000 , and the sub-camera B 900 , and the subjects A to C within the imaging area 20 at the start of imaging are shown. Fig. 20B A state is shown in which, after the start of image capturing, the subject A moves from the left side to the right side of the subject B, and the subject C moves from the right side to the left side of the subject B.

[0266] exist Fig. 20A In the stage, the CPU 101 of the shooting control device 100 is based on Fig.19 The role setting information shown controls the operations of the sub-camera A 8000 and the sub-camera B 900. That is, the CPU 101 determines the left-side subject A among the subjects A and C other than the focus subject (subject B) of the main camera 500 as the tracking target subject of the sub-camera A 8000, and determines the right-side subject C among them as the tracking target subject of the sub-camera B 900.

[0267] The CPU 101 controls the photographing directions of the sub-camera A 8000 and the sub-camera B 900 so as to track the determined tracking target object. In addition, the CPU 101 controls the sub-camera A 8000 and the sub-camera B 900 so as to focus on the determined tracking target object.

[0268] exist Fig. 20B , the subject A tracked by the sub-camera A 8000 does not satisfy the condition “left-side subject” (except the subject of interest of the main camera 500). Therefore, the CPU 101 changes the tracking target subject of the sub-camera A 8000 to the subject A other than the subject of interest (subject B) of the main camera 500 and the left-side subject C among the subjects C. Similarly, the subject C tracked by the sub-camera B 900 does not satisfy the condition “right-side subject” either. Therefore, the CPU 101 changes the tracking target subject of the sub-camera B 900 to the subject A other than the subject of interest (subject B) of the main camera 500 and the right-side subject A among the subjects C. In addition, the CPU 101 controls the sub-camera A 8000 and the sub-camera B 900 so as to focus on the determined tracking target subject.

[0269] like Fig. 20B As shown, when the subject moves significantly, the possibility that there are other subjects in front of it increases, so the tracking target subject is likely to be hidden. By configuring the setting so that the sub-camera arranged on the left side relative to the shooting area 20 tracks the subject located on the left side, it is possible to provide a provision for effectively changing the tracking target subject when the tracking target subject moves significantly to the right side. The same is true for the sub-camera arranged on the right side relative to the shooting area 20.

[0270] This embodiment can provide regulations related to the control content of making different secondary cameras track different subjects under the same role. Therefore, the shooting control device 100 of this embodiment can automatically control the secondary cameras to shoot videos tracking different subjects based on the information obtained from the status and video of the main camera.

[0271] (Variation Example)

[0272] In the present embodiment, the shooting direction of the sub-camera for tracking a specific subject is estimated by transforming the coordinates of the position of the subject area detected from the video of the overhead camera 300. However, similar estimation may be performed based on the videos of the sub-camera A 8000 and the sub-camera B 900. In this case, although the processing load of the shooting control device 100 increases, coordinate transformation is not required; as a result, the control accuracy of the shooting direction of the sub-camera can be improved.

[0273] (Second Modification)

[0274] In addition, the pan / tilt values ​​used to control the sub-camera A 8000 and the sub-camera B 900 in this embodiment are calculated to automatically track the tracking target. However, automatic tracking is not indispensable. For example, the character setting information may include a provision for controlling the pan value of the sub-camera A 8000 and the sub-camera B 900 according to the zoom value of the main camera 500.

[0275] As an example, when the zoom state of the main camera 500 is the "wide-angle end", the sub-camera A 8000 may be controlled to point to a direction tilted 45 degrees to the left, and the sub-camera B 900 may be controlled to point to a direction tilted 45 degrees to the right. On the other hand, when the zoom state of the main camera 500 is the "telephoto end", both the sub-camera A 8000 and the sub-camera B 900 may be controlled to point to the center direction at an angle of 0 degrees.

[0276] By performing the above control, the shooting directions of the plurality of sub-cameras can be controlled synchronously with the zoom state of the main camera 500, thereby enhancing the performance effect. For example, when the main camera 500 is zooming in on a specific subject, the sub-camera A8000 and the sub-camera B900 can cooperatively change their shooting directions while zooming in on the same subject. By performing the above control, in a performance in which images of a plurality of cameras are displayed on a plurality of monitors, a plurality of captured images obtained by controlling the plurality of cameras in synchronization with each other can be simultaneously viewed. In addition, by automatically controlling the separate cameras to which the same role is assigned, the difference between the temporal changes in the angle of view of the cameras is reduced compared to the case in which the respective cameras are manually operated, the changes in the angle of view have a strong sense of integrity, and the performance effect of further improving the sense of reality can be expected.

[0277] (Third Modification)

[0278] Furthermore, although the present embodiment is described using an example in which the main camera 500 is controlled as the master, and the sub-camera A8000 and the sub-camera B900 are controlled as slaves, the master-slave relationship between the main camera 500 and the sub-camera A8000 may be changed dynamically. For example, among the main camera 500, the sub-camera A8000, and the sub-camera B900, the camera that is shooting the main video may be controlled as the master, and the other cameras may be controlled as slaves. In this case, the shooting control device 100 may obtain information indicating which video of the camera is selected as the main video from an external device (such as a switch for selecting a video), or distinguish such information based on a tally signal. Similar control may be performed when selecting as a video related to viewing or recording of the viewer (such as a video for recording) in addition to when making a selection as the main video.

[0279] In addition, the master-slave relationship between the cameras can be switched when the sub-camera A 8000 or the sub-camera B 900 is manually operated. In this case, when the user manually operates one of the sub-cameras, control of the other cameras is started with the sub-camera acting as the master, thereby providing convenience.

[0280] ·<Third Embodiment>

[0281] Next, a third embodiment of the present invention will be described. The imaging system according to the present embodiment provides information indicating the shooting status of the main camera and the sub-camera to the user in a visual manner; as a result, the convenience of the imaging system is improved.

[0282] Fig.21 1 is a schematic diagram showing an exemplary configuration of an imaging system 10 according to a third embodiment of the present invention. Here, for ease of explanation and understanding, an exemplary configuration based on the imaging system 10 according to the first embodiment is shown. The description of the first embodiment is applicable to constituent elements other than the display device 800.

[0283] The display device 800 presents information of the sub-camera 400 and the main camera 500, especially information related to the subject currently being photographed, to the user using the camera system. Although the display device 800 is described as a component separate from the photographing control device 100 and the character control device 600 in the present embodiment, the function of the display device 800 can be implemented by at least one of the photographing control device 100 and the character control device 600.

[0284] The display device 800 displays the shooting information generated by the shooting control device 100. In the case where the shooting control device 100 generates image data for display, the display device 800 may be a device that exclusively provides a display function, such as a liquid crystal display, or may be any electronic device that can be used as an external display device of the shooting control device 100. In addition, the display device 800 may be any electronic device (information processing device, smart phone, tablet terminal, etc.) that can communicate with the shooting control device 100 and has a display function. The display device 800 may generate and display image data based on the shooting information obtained from the shooting control device 100.

[0285] Fig. 22 It shows the composition Fig.21 The camera system 10 is a block diagram of an exemplary functional configuration of each device. Figure 2 The related description is applicable to devices other than the display device 800 .

[0286] The display device 800 is configured such that a CPU 801 , a RAM 802 , a ROM 803 , a user input unit 804 , a network interface (I / F) 805 , an image processing unit 806 , and a display unit 811 are mutually connected via an internal bus 810 .

[0287] The CPU 801 is a microprocessor capable of executing programmed instructions. For example, the CPU 801 implements the functions of the display device 800 described later by reading a program stored in the ROM 803 into the RAM 802 and executing the program. For example, the CPU 801 can implement the functions of the display device 800 by executing a display control application operating on the basic software (OS).

[0288] The RAM 802 is used to load programs executed by the CPU 801 and temporarily store data processed by the CPU 801 , currently processed data, etc. In addition, a part of the RAM 802 can be used as a video memory of the display unit 811 .

[0289] The ROM 803 is a nonvolatile rewritable memory, and stores programs (OS and applications) executed by the CPU 801 , user data, and the like.

[0290] The user input unit 804 is an input device such as a mouse, a keyboard, and a touch panel. The display apparatus 800 accepts a user instruction via the user input unit 804.

[0291] The network I / F 805 is an interface for connecting the display device 800 to the communication network 700. The display device 800 (CPU 801) can communicate with an external device (such as the shooting control device 100) in the communication network 700 via the network I / F 805. Note that the display device 800 can communicate with the external device (such as the shooting control device 100) in the communication network 700 via other communication interfaces (USB, etc.) to communicate with external devices.

[0292] The CPU 801 obtains information (eg, a network address) necessary for communicating with a device in the communication network 700 at an arbitrary timing, and stores the information in the RAM 802 .

[0293] The image processing unit 806 generates image data (display image data) to be displayed on the display unit 811 from the image data obtained via the network I / F 805. In addition, the image processing unit 806 generates display image data based on the shooting status of the main camera 500 and the sub-camera 400 received from the shooting control device 100. The image processing unit 806 stores the generated display image data in the RAM 802. Note that the image processing unit 806 can reduce the amount of data by applying compression processing to the display image data as needed. In addition, the image processing unit 806 can apply image quality adjustment such as color correction, exposure control, and sharpness correction to the received image data.

[0294] The display unit 811 is a display device such as a liquid crystal display (LCD), etc. The display unit 811 displays a GUI screen provided by an OS, a display control application, etc. The display unit 811 also displays an image indicating the shooting status of the main camera 500 and the sub-camera 400 .

[0295] The operation of the display device 800 is controlled by a user instruction issued via the user input unit 804 and an instruction (command) received from the shooting control device 100 or the like via the network I / F 805 .

[0296] and Figure 3 similar, Fig.23 4 is a diagram showing a processing sequence performed when the photographing control device 100 controls the operation of the sub-camera 400, focusing on the main operations and signal flows. The functional blocks shown within the photographing control device 100 schematically indicate the main operations, and correspond to the main functions provided by the photographing control application. Fig.23 The functional blocks are composed of the CPU 101 executing the shooting control application and Fig. 22 The method may be implemented by combining one or more functional blocks of the photographing control device 100 shown in FIG.

[0297] pass Fig.23 and Figure 3As can be understood from the comparison between the main camera 500 and the sub-camera 400, the shooting control device 100 of the present embodiment has the functions of the subject information superposition unit 126 and the angle of view information superposition unit 127. The subject information superposition unit 126 superimposes information indicating the tracking target subject of each of the main camera 500 and the sub-camera 400 on the video obtained from the overhead camera 300. In addition, the angle of view information superposition unit 127 superimposes information indicating the angle of view of each of the main camera 500 and the sub-camera 400 on the video obtained from the overhead camera 300. The operations of the subject information superposition unit 126 and the angle of view information superposition unit 127 can be switched between the enabled state and the disabled state individually or dynamically. Note that it is allowed to adopt a configuration in which a display control device other than the shooting control device 100 includes the subject information superposition unit 126 and the angle of view information superposition unit 127, receives various types of information from the shooting control device 100, and outputs an image generated by the display control device to the display device 800.

[0298] Use below Fig.24A and Fig. 24B 1 and 2, and the like, to describe the operation for generating an image on which the photographing information of the main camera 500 and the sub-camera 400 are superimposed, among the operations performed by the photographing control device 100 in this embodiment. The photographing control device 100 can perform the operation described below in parallel with other operations such as the automatic photographing control operation of the sub-camera.

[0299] Fig.24A 1 is a flowchart related to an operation of superimposing information of a tracking target subject (the information of the tracking target subject is one type of photographic information) on a video from the overhead camera 300. Fig. 24B 1 is a flowchart related to an operation of superimposing information of a viewing angle (the information of the viewing angle is a type of photographing information) on a video from the overhead camera 300. It is assumed here that the subject information superimposing unit 126 superimposes the information of the tracking target subject, and the viewing angle information superimposing unit 127 further superimposes the information of the viewing angle. Note that the order of superimposition may be reversed, and whether or not to perform superimposition may be individually controlled.

[0300] In addition, the video of the overhead camera 300 is executed once for every predetermined number of frames. Fig.24A and Fig. 24B The operations shown in the flowchart of (e.g., once per frame, once every two frames, etc.).

[0301] In step S701 , the CPU 101 starts obtaining the video signal (image data) IMG from the overhead camera 300 , similarly to step S201 .

[0302] In step S702 , the CPU 101 obtains information ANGLE indicating the photographing direction from the main camera 500 , similarly to step S202 .

[0303] In step S703, the CPU 101 (recognition unit 121) reads out the image data IMG corresponding to one frame from the RAM 102, and performs the subject detection processing described in relation to step S203. Then, for each subject area, the CPU 101 determines the position where the indicator is to be superimposed based on the result of the subject detection. It is assumed here that a frame indicating a rectangular area inscribed in the subject area (such as Fig. 8A Therefore, for each object region, the CPU 101 determines the rectangular information DETECT_POSI as information for specifying the position of the rectangular region based on the output of the inference unit 104, and stores it in the RAM 102.

[0304] The rectangle information DETECT_POSI may specifically be the image coordinates of the vertices of the rectangular region (the vertices of the rectangular region are diagonals). Here, it is assumed that the image coordinates are the coordinates of the upper left vertex (xul, yul) and the coordinates of the lower right vertex (xdr, ydr).

[0305] In step S704, similarly to step S204, the CPU 101 as the attention subject determination unit 122 determines the attention subject of the main camera 500. The CPU 101 stores identification information ID[n] corresponding to the subject area determined as the attention subject of the main camera 500 in the RAM 102 as identification information MAIN_SUBJECT of the attention subject.

[0306] In step S705, similarly to step S205, the CPU 101 as the tracking target subject determination unit 123 obtains the control content CAMERA_ROLE corresponding to the role set to the sub-camera 400. Specifically, the CPU 101 reads out the control content CAMERA_ROLE from the RAM 102.

[0307] In step S706, similarly to step S206, the CPU 101 as the tracking target subject determination unit 123 determines the subject to be tracked and photographed by the sub-camera 400 based on the control content CAMERA_ROLE. The CPU 101 determines the subject to be tracked and photographed by the sub-camera 400 based on the provisions ( Figure 4 ) determines the tracking target object of the sub-camera 400.

[0308] In the case where the indicator indicating the shooting angle of view of the tracking target object is displayed instead of the indicator of the object area and the indicator of the tracking target object of each camera, the CPU 101 executes after the processing of step S706 ends. Fig. 24B Step S801.

[0309] In step S707, the CPU 101 as the subject information superimposition unit 126 reads out the same image data IMG as the image data IMG read out from the RAM 102 in step S703. Then, the CPU 101 reads out the rectangle information DETECT_POSI of each subject area from the RAM 102, and superimposes the indicator based on the rectangle information DETECT_POSI on the image data IMG. As described above, the indicator is a rectangular frame whose diagonal vertices have the coordinates indicated by the rectangle information DETECT_POSI. In addition, here, the color of the rectangular frame is different between different subject areas. The CPU 101 (subject information superimposition unit 126) stores the image data superimposed with the indicator of the subject area in the RAM 102.

[0310] Fig.25 : is a diagram showing an example of the relationship between the identification ID of the subject detected from the frame image of the video of the overhead camera 300, the rectangular information DETECT_POSI, and the display color of the indicator of each subject area. The position information (rectangular information DETECT_POSI) of the circumscribed rectangular area of ​​the subject area corresponding to the identification ID "ID1" is (xul1, yul1) and (xdr1, ydr1), and the indicator color "red" is associated with it. This means that for the subject whose identification ID is determined to be ID1, a red indicator (here a rectangular box) is superimposed at the position in the image indicated by the rectangular information. The same is true for the other identification IDs (i.e., ID2 and ID3). Note that the color assigned to the identification ID can be predetermined.

[0311] Fig.26 An example of an image indicated by image data with an indicator of a subject area superimposed thereon is shown. It is assumed here that three human subjects A to C are detected from the overhead image and are assigned ID1 to ID3 as identification IDs, respectively. The overhead image is one frame of video from the overhead camera 300.

[0312] In such Fig.25 In the case where the indicator colors are defined as shown, indicators 261 to 263 in the form of red, green, and blue rectangular frames, respectively, are superimposed on the areas of the subjects A to C.

[0313] Note that the indicator of each subject is not limited to the indicator in the form of a rectangular frame shown as an example. For example, the rectangular frame may include at least a portion of the subject area. In addition, within the rectangular frame, the area outside the subject area may be filled with the same color as the frame. In addition, the indicator may be translucent, for example. The use of a translucent indicator allows the subject area to be visually identified even when the indicator overlaps with the subject area. The indicator of each subject may contact (or not contact) the subject area associated with it, as long as its correspondence with the subject area is clear.

[0314] In step S708, the CPU 101 reads out the image data generated in step S707 from the RAM 102. Then, the CPU 101 generates image data SUBJECT_IMG on which identification information of the main camera 500 and the sub-camera 400 is superimposed near the regions of the tracking target objects of the main camera 500 and the sub-camera 400, respectively.

[0315] First, the CPU 101 reads out MAIN_SUBJECT indicating the ID of the subject of interest (tracking target subject) of the main camera 500 and SUBJECT_ID indicating the tracking target subject ID of the sub-camera 400 from the RAM 102. The CPU 101 superimposes the identification information of the main camera 500 (here, it is assumed that the characters “main camera”) near the subject area corresponding to the identification ID of MAIN_SUBJECT. Similarly, the CPU 101 superimposes the identification information of the sub-camera 400 (here, it is assumed that the characters “sub-camera”) near the subject area corresponding to SUBJECT_ID.

[0316] Note that in addition to the identification information of the sub-camera 400, identification information of a role set to the sub-camera 400 may be superimposed. For example, in the case where a role "Master Follower" is set to the sub-camera 400, characters "Sub-Camera (MF)" may be superimposed.

[0317] The position where the camera identification information is superimposed may be, for example, directly above the subject indicator in the form of a rectangular frame. In this case, the CPU 101 may determine the position where the camera identification information is superimposed based on the rectangular information. Note that the CPU 101 may determine the position where the camera identification information is superimposed based on the position information of the subject area. In addition, the camera identification information may also be semi-transparent.

[0318] The position where the identification information of the camera is superimposed can be determined separately according to, for example, the position of the corresponding subject area. For example, in the case where the subject area is close to the edge of the image, the display position can be changed. Specifically, if the subject area is close to the upper edge of the image, the identification information of the camera can be superimposed below the subject area, and if the subject area is close to the left edge of the image, the identification information of the camera can be superimposed on the right side of the subject area. In addition, the position can be dynamically changed in consideration of the distance between the subject areas, the overlap of the subject areas, etc.

[0319] In addition, the identification information of the camera is not limited to the role of the camera (primary or secondary). For example, the identification information of the camera can be any information that allows the camera to be identified, such as the IP address and model name of the camera. In addition, multiple types of identification information can be superimposed at the same time, or in a switchable manner.

[0320] Furthermore, the color of the identification information of the camera may be a color associated with the corresponding subject, ie, the same color as the indicator of the corresponding subject area. Alternatively, a predetermined fixed color may be used for each of the main camera and the sub-camera.

[0321] The CPU 101 writes data of an image obtained by superimposing an indicator indicating a subject area and information of a tracking target subject of each camera as SUBJECT_IMG into the RAM 102. Note that the CPU 101 may output the image data SUBJECT_IMG to the display device 800 as display image data OUT_IMAGE without superimposing the angle of view information.

[0322] FIG. 27A to FIG. 27C An example of a case where, for human subjects A to C present in the photographing area, the tracking target subject is different between the main camera 500 and the sub-camera 400. Specifically, this is a case where the sub-camera 400 is performing automatic photographing according to the role of "sub-follow" or "sub-confrontation". In addition, FIG. 27D to FIG. 27F Shown are FIG. 27A to FIG. 27C Corresponding to the generated image data SUBJECT_IMG.

[0323] The area of ​​the subjects A to C in the overhead image is superimposed with a Fig.25 An indicator in the form of a rectangular frame of a color showing the relationship shown is provided regardless of the tracking target object of the main camera 500 and the sub-camera 400 .

[0324] Fig.27AThis is the state where the main camera 500 and the sub-camera 400 are tracking the subject B and the subject A, respectively. Therefore, the character 272 of “main camera” is superimposed near the area of ​​the subject B (subject B is the tracking target subject of the main camera 500). In addition, the character 271 of “sub-camera” is superimposed near the area of ​​the subject A (subject A is the tracking target subject of the sub-camera 400).

[0325] Fig.27B This is the state where the main camera 500 and the sub-camera 400 are tracking the subject A and the subject B, respectively. Therefore, the character 272 of “main camera” is superimposed near the area of ​​the subject A (subject A is the tracking target subject of the main camera 500). In addition, the character 271 of “sub-camera” is superimposed near the area of ​​the subject B (subject B is the tracking target subject of the sub-camera 400).

[0326] Fig.27C This is a state where the main camera 500 and the sub-camera 400 are tracking the subject C and the subject A, respectively. Therefore, the character 272 of “main camera” is superimposed near the area of ​​the subject C (the subject C is the tracking target subject of the main camera 500). In addition, the character 271 of “sub-camera” is superimposed near the area of ​​the subject A (the subject A is the tracking target subject of the sub-camera 400).

[0327] Next, use Fig. 24B The flowchart shown describes a process of adding an indicator indicating the photographing angle of view of a tracking target object to an overhead image.

[0328] In step S801, the CPU 101 as the viewing angle information superimposing unit 127 reads out the overhead image data SUBJECT_IMG generated in step S708 on which the indicator of the subject area and the indicator of the tracking target subject of each camera are superimposed, from the RAM 102. Note that, in the case where the indicator of the subject area and the indicator of the tracking target subject of each camera are not superimposed, the CPU 101 reads out the image data IMG of the same frame as the image data read out from the RAM 102 in step S703.

[0329] In step S802, the CPU 101 obtains the current zoom values ​​of the main camera 500 and the sub-camera 400. The value most recently obtained from the main camera 500 and stored in the RAM 102 by the zoom value calculation unit 125 may be used as the zoom value MAIN_ZOOM of the main camera 500. In addition, the zoom value Z_VALUE most recently determined and stored in the RAM 102 by the zoom value calculation unit 125 in step S207 may be used as the zoom value of the sub-camera 400. Note that the CPU 101 can obtain the zoom value from the main camera 500 and the sub-camera 400 by transmitting a command for obtaining the zoom value via the network I / F 105.

[0330] In step S803, the CPU 101 reads out image data of an indicator for indicating the shooting angle of view, for example, from the ROM 103. It is assumed here that a specific image is scaled to a size corresponding to the angle of view, and the resulting image is used as an indicator indicating the shooting angle of view. Note that the indicator indicating the shooting angle of view may be in other forms, such as an indicator indicating a zoom value using characters.

[0331] In step S804, the CPU 101 generates an indicator indicating the photographing angle of view of the tracking target object of the main camera 500 by resizing the image obtained in step S803 according to the zoom value of the main camera 500 obtained in step S802. For the sub-camera 400, the CPU 101 similarly generates an indicator indicating the photographing angle of view of the tracking target object.

[0332] Regarding the resizing rate of the image, the corresponding relationship between the resizing rate and the zoom value is pre-stored in the ROM 103, so that the smaller the zoom value (the narrower the angle of view), the larger the image is resized, and the larger the zoom value (the wider the angle of view), the smaller the image is resized. In addition, in the present embodiment, the position DRAW_POSI in the image of the indicator indicating the shooting angle of view of the tracking target object to be superimposed on the overhead image is pre-stored in the ROM 103. For each of the main camera 500 and the sub-camera 400, the position DRAW_POSI defines the coordinates of the vertices of the diagonal corners of the rectangular area for displaying the indicator. Note that the respective positions DRAW_POSI can be stored according to the number of sub-cameras 400.

[0333] Here we assume that Fig.28D and Fig.28EAs shown, the indicator indicating the shooting angle of view is displayed at the bottom of the overhead image. In this case, the position DRAW_POSI includes the coordinates of the upper left vertex (drawM_Xs, drawM_Ys) and the coordinates of the lower right vertex (drawM_Xe, drawM_Ye) of the rectangular area for displaying the indicator of the shooting angle of view of the main camera 500. In addition, the position DRAW_POSI also includes the coordinates of the upper left vertex (drawS_Xs, drawS_Ys) and the coordinates of the lower right vertex (drawS_Xe, drawS_Ye) of the rectangular area for displaying the indicator of the shooting angle of view of the sub-camera 400.

[0334] The CPU 101 superimposes the zoomed image in the rectangular region indicated by the position DRAW_POSI in the overhead image.

[0335] Use below FIG. 28A to FIG. 28E The operations related to the generation and superposition of the indicator indicating the photographing angle of view are described in more detail. Fig.28A The following state is shown: among human subjects A to C existing in the photographing area, the tracking target subject of the main camera 500 is subject B, and the tracking target subject of the sub-camera 400 is subject A. Specifically, this is a case where the sub-camera 400 is performing automatic photographing according to the character "auxiliary follow". Fig.28B and Fig.28C The video of the main camera 500 and the video of the sub-camera 400 are respectively shown.

[0336] Fig.28D It shows that when the shooting angles of the main camera 500 and the sub-camera 400 are both wide angles, Fig.28A An example of display of an indicator indicating the shooting angle of view in the state of Fig.28E FIG. 4 shows that when the shooting angles of the main camera 500 and the sub-camera 400 are both telephoto, Fig.28A An example of display of an indicator indicating the shooting angle of view in the state of .

[0337] Here, an image (human body model image) having a simple shape of a figure imitating a torso and a head of a human body is used to generate the indicator. Note that for ease of understanding and explanation, it is assumed that the main camera 500 and the sub-camera 400 have the same zoom value range, and control is performed so that the zoom value of the sub-camera 400 is equal to the zoom value of the main camera 500 due to the character "auxiliary follow". Fig.28D and Fig.28E As shown, a small indicator is displayed when the shooting angle of view is a wide angle, and a large indicator is displayed when the shooting angle of view is a telephoto.

[0338] In step S805, the CPU 101 performs processing for clarifying which cameras the indicators of the shooting angle of view correspond to. Specifically, the CPU 101 Fig.25 The color of the relationship between the identification ID and the indicator color is given to the outline of the rectangular area for displaying the indicator. Specifically, the CPU 101 renders at least a portion of the outline of the rectangular area for displaying the indicator in a color corresponding to the identification ID of the subject indicated by the indicator. Alternatively, the CPU 101 superimposes an image of a color corresponding to the identification ID of the subject indicated by the indicator on at least a portion of the outline of the rectangular area for displaying the indicator.

[0339] exist Fig.28A In the example shown in FIG. 5 , the tracking target subject of the main camera 500 is subject B. Fig.25 As shown, the color of the indicator corresponding to the subject B (identifier ID2) is green. Therefore, the CPU 101 gives green to the outline of the rectangular area for displaying the indicator indicating the shooting angle of view of the subject B. Similarly, the CPU 101 gives red to the outline of the rectangular area for displaying the indicator indicating the shooting angle of view of the subject A (identifier ID1).

[0340] Note that the area for displaying the indicator indicating the shooting angle of view may also include identification information of the camera corresponding to the indicator for tracking the target subject. Including the identification information of the camera can more directly understand which camera the shooting angle of view corresponds to.

[0341] CPU 101 Generation Fig.28D and Fig.28E The CPU 101 sends the display image data OUT_IMAGE to the display device 800. The display image data OUT_IMAGE includes at least one of the following information: (1) indicating the shooting angle of view and the identification information of the subject corresponding to the indicator. FIG. 27D to FIG. 27F (2) an indicator indicating the shooting angle of view and identification information of the object corresponding to the indicator. Note that the CPU 101 can display the display image data OUT_IMAGE on the display unit 108.

[0342] Upon receiving the display image data OUT_IMAGE via the network I / F 805, the CPU 801 of the display device 800 stores it in the RAM 802. Then, the CPU 801 displays the display image data OUT_IMAGE on the display unit 811. The CPU 801 may convert the display image data OUT_IMAGE into data suitable for display on the display device 811 as needed. For example, the CPU 801 may scale the display image data OUT_IMAGE according to the resolution of the display device 811.

[0343] An indicator that indicates the viewing angle with a size is generated by scaling an image of a shape obtained by deforming a torso and a head according to a zoom value. However, an indicator that does not use a size to indicate the viewing angle may be generated. For example, the size of a range used as an indicator in a predetermined image (e.g., an image obtained by deforming the entire human body) may vary according to the zoom value. Specifically, the range to be used as an indicator may become narrower as the viewing angle decreases; for example, in the case where the shooting viewing angle is a wide angle, the entire image (full body) may be used as an indicator, and in the case where the shooting viewing angle is a telephoto, only the upper body in the image may be used as an indicator.

[0344] In addition, an image imitating a subject may not be used as an indicator. For example, instead of an image of a shape obtained by deforming a torso and a head, an image of a fruit (e.g., an apple), a musical instrument (e.g., a guitar), etc. may be scaled and used as an indicator of the shooting angle of view. Alternatively, an image of a subject cut out from an overhead image may be used. Instead of scaling, one of a plurality of pre-prepared images having different sizes may be selected based on a zoom value. An indicator that changes size according to a zoom value may be generated using any known method.

[0345] In the case where there are multiple sub-cameras 400, it is sufficient to display an indicator of each sub-camera 400. Starting from the leftmost rectangular area, the indicators indicating the shooting angle of view may be displayed in the order of the main camera, sub-camera 1, sub-camera 2, etc. Alternatively, the indicator indicating the shooting angle of view of the main camera may be displayed on the rectangular area located at the center in the horizontal direction, and the indicators of the shooting angle of view of each sub-camera may be displayed on the rectangular areas on the left and right in a predetermined order.

[0346] In addition, in the present embodiment, it is assumed that each camera has one tracking target subject. However, each camera may have multiple tracking target subjects. An indicator indicating the shooting angle of view of a camera having multiple tracking target subjects may be generated using an image based on the number of tracking target subjects. For example, for tracking Fig.26The camera of the three subjects (subject A to subject C) shown can generate an indicator indicating the shooting angle of view by scaling the image of the state in which the three subjects are arranged in a row. The same is true for other numbers of subjects.

[0347] Alternatively, in the case where multiple subjects are grouped and tracked as a group, an indicator may be generated by scaling an image of one person and then superimposing identification information (e.g., a group name) of subjects. In this case, a group of subjects may be assigned an indicator color different from that of individual subjects. Alternatively, an indicator containing all colors assigned to individual subjects may be used to indicate subjects belonging to the same group. For example, according to Fig.25 In the example shown, when subjects A to C belong to the same group, an indicator (eg, a rectangular frame) composed of three colors (ie, red, green, and blue) may be superimposed on each region of subjects A to C.

[0348] Furthermore, in the present embodiment, although a rectangular frame of a color associated with the subject is used in visually associating the tracking target subject and the shooting angle of view of the same camera with each other, other methods may be used. For example, for the same camera, it is allowed to use an image of a solid line, a dotted line, a dashed line, a double line, or the like connecting between the area of ​​the tracking target subject and the display area of ​​the indicator indicating the shooting angle of view. It is not limited to a rectangular frame, and an arrow indicating the subject area, or a frame of a figure (such as a circle and a trapezoid) surrounding the subject area may be used.

[0349] In addition, the icon of the corresponding camera may be displayed in association with the indicator indicating the shooting angle of view. In this way, for example, in the case where the cameras have different external shapes, it is possible to more clearly display which cameras the shooting angle of view corresponds to. For example, in the case where the main camera 500 is a horizontally long video camera and the sub-camera 400 is a circular PTZ camera, an icon imitating the appearance of the camera is displayed in association with the indicator indicating the shooting angle of view so that it is possible to recognize at a glance which camera the shooting angle of view corresponds to. In addition, the type of camera can also be understood through the icon.

[0350] In addition, character information may be used to present the shooting angle of view. For example, an image showing numeric characters indicating a zoom value or characters such as "far" (wide angle) and "near" (telephoto) may be used as an indicator indicating the shooting angle of view. Any other expression capable of indicating the shooting angle of view may be used.

[0351] In addition, in the present embodiment, the indicator indicating the tracking target subject and shooting angle of view of each camera in a visual manner has been exemplarily described as shooting information. However, the shooting information is not limited to this. For example, the counting status of each camera can be visualized. Specifically, the indicator of the tracking target subject and shooting angle of view of the camera whose counting status is currently being broadcast or currently being distributed can be displayed in association with a mark having the same color as the color of the count light and character information indicating "currently being broadcast" or "currently being distributed". In the case of using a switch to select a video to be output from the camera system, the counting status superimposed on the overhead image is displayed, and in addition to making it easy to understand the subject and angle of view of each camera, it is also possible to easily understand the subject and angle of view of the video currently being output.

[0352] As described above, according to the present embodiment, in an image capturing system including a sub-camera that automatically captures images in cooperation with a main camera, capturing information indicating the subject captured by each camera and its capturing angle of view can be understood in real time. This makes it easy to perform remote operations to switch the main camera and appropriately change the angle of view of each camera.

[0353] ·<Fourth embodiment>

[0354] Next, a fourth embodiment of the present invention will be described. The third embodiment was described using an example of presenting the photographing information of each camera to the user by superimposing the photographing information on the overhead image. In the fourth embodiment, the photographing information of each camera is presented to the user using a method different from the method of superimposing the photographing information on the overhead image.

[0355] and Figure 3 and Fig.23 similar, Fig.29 4 is a diagram showing a processing sequence performed when the photographing control device 100 controls the operation of the sub-camera 400, focusing on the main operations and signal flows. The functional blocks shown within the photographing control device 100 schematically indicate the main operations, and correspond to the main functions provided by the photographing control application. Fig.29 The functional blocks are composed of the CPU 101 executing the shooting control application and Fig. 22 The method may be implemented by combining one or more functional blocks of the photographing control device 100 shown in FIG.

[0356] pass Fig.29 and Fig.23As can be understood from the comparison between the two, the shooting control device 100 of this embodiment has the function of the GUI generation unit 129 in addition to the constituent elements according to the third embodiment. The GUI generation unit 129 generates a graphical user interface (GUI) as a second display image for presenting the shooting information of each camera to the user. By generating a GUI for presenting the shooting information to the user, a different representation method can be used from the first display image obtained by superimposing the shooting information on the overhead image. This makes it possible to provide other means for allowing the user to understand the shooting information of each camera.

[0357] Next, use Fig.30 The flowchart shown in FIG. 1 describes an operation of generating a GUI image for presenting photographing information of the main camera 500 and the sub-camera 400 among the operations performed by the photographing control device 100 of the present embodiment. The photographing control device 100 can perform the operations described below in parallel with the automatic photographing control operation of the sub-camera, the operations described in the third embodiment, and the like.

[0358] As an example, it is assumed here that a GUI image is generated that presents the cooperative operation content of the sub-camera that matches the set role as shooting information in addition to the tracking target subject and shooting angle of each camera. Although any type of GUI can be generated, it is assumed here that a window that presents shooting information is generated as an example.

[0359] In step S901, Fig.24A Similar to step S707 in FIG. 1 , the CPU 101 generates data of an overhead image on which a rectangular frame image having a color corresponding to the identification ID of the object is superimposed around the object area. The CPU 101 stores the generated image data in the RAM 102.

[0360] In step S902, similarly to step S708, the CPU 101 adds the image of the identification information of the camera to be superimposed near the subject area corresponding to the tracking target subject of the main camera 500 and the sub-camera 400 to the image data generated in step S901. Here, the images of the characters "main camera" and "sub-camera" are added as the images of the identification information of the camera. The CPU 101 stores the data of the overhead image (first display image) superimposed with the shooting information obtained in the processing of steps S901 and S902 in the RAM 102.

[0361] In step S903 , the CPU 101 obtains the role CAMERA_ROLE set to the sub-camera 400 , similarly to step S705 .

[0362] In step S904, the CPU 101 obtains the zoom value of each camera, similarly to step S802.

[0363] In step S905, the CPU 101 as the GUI generation unit 129 starts to generate a GUI image for presenting shooting information. First, the CPU 101 draws the area of ​​each camera in the image of a window having a predetermined size. Here, as an example, a rectangle corresponding to each camera is drawn in a color associated with the identification ID of the tracking target subject of each camera. Fig.31E An example of a window 310 and rectangular frames 311 and 312 corresponding to the respective cameras are shown. Fig.31E In Fig.31A As shown, the tracking target subjects of the main camera 500 and the sub-camera 400 are both the subject B. Therefore, the CPU 101 draws both the rectangular frame 311 corresponding to the main camera 500 and the rectangular frame 312 corresponding to the sub-camera 400 in green.

[0364] Furthermore, in order to indicate the correspondence between the area and the camera, the CPU 101 draws identification information 313 and 314 of the camera above the rectangular frames 311 and 312. Fig.31E , character information indicating “main camera” and “sub-camera” is drawn as an example, but as described above, other types of images such as icons in the external shapes of cameras, etc. may be drawn.

[0365] Furthermore, in the area corresponding to the sub-camera 400 , the CPU 101 draws information indicating control content related to the tracking target subject among control content corresponding to the role CAMERA_ROLE set to the sub-camera 400 .

[0366] exist Fig.31E In the example shown, the area corresponding to each camera (the area within the rectangle) is divided into an upper part and a lower part. Then, the control content and the information of the shooting angle related to the tracking target subject are drawn in the upper part and the lower part, respectively. Note that how to divide the inside of the area and what kind of shooting information is presented in the divided area can be appropriately determined according to the type and amount of shooting information to be provided.

[0367] exist Fig.31E In the example shown, the role "main follow" is set for the sub-camera. Therefore, in the upper part of the area corresponding to the sub-camera 400, the CPU 101 draws information 317 indicating that the tracking target subject of the sub-camera 400 is the same as the tracking target subject of the main camera. Fig.31E , as an example, character information indicating that “the tracking target subject is the same as the main camera” is drawn.

[0368] In step S906, the CPU 101 generates an indicator indicating the shooting angle of view of each camera, similar to step S804. Here, the indicator is generated using an image similar to the third embodiment; however, as described above, other types of images or character information may be used to generate the indicator.

[0369] In step S907, the CPU 101 draws the indicator generated in step S906 in the area corresponding to each camera. Fig.31E In the example shown, the indicator is drawn in the lower part of the area. Then, the CPU 101 stores the data of the GUI image (second display image) obtained by the processing of steps S905 to S907 into the RAM 102.

[0370] The CPU 101 sends the data of the overhead image (first display image) stored in the RAM 102 in step S902 and the data of the GUI image (second display image) stored in the RAM 102 in step S907 as display image data OUT_IMAGE to the display device 800. In this embodiment, the display image data OUT_IMAGE may also be displayed on the display device 108.

[0371] Upon receiving the display image data OUT_IMAGE via the network I / F 805, the CPU 801 of the display device 800 stores it in the RAM 802. Then, the CPU 801 displays the display image data OUT_IMAGE on the display unit 811. The CPU 801 may convert the display image data OUT_IMAGE into data suitable for display on the display device 811 as needed. For example, the CPU 801 may scale the overhead image and the GUI image so that the overhead image and the GUI image are displayed on the display unit 811 as separate windows.

[0372] Will use FIG. 31A to FIG. 31F An example of the display-use image data OUT_IMAGE generated by the CPU 101 in the present embodiment is described.

[0373] Fig.31A and Fig.31B The following state is shown: the main camera 500 and the sub-camera 400 in cooperation with the main camera 500 automatically capture and Fig.28A A similar scene. Fig.31A In the example, the secondary camera 400 is automatically shooting according to the role of "main follower". Fig.31BIn the figure, the sub-camera 400 is automatically shooting according to the role of “Sub-confrontation”. Note that it is assumed here that in the case of the role of “Sub-confrontation”, the subject located on the right side of the subjects other than the focus subject of the main camera 500 is set as the tracking target subject of the sub-camera 400.

[0374] In addition, it is assumed that the main camera 500 and the sub-camera 400 are Fig.31A On the other hand, assuming that Fig.31B The viewing angle of the main camera 500 is Fig.31A Therefore, according to the role of "assistant counterbalance", Fig.31A Compared with the state of FIG. 4 , the shooting angle of view of the sub-camera 400 is controlled to be wider (on the wide-angle side).

[0375] exist Fig.31A In FIG. 4 , both the main camera 500 and the sub-camera 400 are tracking the subject B. Fig.31B In FIG. 5 , the main camera 500 and the sub-camera 400 are tracking the subject A and the subject C, respectively.

[0376] Fig.31C shows the equivalent of Fig.31A An image of the overhead image data generated in step S902 in the shooting state of the main camera 500 and the sub-camera 400 is superimposed. A rectangular frame having a color associated with the identification ID of the subject is superimposed on the area of ​​each subject. In addition, an image indicating identification information (characters "main camera" and "sub-camera") of the main camera 500 and the sub-camera 400 is superimposed above the rectangular frame of the subject B which is the tracking target subject of the main camera 500 and the sub-camera 400.

[0377] Fig.31D shows the equivalent of Fig.31B An image of the overhead image data generated in step S902 under the shooting state of . A rectangular frame having a color associated with the identification ID of the subject is superimposed on the area of ​​each subject. In addition, an image of identification information (characters "main camera") indicating the main camera 500 is superimposed above the rectangular frame of subject A, which is the tracking target subject of the main camera 500. In addition, an image of identification information (characters "sub camera") indicating the sub camera 400 is superimposed above the rectangular frame of subject C, which is the tracking target subject of the sub camera 400.

[0378] Fig.31E shows the equivalent of Fig.31A310. A rectangular frame 311 indicating an area corresponding to the main camera 500 and a rectangular frame 312 indicating an area corresponding to the sub-camera 400 are drawn in the window 310. The rectangular frames 311 and 312 are drawn in a color associated with the identification ID of the tracking target subject of the corresponding camera. In addition, above the rectangular frames 311 and 312, identification information 313 and 314 of the corresponding camera are drawn. In the upper part of the rectangular frame 312 corresponding to the sub-camera 400, information 317 indicating the control content of tracking the target subject under the role currently set to the sub-camera 400 is drawn. In addition, in the lower parts of the rectangular frames 311 and 312, indicators 315 and 316 indicating the shooting angles of view of the main camera 500 and the sub-camera 400 are drawn. The indicators 315 and 316 have the same size.

[0379] Fig.31F shows the equivalent of Fig.31B The image of the GUI image data generated in step S907 under the shooting state. Fig.31E In comparison, the colors of the rectangular frames 311 and 312 are different because the tracking target subjects are different. In addition, since the shooting angle of the main camera 500 is narrow and the shooting angle of the sub-camera 400 is wide, the indicators 315 and 316 have different sizes. In addition, since the sub-camera 400 has a different role, the content of the information 317 indicating the control content of the tracking target subject is different.

[0380] Note that the color associated with the identification ID of the tracking target subject of the camera can be applied to items other than the rectangle indicating the area corresponding to the camera. For example, the color can be applied to the background within the area. In addition, two images can be displayed in the same window by, for example, superimposing a GUI image on an overhead image superimposed with shooting information.

[0381] In the present embodiment, the control content of tracking the target subject is presented as information related to the role of the sub-camera 400. However, the name of the role of the sub-camera 400 (e.g., "Sub-camera") and identification information of the tracking target subject (e.g., subject name) thereof may be presented instead of the control content or in addition to the control content.

[0382] According to the present embodiment, an image presenting photographic information is generated separately from the overhead image; this makes it possible to check the photographic information without being affected by the overhead image and to understand the photographic information more easily, in addition to achieving the advantageous effects of the third embodiment.

[0383] Note that, although the third and fourth embodiments are described using an example in which information visually indicating the shooting status of the main camera and the sub-camera is provided to the user, in the case where the main camera is operated by the photographer, changes in the subject and shooting angle of the main camera may be predetermined. In this case, since the subject and shooting angle of the main camera can be grasped to a certain extent, information visually indicating the shooting status of the sub-camera but not visually indicating the shooting status of the main camera may be provided to the user. Alternatively, the user may select whether to visualize the shooting status of the main camera.

[0384] Furthermore, in a multi-camera imaging system including a large number of secondary cameras, if information visually indicating the shooting status of each camera is provided to the user, there is a risk that the user cannot accurately recognize the information due to the large amount of information. In view of this, the user can select (one or more than one) cameras whose shooting status is to be visualized from among the cameras in the multi-camera imaging system. Note that providing the user with information visually indicating the shooting status of the main camera but not visually indicating the shooting status of the secondary camera has little effect on improving the convenience of the automatic imaging system; therefore, it is desirable to provide the user with information visually indicating the shooting status of at least one secondary camera. It is sufficient to allow the user input unit 804 to perform these selection operations.

[0385] Other embodiments

[0386] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0387] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A display control device used in a camera system, the camera system comprising a first camera device and a second camera device, wherein: In the camera system, based on information related to the first camera, a subject as a tracking target of the second camera and a viewing angle of the second camera are controlled, and the display control device includes: a generating unit configured to generate a display image different from the image captured by the first camera and the image captured by the second camera; and an output unit, configured to output the display image, The generating unit generates, as the display image, an image indicating a subject that is a tracking target of the second imaging device and a viewing angle of the second imaging device.

2. The display control device according to claim 1, wherein: The generating unit generates, as the display image, an image indicating a subject of interest of the first imaging device, the angle of view of the first imaging device, a subject that is a tracking target of the second imaging device, and the angle of view of the second imaging device.

3. The display control device according to claim 1, wherein: The generation unit generates an image as the display image by superimposing an indicator on an image captured by one of a plurality of imaging devices included in the imaging system, the indicator indicating a subject of interest of the first imaging device and a subject that is a tracking target of the second imaging device.

4. The display control device according to claim 1, wherein: The generating unit generates an image by superimposing an indicator on an image captured by a third camera included in the camera system, the indicator indicating a subject of interest of the first camera and a subject that is a tracking target of the second camera, the third camera capturing an entire capturing area of ​​the first camera and the second camera.

5. The display control device according to claim 4, wherein: In the image captured by the third camera capturing the entire capturing area, the indicator includes an indicator in the form of a rectangular frame including a region of the object of interest and an indicator in the form of a rectangular frame including a region of the object being tracked by the second camera.

6. The display control device according to claim 5, wherein: Each of the rectangular frame-shaped indicators has a color that is predetermined according to the subject.

7. The display control device according to claim 6, wherein: The display image includes an indicator indicating a viewing angle of the first camera device, and At least a portion of the indicator indicating the angle of view of the first camera has the same color as a color of an indicator in the form of a rectangular frame including a region of the object of interest.

8. The display control device according to claim 7, wherein: The indicator indicating the angle of view of the first camera includes identification information of the first camera.

9. The display control device according to claim 6, wherein: The display image includes an indicator indicating a viewing angle of the second camera device, and At least a part of the indicator indicating the angle of view of the second camera has the same color as a color of an indicator in the form of a rectangular frame including a region of the subject that is a tracking target of the second camera.

10. The display control device according to claim 9, wherein: The indicator indicating the angle of view of the second camera includes identification information of the second camera.

11. The display control device according to claim 1, wherein: When the subject of interest of the first camera device and the subject being tracked by the second camera device are different, the generating unit generates an image including an indicator indicating the subject of interest of the first camera device and an indicator indicating the subject being tracked by the second camera device, which are distinguishable from each other, as the display image.

12. The display control device according to claim 1, wherein: The indicator indicating the angle of view of the second camera device in the display image indicates the range of the angle of view by the size of the indicator.

13. The display control device according to claim 1, wherein: The display image includes information related to a role set for the second camera.

14. The display control device according to claim 13, wherein: The information related to the character includes one or more of a name of the character and control content of the second camera according to the character.

15. A camera system, comprising: a first camera device; a second camera device; a shooting control device for controlling a subject as a tracking target of the second camera device and a viewing angle of the second camera device based on information related to the first camera device; as well as A display control device according to any one of claims 1 to 14.

16. The camera system according to claim 15, further comprising: A display device displays the display image.

17. The camera system according to claim 15, further comprising: A third camera device is configured to capture the entire capture area of ​​the first camera device and the second camera device. The generation unit generates an image by superimposing an indicator on the image captured by the third camera, the indicator indicating the object of interest of the first camera and the object being a tracking target of the second camera.

18. A display control method implemented in a camera system, the camera system comprising a first camera device and a second camera device, wherein: In the camera system, a subject as a tracking target of the second camera and a viewing angle of the second camera are controlled based on information related to the first camera, and the display control method includes: generating a display image different from the image captured by the first camera and the image captured by the second camera; and outputting the display image, The generating includes generating an image indicating a subject that is a tracking target of the second camera device and a viewing angle of the second camera device as the display image.

19. A computer readable medium storing a program having instructions, wherein: When the instructions are executed by one or more processors in the imaging system, the one or more processors are caused to perform the display control method according to claim 18 .

20. A computer program product storing a program having instructions, wherein: When the instructions are executed by one or more processors in the imaging system, the one or more processors are caused to perform the display control method according to claim 18 .

Images