Imaging control apparatus, imaging control method, imaging system, and storage medium

Abstract

The invention provides an imaging control apparatus, an imaging control method, an imaging system, and a storage medium. The imaging control apparatus includes: at least one processor and / or circuit configured to act as the following: an acquisition unit configured to acquire information from a state of a main camera and / or from a video captured by the main camera among a plurality of cameras; and a control unit configured to control an operation of a sub-camera among the plurality of cameras based on a role assigned to the sub-camera and the information, the acquisition unit acquires, as the information, information on a viewing angle of the main camera, information on a distance between the main camera and a target subject of the main camera, and information on a distance between the sub-camera and a tracking target of the sub-camera, and wherein the information on the viewing angle of the main camera, the information on the distance between the sub-camera and the tracking target of the sub-camera, the control unit controls a viewing angle of the sub-camera based on the character assigned to the sub-camera and the information.

Description

Technical Field

[0001] This disclosure relates to camera control devices, camera control methods, camera systems, and computer-readable storage media. Background Technology

[0002] Japanese Patent Application Publication No. 2020-25248 discloses a camera system in which multiple cameras are divided into main cameras and secondary cameras, and the secondary cameras are controlled to capture images of the same subject as those captured by the main cameras. The camera system disclosed in Japanese Patent Application Publication No. 2020-25248 can automatically control the secondary cameras, thereby saving labor costs.

[0003] Assuming that the camera system disclosed in Japanese Patent Application Publication No. 2020-25248 can control the field of view of the secondary camera in cooperation with the zoom control of the main camera, however, even if the user intends to use the main camera and the secondary camera to photograph the subject from the same field of view, the secondary camera may still photograph the subject from an unexpected field of view. Summary of the Invention

[0004] This disclosure discloses a camera control device and a camera control method configured to enable the secondary camera to capture images from a desired angle when automatically controlling the secondary camera in cooperation with a main camera.

[0005] In one aspect of this disclosure, a camera control device is provided, comprising: an acquisition unit configured to acquire information from the state of a main camera or from video captured by the main camera among a plurality of cameras; and a control unit configured to control the operation of a secondary camera based on a role assigned to a secondary camera among the plurality of cameras and the information, wherein the acquisition unit acquires information about the viewpoint of the main camera, information about the distance between the main camera and a target subject of the main camera, and information about the distance between the secondary camera and a tracked target of the secondary camera as the information, and wherein the control unit controls the viewpoint of the secondary camera based on the role assigned to the secondary camera and the information.

[0006] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of embodiments is given by way of example. Attached Figure Description

[0007] Figure 1 This is a schematic diagram illustrating a camera system according to a first embodiment of the present disclosure.

[0008] Figure 2 This is a block diagram illustrating an example of the functional structure of a device constituting a camera system according to the first embodiment.

[0009] Figure 3 This is a diagram illustrating a camera control device according to an embodiment, focusing on the main operations and signal flow.

[0010] Figure 4 This is a diagram illustrating an example of the roles and control details that can be assigned to a secondary camera according to an embodiment.

[0011] Figure 5 This is a flowchart of the role determination process based on the first embodiment.

[0012] Figure 6A , Figure 6B , Figure 6C and Figure 6D This is a flowchart of the various operations of the camera system apparatus according to an embodiment.

[0013] Figure 7A and Figure 7B This is a diagram illustrating the coordinate transformation according to an embodiment.

[0014] Figure 8A and Figure 8B This is a diagram relating to subject detection and coordinate transformation according to an embodiment.

[0015] Figure 9A , Figure 9B and Figure 9C This is a schematic diagram illustrating the control of the operation of the secondary camera according to the first embodiment.

[0016] Figure 10A , Figure 10B and Figure 10C This is a schematic diagram illustrating another control for the operation of the secondary camera according to the first embodiment.

[0017] Figure 11 This is a diagram illustrating the calculation of the panning angle according to an embodiment.

[0018] Figure 12 This is a diagram illustrating the calculation of the pitch angle according to an embodiment.

[0019] Figure 13A , Figure 13B and Figure 13C This is a diagram illustrating an example of the mapping of zoom control values ​​between the main camera and the zoom camera according to the first embodiment.

[0020] Figure 14 This is a flowchart of the process for determining the control details corresponding to the role of the secondary camera according to the first embodiment.

[0021] Figure 15A and Figure 15B This is a schematic diagram illustrating the control corresponding to the role of the secondary camera according to the first embodiment.

[0022] Figure 16 This is a schematic diagram illustrating the control corresponding to the role of the secondary camera according to the first embodiment.

[0023] Figure 17 This is a schematic diagram illustrating the operation for tracking multiple subjects in a variation of the first embodiment.

[0024] Figure 18 This is a schematic diagram illustrating a camera system according to a second embodiment of the present disclosure.

[0025] Figure 19 This is a diagram illustrating an example of assignable roles and control details according to the second embodiment.

[0026] Figure 20A and Figure 20B This is a schematic diagram illustrating an example of target tracking control by a secondary camera according to a second embodiment.

[0027] Figure 21A and Figure 21B It is a diagram showing the positional relationship between the main camera, the secondary camera, and the subject.

[0028] Figure 22A and Figure 22B This is a diagram showing the perspectives of the main camera and the secondary camera. Detailed Implementation

[0029] Exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0030] It should be understood that the following embodiments do not limit the scope of this disclosure according to the claims. Although the embodiments describe multiple features, not all features are absolutely necessary, and multiple features can be freely combined. In the drawings, the same or similar parts are given the same reference numerals, and repeated descriptions are omitted.

[0031] First Embodiment

[0032] Overview of Multi-Camera System

[0033] Figure 1 This is a schematic diagram illustrating the construction of a multi-camera imaging system 10 (hereinafter referred to as the "camera system") according to this embodiment. The camera system 10 includes a plurality of cameras 300, 400, and 500, a camera control device 100, and a character control device 600. The plurality of cameras 300, 400, and 500, the camera control device 100, and the character control device 600 are connected to enable communication via a communication network 700.

[0034] The communication network 700 conforms to the IEEE 802.3 series, IEEE 802.11 series, or any known wired or wireless communication standards. Each of the multiple cameras 300, 400, and 500, the camera control device 100, and the character control device 600 includes a communication interface conforming to the standards of the communication network 700.

[0035] Camera 300, one of multiple cameras 300, 400, and 500, captures the entire predetermined imaging range. The imaging range is, for example, set in a photography studio, an area where the subject to be photographed can be present. Therefore, the image captured by camera 300 includes all subjects within the imaging range.

[0036] Camera 300 is designed to capture images of subjects to be photographed that are present within its field of view. Therefore, the orientation and viewing angle of camera 300 depend on its position and are substantially fixed during recording. Camera 300 preferably captures the entire field of view without being obstructed by subjects outside the field of view. For this purpose, camera 300 is positioned to overlook the entire field of view. Hereinafter, camera 300 is referred to as a bird's-eye view camera to distinguish it from other cameras 400 and 500, whose orientation and viewing angle are substantially not fixed. However, the mounting position of camera 300 is not limited to a position overlooking the field of view. The operation of the bird's-eye view camera 300 can be controlled by camera control device 100.

[0037] Cameras 400 and 500 are, for example, pan-tilt-zoom (PTZ) cameras and their operation can be controlled from external devices, including controlling the camera direction (pan and tilt angles) and the viewing angle (zoom). Here, the operation of camera 500 is controlled by the user of the camera system, and the operation of camera 400 is controlled by the camera control unit 100. Since the camera control unit 100 controls the operation of camera 400 based on the state of camera 500, camera 500 is referred to hereinafter as the main camera, and camera 400 as the secondary camera. Although only one secondary camera 400 is shown for ease of illustration and understanding, there may be two or more secondary cameras. The main camera 500 can be directly operated by the user. Cameras 400 and 500 can be configured such that the camera direction (pan and tilt angles) can be controlled by mounting the camera body on a camera platform. Cameras 400 and 500 can be configured such that interchangeable zoom lenses are mounted on the camera body.

[0038] In this embodiment, the character control device 600 can be operated by an operator. The camera control device 100 can also be operated by an operator (user), but the presence of an operator is not mandatory. The operator of the character control device 600 can also act as a user of the camera control device 100. Since the camera work of the bird's-eye view camera 300 and the secondary camera 400 is controlled by the camera control device 100, a cameraman is not required. The main camera 500 is operated by an operator or a cameraman. In this way, the fact that some devices do not require an operator or cameraman saves labor.

[0039] Although Figure 1 The diagram shows all signals transmitted via communication network 700; however, video signals and control signals can be transmitted using different methods, for example. For instance, multiple cameras 300, 400, and 500 can individually supply video signals directly to camera control unit 100 via cables. Cameras 300, 400, and 500, as well as camera control unit 100, include communication circuitry conforming to video signal standards. Examples of video signal standards include, but are not limited to, the Serial Digital Interface (SDI) standard and the High Definition Multimedia Interface (HD Multimedia Interface).

[0040] The camera control unit 100 detects a subject from the video signal received from the bird's-eye view camera 300. Based on the subject detection result, the state of the main camera 500, and the role assigned to the secondary camera 400, the camera control unit 100 determines the shooting direction and viewing angle of the secondary camera 400. The camera control unit 100 sends a control command to the secondary camera 400 including the determined shooting direction and viewing angle. By changing the role settings, the method used to determine the shooting direction and viewing angle of the secondary camera 400 can be changed, thereby increasing the flexibility in controlling the operation of the secondary camera 400.

[0041] Example of the functional construction of the device

[0042] Figure 2 It shows the composition Figure 1 The diagram illustrates an example of the functional configuration of a multi-camera imaging system 10. The configurations shown as functional blocks in the diagram can be implemented using integrated circuits (such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs)), discrete circuits, or a combination of memory and a processor that executes the program stored in the memory. A functional block can be implemented using multiple integrated circuit packages, or multiple functional blocks can be implemented using a single integrated circuit package. The same functional block can be implemented using different configurations depending on its operating environment or required performance.

[0043] Camera control device 100

[0044] First, an example of the functional configuration of the camera control device 100 will be described. The camera control device 100 can be a general-purpose computer such as a personal computer or a workstation. The camera control device 100 is configured such that the central processing unit (CPU) 101, random access memory (RAM) 102, read-only memory (ROM) 103, inference unit 104, network interface (I / F) 105, user input unit 106, and display unit 108 are interconnected via an internal bus 110.

[0045] CPU 101 is a microprocessor capable of executing programmed instructions. CPU 101 implements the functions of camera control device 100 (described later) by reading programs stored in ROM 103 into RAM 102 and executing them. CPU 101 can also implement the functions of camera control device 100 by executing camera control applications running on its operating system (OS).

[0046] RAM 102 is used to load programs to be executed by CPU 101 and to temporarily store data to be processed by CPU 101 or being processed by CPU 101. A portion of RAM 102 can also be used as video memory for display unit 108.

[0047] ROM 103 is a non-volatile memory that stores programs, user data, etc., to be executed by CPU 101 (OS and applications).

[0048] The inference unit 104 uses a machine learning model to perform subject region detection processing on images captured by the bird's-eye view camera 300. The inference unit 104 can be implemented using hardware circuitry capable of high-speed operation of the machine learning model, such as a graphics processing unit (GPU) or a neural network processing unit (NPU). Alternatively, the inference unit 104 can be implemented using reconfigurable logic circuitry such as an FPGA. The CPU 101 can implement the functionality of the inference unit 104 by executing a program.

[0049] The machine learning model can be a convolutional neural network (CNN) trained according to the type of subject to be detected. Here, the inference unit 104 detects human or face regions from the input image as subject regions. For each detected subject region, the inference unit 104 outputs the position, size, and detection reliability of the rectangular region enclosed by each subject region. Various types of machine learning models can be used to perform different types of subject region detection processing on the same input image. The inference unit 104 can also perform subject region detection processing using known methods without using a machine learning model. The inference unit 104 can detect subject regions using methods such as those utilizing local feature values ​​(e.g., Scale Invariant Feature Transform (SIFT) or Speed-Up Robust Feature Transform (SURF)), and those utilizing pattern matching.

[0050] Network I / F 105 is an interface for connecting camera control device 100 to communication network 700. Camera control device 100 (CPU 101) can communicate with external devices (such as bird's-eye view camera 300, secondary camera 400, main camera 500, and character control device 600) on communication network 700 via Network I / F 105. Camera control device 100 can also communicate with external devices via another communication interface (e.g., Universal Serial Bus (USB) or Bluetooth (not shown)).

[0051] The CPU 101 acquires the network addresses of each device (bird's-eye view camera 300, secondary camera 400, main camera 500, and character control device 600) on the communication network 700 at any given time and stores the network addresses in RAM 102 for communication. The CPU 101 also acquires information about each device (device type and name) at any given time (e.g., during the first communication) and stores this information in RAM 102. Therefore, the CPU 101 knows at least the identification information and type of the bird's-eye view camera 300, secondary camera 400, main camera 500, and character control device 600. Alternatively, the user can assign desired names to each device.

[0052] User input unit 106 is an input device such as a mouse, keyboard, or touchpad (not shown). Camera control device 100 receives instructions from the user via user input unit 106.

[0053] Display unit 108 is a display such as a liquid crystal display (LCD). Display unit 108 displays a graphical user interface (GUI) screen provided by an OS or camera control application.

[0054] Bird's-eye view camera 300

[0055] Next, an example of the functional construction of the bird's-eye view camera 300 will be described.

[0056] CPU 301 is a microprocessor capable of executing programmed instructions. For example, CPU 301 controls the operation of various functional blocks to implement the functions of the bird's-eye view camera 300, which will be described later, by reading programs stored in ROM 303 into RAM 302 and executing these programs.

[0057] RAM 302 is used to load programs to be executed by CPU 301 and to temporarily store data to be processed by CPU 301 or currently being processed by CPU 301. RAM 302 can also be used as a buffer for video signals acquired by a camera.

[0058] ROM 303 is a rewritable, non-volatile memory. ROM 303 stores programs to be executed by CPU 301, settings of the bird's-eye view camera 300, user data, etc. ROM 303 can also be used as a storage destination for video signals. ROM 303 may include internal memory and a removable memory card.

[0059] Image sensor 307 includes a camera optics system and an image sensor. The image sensor may be a known charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) color image sensor including, for example, a primary color Bayer array of color filters. The image sensor includes: a pixel array in which a plurality of pixels are arranged two-dimensionally; and peripheral circuitry for reading signals from each pixel. Each pixel accumulates a charge corresponding to the amount of incident light through photoelectric conversion. By reading signals from each pixel having voltages corresponding to the amount of charge accumulated during the exposure period, a set of pixel signals (analog image signals) representing an image of the subject formed on the imaging plane is obtained.

[0060] The image processing unit 306 generates signal or image data corresponding to the intended use by applying predetermined signal processing and image processing to the analog image signal output from the image sensor 307, and acquires and / or generates various information.

[0061] Examples of processing applied by the image processing unit 306 may include preprocessing, color interpolation processing, correction processing, detection processing, data transformation processing, evaluation value calculation processing, and special effects processing.

[0062] Preprocessing may include analog-to-digital (A / D) conversion, signal amplification, reference level adjustment, and defective pixel correction.

[0063] When the image sensor 307 is equipped with a color filter and compensates for the values ​​of color components not included in the individual pixel data constituting the image data, color interpolation processing is performed. Color interpolation processing is also known as demosaic processing.

[0064] Correction processing may include white balance adjustment, grayscale correction, correction of image degradation caused by optical aberrations of the camera optical system (image restoration), correction of the effects of peripheral light attenuation of the camera optical system, and color correction.

[0065] Data transformation processing can include region cropping (trimming), combining, scaling, encoding and decoding, and header information generation (data file generation). Data transformation processing can also include the generation of video signals to be output externally and the generation of video data to be recorded in ROM 308.

[0066] The evaluation value calculation process may include: generating the signal and evaluation value for autofocus detection (AF) and generating the evaluation value for automatic exposure control (AE). AF and AE are performed by CPU 301.

[0067] Special effects processing can include applying blur effects, changing color tones, and rewriting.

[0068] These processes are merely examples of the processes that can be applied by the image processing unit 306, and are not intended to limit the processes that can be applied by the image processing unit 306.

[0069] The image processing unit 306 outputs the acquired or generated information and data to the CPU 301 or RAM 302 depending on the application being used.

[0070] The types and settings of applications applied by the image processing unit 306 can be controlled by sending commands from the camera control device 100 to the bird's-eye view camera 300.

[0071] Network I / F 305 is an interface used to connect the bird's-eye view camera 300 to the communication network 700. The bird's-eye view camera 300 (CPU 301) can communicate with external devices (such as camera control unit 100, secondary camera 400, main camera 500, and character control unit 600) on the communication network 700 via Network I / F 305. The bird's-eye view camera 300 can also communicate via another communication interface (e.g., USB or...). (Not shown) Communicates with external devices.

[0072] Secondary camera 400

[0073] Next, an example of the functional construction of the secondary camera 400 will be described. It is assumed that the function blocks with the same names in the secondary camera 400 and the bird's-eye view camera 300 have the same function, so their descriptions are omitted.

[0074] As described above, the secondary camera 400 is a PTZ camera, and the direction and viewing angle of images captured from the outside can be controlled. To this end, the secondary camera 400 includes a drive unit 409 and a drive I / F 408 capable of panning, tilting, and zooming operations. The drive I / F 408 is the communication interface between the drive unit 409 and the CPU 401.

[0075] The drive unit 409 includes a pan / tilt mechanism that supports the secondary camera 400 to enable panning and tilting operations, a zoom mechanism that changes the viewing angle of the camera optical system, and a motor for driving these mechanisms. The zoom mechanism can use image scaling (zoom in / zoom out) performed by the image processing unit 406.

[0076] The drive unit 409 drives the motor in response to instructions received from the CPU 401 via the drive I / F 408 to adjust the optical axis direction and viewing angle of the camera optical system.

[0077] Main camera 500

[0078] Next, an example of the functional structure of the main camera 500 will be described. It is assumed that function blocks with the same names in the main camera 500 and the secondary camera 400 have the same function, so their descriptions are omitted. The main camera 500 is operated by the user. Here, the user sends commands via the communication network 700 to operate the main camera 500 remotely.

[0079] However, if the main camera 500 is not a PTZ camera, the main camera 500 can be directly operated by the user.

[0080] The camera control unit 100 (CPU 101) can obtain information about the shooting direction and viewing angle of the secondary camera 400 and the main camera 500 respectively from the secondary camera 400 and the main camera 500 via the network I / F 505. The shooting direction can be the respective pan and pitch angles of the drive units 409 and 509 relative to a predetermined reference direction at 0°. The reference direction can be the direction facing the shooting range.

[0081] Character control device 600

[0082] Next, an example of the functional structure of the character control device 600 will be described.

[0083] CPU 601 is a microprocessor capable of executing programmed instructions. For example, CPU 601 controls the operation of various function blocks to realize the function of the role control device 600 by reading role setting programs stored in ROM 603 into RAM 602 and executing these programs.

[0084] RAM 602 is used to load programs to be executed by CPU 601 and to temporarily store data to be processed by CPU 601 or currently being processed by CPU 601. A portion of RAM 602 can also be used as video memory for display unit 608.

[0085] ROM 603 is a rewritable non-volatile memory that stores programs to be executed by CPU 601, settings of role control device 600, user data, etc.

[0086] The user input unit 611 is an input device such as a button, dial, joystick, or touch panel. The character control device 600 receives instructions from the user regarding the character settings of the secondary camera 400 via the user input unit 611.

[0087] Network I / F 605 is an interface for connecting the character controller 600 to the communication network 700. The character controller 600 (CPU 601) can communicate with external devices (such as the bird's-eye view camera 300, the secondary camera 400, and the camera control unit 100) on the communication network 700 via Network I / F 605. The character controller 600 can also communicate with external devices via another communication interface (e.g., USB or Bluetooth (not shown)).

[0088] Display unit 608 is a display such as a liquid crystal display (LCD). Display unit 608 displays a GUI screen provided by an OS or role setting application.

[0089] The character control device 600 stores character setting information, for example, in ROM 603. The character setting information is information that associates the identification information of the secondary camera 400 with information indicating the assigned character. CPU 601 executes a character setting application to display a character setting screen on display unit 608. For example, the character setting screen displays the identification information of the secondary camera 400 (e.g., its network address or a name set by the user) and the name of the currently set character in association. The initial value of the currently set character may be a preset default character. The user can change the currently displayed character associated with the desired secondary camera 400 by operating user input unit 611.

[0090] When the CPU 601 detects a user operation indicating the completion of the setting operation (such as an operation on the OK button included in the character setting screen), it updates the character setting information stored in the ROM 603 according to the content of the character setting screen.

[0091] When the CPU 601 receives a role acquisition command via the network I / F 605, it reads the role setting information stored in the ROM 603 and sends the information to the source of the role acquisition command.

[0092] although Figure 1 and Figure 2 The character control device 600 is shown as a separate device, but the camera control application executed by the camera control device 100 can provide the same functionality as the character control device 600. Characters can be directly assigned to the secondary camera 400, and the camera control device 100 can acquire the assigned characters from the secondary camera 400.

[0093] The role that can be assigned to the secondary camera 400 is a predetermined role that indicates how to use the information obtained from the main camera 500 in controlling the operation of the secondary camera 400. Here, in one example, information about the main camera 500 is used to control the target tracking and zoom operations of the secondary camera 400.

[0094] Figure 4 This shows an example of the types of characters that can be assigned to the secondary camera 400 and the control details associated with each character. The control details for each character can be... Figure 4 The table shown is stored in the ROM 603 of the character control device 600 and the ROM 103 of the camera control device 100. Here, any one of "Master Follower", "Master Counter", "Auxiliary Follower", and "Auxiliary Counter" can be set as a character. If there are two or more secondary cameras 400, characters can be assigned to each secondary camera.

[0095] For the secondary camera 400, which plays a "master-follower" role, the camera control unit 100 (CPU 101) sets the same tracking target as the tracking target of the primary camera 500. Furthermore, when the primary camera 500 is zoomed in or out, the camera control unit 100 performs zoom control so that the size of the tracking target within the field of view of the secondary camera 400 is in phase with the size of the subject within the field of view of the primary camera 500. Here, "in phase" means that the direction of change of the size of the tracking target within the field of view (the direction in which the subject size increases or decreases) is the same. Conversely, "out of phase" means that the direction of change of the size of the tracking target within the field of view (the direction in which the subject size increases or decreases) is opposite.

[0096] Here, we will refer to Figures 9A to 9C The operation is described when the role assigned to the secondary camera 400 is "master follower". The camera control unit 100 controls the secondary camera 400, which has been assigned the "master follower" role, in order to track the target subject of the main camera 500.

[0097] Therefore, when the target subject of the main camera 500 is determined to be subject B, such as Figure 9AAs shown, CPU 101 identifies subject B as the tracking target of secondary camera 400. Subsequently, when it is determined that the target subject of primary camera 500 has changed to subject A, as... Figure 9B As shown, CPU 101 changes the tracking target of secondary camera 400 to subject A. Similarly, when it is determined that the target subject of main camera 500 has changed to subject C, as... Figure 9C As shown, CPU 101 changes the tracking target of secondary camera 400 to the subject C.

[0098] For the secondary camera 400, which plays the role of "master counter", the camera control unit 100 (CPU 101) sets the same tracking target as the tracking target of the master camera 500.

[0099] Furthermore, when the main camera 500 is zoomed in or out, the camera control device 100 performs zoom control so that the size of the tracked target within the field of view of the secondary camera 400 is inversely proportional to the size of the target subject within the field of view of the main camera 500. Therefore, when a zoom operation is performed on the main camera 500 to make the target subject appear larger, the camera control device 100 (CPU 101) performs zoom control on the secondary camera 400, which plays this role, to make the tracked target appear smaller. As used herein, "zoom in" refers to changing the zoom in the telephoto direction (i.e., towards the telephoto end), while "zoom out" refers to changing the zoom in the wide-angle direction (i.e., towards the wide-angle end). In the scaling control of the image processing unit 406, "zoom in" means decreasing the area cropped from the image and increasing the magnification of the cropped area relative to the magnification before the area was changed. In contrast, "zoom out" means increasing the area cropped from the image and decreasing the magnification of the cropped area relative to the magnification before the area was changed.

[0100] For the secondary camera 400, which plays a "tracking assist" role, the camera control unit 100 (CPU 101) sets a tracking target that is different from the tracking target of the main camera 500. Furthermore, when the main camera 500 is zoomed in or out, the camera control unit 100 performs zoom control so that the size of the tracking target within the field of view of the secondary camera 400 is equal to the size of the target subject within the field of view of the main camera 500.

[0101] Reference Figures 10A to 10C The operation is described when the role assigned to the secondary camera 400 is "assistant follow". The camera control device 100 controls the secondary camera 400, which has been assigned the "assistant follow" role, to track a subject on the left side that is different from the target subject of the main camera 500.

[0102] Therefore, as Figure 10AAs shown, when the target subject of the main camera 500 is determined to be subject B, the CPU 101 identifies subject A, which is on the left side of subjects A and C, as the tracking target of the secondary camera 400. Subsequently, as... Figure 10B As shown, when it is determined that the target subject of the main camera 500 has changed to subject A, the CPU 101 changes the tracking target of the secondary camera 400 to subject B on the left of subject B and subject C. Figure 10C As shown, when it is determined that the target subject of the main camera 500 has changed to subject C, the CPU 101 changes the tracking target of the secondary camera 400 to subject A on the left of subject A and subject B.

[0103] For the secondary camera 400, which acts as an "auxiliary counter," the camera control unit 100 (CPU 101) sets a tracking target that is different from the tracking target of the main camera 500. When the main camera 500 is zoomed in or out, the camera control unit 100 performs zoom control so that the size of the tracking target in the field of view of the secondary camera 400 is inversely proportional to the size of the target subject in the field of view of the main camera 500.

[0104] Here, for the secondary camera 400, which functions as both a "tracking assist" and a "tracking counter," the subject on the left side of the image that differs from the target subject of the main camera 500 is designated as the tracking target of the secondary camera 400. The tracking target of the secondary camera 400 can be set according to another condition. For example, a subject on the right, top, or bottom side of the image that differs from the target subject of the main camera 500 can be designated as the tracking target of the secondary camera 400. Alternatively, a subject located on the outermost or innermost side of the image that differs from the target subject of the main camera 500 can be designated as the tracking target of the secondary camera 400.

[0105] You can either perform only one of the tracking target settings and zoom control, or you can add another control.

[0106] The character setting information stored in the ROM 603 by the character control device 600 includes information (type name and number assigned to type) indicating the character associated with the identification information of the secondary camera 400. The CPU 101 of the camera control device 100 obtains the character setting information from the character control device 600 and performs operation control of the secondary camera 400 corresponding to the type of character assigned to the secondary camera 400.

[0107] By dynamically changing the role assigned to the secondary camera 400 using the role control device 600, the tracking target of the secondary camera 400 can be changed, enabling flexible automatic video recording.

[0108] When the role assigned to the secondary camera 400 changes, the role control device 600 can notify the external device (e.g., the camera control device 100) of the change. This allows the change in the assigned role to be immediately reflected in the operation control of the secondary camera 400.

[0109] Operating instructions for the device

[0110] The operation of the apparatus of the multi-camera imaging system will be described. Here, the camera control device 100 automatically controls the imaging operation of the secondary camera 400 based on the video captured by the bird's-eye view camera 300, the information obtained from the main camera 500, and the role assigned to the secondary camera 400.

[0111] Figure 3 This diagram illustrates a series of processes performed when the camera control device 100 controls the operation of the secondary camera 400, focusing on the main operations and signal flow. The functional blocks shown in the camera control device 100 schematically illustrate the main operations corresponding to the main functions provided by the camera control application. These operations are performed by the CPU 101, which executes the camera control application, and by… Figure 2 The camera control device 100 shown is implemented by one or more functional blocks. Figure 3 Functional blocks within.

[0112] Figure 5 This is a flowchart illustrating the operation of the CPU 101, which serves as the role determination unit 120.

[0113] Figures 6A to 6D This is a flowchart of the various operations of the camera control device 100, the bird's-eye view camera 300, the main camera 500, and the secondary camera 400.

[0114] The following description assumes that the three-dimensional coordinates of the viewpoint position and shooting direction (optical axis direction) of the bird's-eye view camera 300 are known by the camera control device 100. Known position information, such as the three-dimensional coordinates of the viewpoint positions of the secondary camera 400 and the main camera 500, and the coordinates of markers arranged within the shooting range, are pre-stored in the ROM 103 as the default position information REF_POSI. The coordinate system used for the position is pre-determined according to the type of position.

[0115] (Operation of Role Determination Unit 120)

[0116] First, refer to Figure 5 The flowchart shown is described below. Figure 3 The operation of the CPU 101, which serves as the character determination unit 120, is described below. The operation is achieved by executing the camera control application through the CPU 101.

[0117] Figure 5The operations shown in the flowchart can be performed at a timed period, but are not limited to, at least before the control of the secondary camera 400's camera operation begins. This operation is also performed when a notification is received from the character control device 600 via network I / F 105 that the character assignment of the secondary camera 400 has been changed.

[0118] In S101, the CPU 101, which serves as the character determination unit 120, obtains the character (character setting information) corresponding to the secondary camera 400 from the character control device 600. For example, the CPU 101 can obtain the character setting information from the character control device 600 by sending a character acquisition command to the character control device 600 via the network I / F 105. The CPU 101 stores the obtained character setting information in the RAM 102.

[0119] In S103, the CPU 101, based on the recognition information of the secondary camera 400 and referring to the role setting information stored in RAM 102, obtains the operation control details of the secondary camera 400. The CPU 101, acting as the role determination unit 120, sends the obtained operation control details (CAMERA_ROLE) to the tracking target determination unit 123. In practice, the CPU 101 stores the operation control details in a specific area of ​​RAM 102 and refers to these operation control details when acting as the tracking target determination unit 123.

[0120] In S104, the CPU 101, which serves as the character determination unit 120, sends the acquired operation control details (CAMERA_ROLE) to the zoom value calculation unit 125. In practice, the CPU 101 stores the operation control details in a specific area of ​​RAM 102 and refers to these operation control details when used as the zoom value calculation unit 125.

[0121] (Operation of camera control device 100)

[0122] Next, we will refer to Figure 3 and Figure 6A This describes the operation of the camera control device 100 controlling the secondary camera 400 to capture images. The operation described below corresponds to... Figure 3 The CPU 101 serves as the identification unit 121, the target subject determination unit 122, the tracking target determination unit 123, the pan / tilt value calculation unit 124, and the zoom value calculation unit 125. The operation described below is achieved by executing the camera control application through the CPU 101.

[0123] In S201, CPU 101 sends a camera instruction command to bird's-eye view camera 300 via network I / F 105 using a predetermined protocol. In response to the command, bird's-eye view camera 300 begins supplying video signals (moving image data) IMG to video input unit 107. CPU 101 begins storing the video signals received by video input unit 107 into RAM 102, and then executes S202.

[0124] In S202, CPU 101 acquires ANGLE information indicating the shooting direction from the main camera 500. Specifically, CPU 101 sends a shooting direction acquisition command to the main camera 500 via network I / F 105 using a predetermined protocol. In response to the shooting direction acquisition command, the CPU 501 of the main camera 500 sends ANGLE information indicating the current shooting direction of the main camera 500 to the camera control device 100. An example of ANGLE information could be the pan and tilt angles of the drive unit 509. CPU 101 stores the acquired ANGLE information in RAM 102.

[0125] In S203, the recognition unit 121 detects the subject from the image captured by the bird's-eye view camera 300 and performs the following processing to recognize the detected subject.

[0126] (1) Apply subject region detection processing to the input frame image and store the detection results.

[0127] (2) Convert the position information (image coordinates) of each detected subject area.

[0128] (3) Apply identification processing to each detected subject area to specify identification information (for new subjects, apply the information used for identification processing).

[0129] (4) For each detected subject area, the identification information ID[n] is stored in association with the location information POSITION[n].

[0130] The recognition unit 121 is mainly implemented by the CPU 101 and the inference unit 104. The CPU 101 reads a frame of video received from the bird's-eye view camera 300 from the RAM 102 and inputs the frame image into the inference unit 104.

[0131] The processing performed by the identification unit 121 will be described step by step below.

[0132] (1) First, the inference unit 104 inputs the frame image into the machine learning model and detects the subject region. The inference unit 104 stores the position, size, and detection reliability of each detected subject region output from the machine learning model as detection results in RAM 102. The position and size of each subject region can be any information that can specify the position and size of the rectangular region surrounding the subject region. Here, the center coordinates, width, and height of the lower edge of the rectangular region are used as the position and size of the subject region.

[0133] The inference unit 104 stores the detection results of the first frame image in RAM 102 in association with the subject identification information ID[n], where n is an integer indicating the index of the subject and takes a value from 1 to the total number of detected subject regions. The inference unit 104 also stores the subject regions detected from the first frame image as templates for identifying each subject in RAM 102 in association with the subject identification information ID[n]. If template matching is not used to identify each subject, then storing the template is unnecessary.

[0134] Figure 8A This shows the result of inference unit 104 on ... Figure 7A This is an example of the result of subject detection processing performed on video captured by the bird's-eye view camera 300. Here, the regions A to C of the subjects located within the camera's field of view 20 are detected, and the coordinates (foot coordinates) of the center of the lower edge of each rectangular region circumscribed by the subject region are output as its position.

[0135] If the mark is as follows Figure 7B As shown, the known positions arranged within the camera range 20 are used for coordinate transformation, as described later. Then, the CPU 101 detects the frame image ( Figure 7A The system includes labeled images and stores their positions in RAM 102. The detection of labeled images can be configured to be performed by the inference unit 104. Labeled images can be detected using any known method, such as pattern matching using a template of the labeled images. Labeled images can also be detected using a pre-stored machine learning model for label detection.

[0136] (2) Next, the coordinate transformation performed by the inference unit 104 will be described. Figure 7A The image is shown schematically by a bird's-eye view camera 300. Figure 7B The image range 20 is schematically shown as viewed from directly above the center. The inference unit 104 converts the position of the subject area in the coordinate system of the bird's-eye view camera 300 into the coordinate system (planar coordinate system) when viewed from directly above the center of the image range 20.

[0137] The coordinates are converted to values ​​in a planar coordinate system because this facilitates the calculation of the panning angle (the angle of movement in the horizontal plane) for the secondary camera 400 to capture a specific subject. Here, it is assumed that the secondary camera 400 is mounted such that the drive unit 409 performs panning operations in a horizontal plane parallel to the ground of the imaging range 20.

[0138] Various methods can be used for coordinate transformation; here, markers are placed at multiple locations on the ground within the camera range 20, and the coordinates are transformed from the bird's-eye view coordinate system to the planar coordinate system based on the marker positions in the image captured by the bird's-eye view camera 300. Coordinate transformation can also be performed without using markers, for example, by using the viewpoint position and camera direction of the bird's-eye view camera 300.

[0139] Coordinate transformation can be performed according to Equation 1, which uses the homography transformation matrix H.

[0140] Mathematical Formula 1

[0141]

[0142] In this context, x and y on the right are the horizontal and vertical coordinates in the bird's-eye view camera coordinate system, while X and Y on the left are the horizontal and vertical coordinates in the planar coordinate system.

[0143] The homography transformation matrix can be calculated by substituting the coordinates of the four markers detected from the image and the coordinates of the four markers (known) arranged in the camera range 20 into Equation 1 to solve a system of simultaneous equations. If the positional relationship between the camera range 20 and the bird's-eye view camera 300 is fixed, the homography transformation matrix H can be pre-calculated during test shooting and can be stored in, for example, ROM 103.

[0144] CPU 101 reads the positions of the subject area from RAM 102 in sequence and converts these positions into values ​​in a planar coordinate system. Figure 8B The following state is schematically shown: using Equation 1 and the homography transformation matrix H stored in ROM 103, from... Figure 8A The foot coordinates (x, y) of each subject region detected in the image captured by the bird's-eye view camera 300 are converted to coordinates (X, Y) in a planar coordinate system. In other words, Figure 8B The foot coordinates (XA, YA), foot coordinates (XB, YB), and foot coordinates (XC, YC) of subject A, subject B, and subject C are shown.

[0145] CPU 101 stores the foot coordinates of each subject obtained through coordinate transformation as POSITION[n] in RAM 102. CPU 101 also stores the main camera coordinates (XM, YM) and the secondary camera coordinates (XS, YS) as position information about the position of the main camera 500 and the secondary camera 400 in RAM 102, respectively.

[0146] (3) Next, the operation of the inference unit 104 in assigning the identification information ID[n] of each subject will be described. Here, template matching is used to identify the subject. The subject is identified from the processing result of the subject detection process starting from the second time. For the first processing result, the identification information ID[n] is newly assigned to the subject region.

[0147] The inference unit 104 specifies the identification information ID[n] of the detected subject region by using template matching of templates stored in RAM 102. Therefore, subjects within the camera range 20 are identified. The inference unit 104 calculates an evaluation value indicating the correlation between individual templates for each detected subject region. The inference unit 104 specifies the identification information ID[n] corresponding to the template with the highest correlation, equal to or higher than a predetermined threshold, as the identification information ID[n] of the subject region. The evaluation value can be any known value, such as the sum of absolute differences between pixel values.

[0148] The inference unit 104 assigns a new identification information ID[n] to the subject region that has no correlation with any template that is equal to or higher than a certain threshold, and adds the image of the subject region to the template.

[0149] The inference unit 104 can update existing templates using the subject regions detected in the last frame image, or delete templates that do not have subject regions with a correlation equal to or higher than a specific threshold within a predetermined time period. The inference unit 104 can also store templates corresponding to frequently occurring identification information IDs [n] in the ROM 103.

[0150] Subjects can be identified using methods other than template matching. For example, subject regions closest to at least one of the last detected location and size can be assigned the same identification information ID[n]. The location of a subject region in the current frame image can be predicted using methods such as Kalman filters based on position transitions in multiple past detections associated with the same identification information, and then the same identification information (ID) can be assigned to the subject region closest to the predicted location. Combinations of these methods can be used. By avoiding template matching, the accuracy of identifying different subjects with similar appearances can be improved.

[0151] (4) The inference unit 104 stores the specified identification information ID[n] in RAM 102 in association with the position (in the plane coordinate system) POSITION[n] of the corresponding subject area.

[0152] In processes (1) to (4), the processing other than subject detection can be performed by CPU 101 instead of inference unit 104.

[0153] Here, video captured by the bird's-eye view camera 300 is used to obtain identification information ID[n] and position POSITION[n] of the subject within the camera range 20. However, video captured by the secondary camera 400 can also be used. When multiple secondary cameras 400 are provided, the CPU 101 executes for each secondary camera 400. Figure 6A The operation is illustrated in the flowchart. The position of each subject area is output as a value in the coordinate system of each secondary camera 400. Therefore, the bird's-eye view camera 300 is not necessary; however, when using the bird's-eye view camera 300, the subject detection accuracy is considered to be higher.

[0154] Return to reference Figure 6A In S204, it is used as Figure 3 The CPU 101 of the target subject determination unit 122 determines the target subject as the tracking target of the main camera 500. The CPU 101 can determine the target subject of the main camera 500 from the subjects detected in S203 based on the shooting direction of the main camera 500 obtained in S202. The CPU 101 stores the identification information ID[n] corresponding to the subject area of ​​the target subject determined to be the main camera 500 in RAM 102 as the identification information MAIN_SUBJECT of the target subject.

[0155] For example, CPU 101 can identify the subject closest to the main camera 500 in the camera direction in the planar coordinate system as the target subject of the main camera 500. When multiple subjects exist that are at a distance equal to or less than a threshold from the camera direction of the main camera 500, the user can select the target subject from them.

[0156] When the user selects a target subject, the CPU 101 causes the display unit 108 or an external display device to display a frame image that has undergone subject detection processing in S202, along with an indicator indicating the shooting direction and an indicator indicating candidate subject areas for use as the target subject. The indicator for the subject area can be, for example,... Figure 8A The rectangle indicating the outer edge of the subject area, or another indicator, can be shown. The CPU 101 can cause the display unit 108 to display a message prompting the user to select a target subject in the image.

[0157] Users can select a subject area corresponding to a desired target subject by operating the user input unit 106 (input device). Although no particular restrictions are imposed on the selection method, selection can be performed by operations that specify the desired subject area via mouse or keyboard manipulation.

[0158] When CPU 101 detects a user operation to specify a subject area, it stores the identification information ID[n] corresponding to the specified subject area in RAM 102 as the identification information MAIN_SUBJECT of the target subject.

[0159] In S205, it is used as Figure 3 The CPU 101 of the target determination unit 123 in the middle obtains the control details CAMERA_ROLE corresponding to the role assigned to the secondary camera 400. Specifically, the CPU 101 reads the control details CAMERA_ROLE by referring to the target determination unit 123 in the middle. Figure 5 The described role determines the control details CAMERA_ROLE acquired and stored in RAM 102. When multiple secondary cameras 400 are provided, CPU 101 performs the processing in S205 to S207 for each secondary camera 400.

[0160] In S206, the CPU 101, which serves as the tracking target determination unit 123, determines the subject (tracking target) to be tracked and photographed by the secondary camera 400 based on the control details CAMERA_ROLE. The CPU 101 determines the tracking target of the secondary camera 400 based on the role of the tracking target included in the control details CAMERA_ROLE. Figure 4 ).

[0161] When the target being tracked by the secondary camera 400 is set to be the same as the target subject of the main camera 500, the CPU 101 will set the identification information MAIN_SUBJECT of the target subject determined in S203 to the identification information SUBJECT_ID of the target being tracked by the secondary camera 400.

[0162] When the tracking target of the secondary camera 400 is set to a subject on the left side that is different from the target subject of the main camera 500, the CPU 101 detects a subject region on the left side of the subject region other than the target subject detected in S203. The CPU 101 sets the identification information ID[n] corresponding to the detected subject region to the identification information SUBJECT_ID of the tracking target of the secondary camera 400.

[0163] The CPU 101 writes the identification information SUBJECT_ID of the detected tracking target into the RAM 102. When the tracking target can change depending on the secondary camera 400, the CPU 101 stores the identification information SUBJECT_ID of the tracking target in association with the identification information of the secondary camera 400.

[0164] When the tracked target has changed, the CPU 101 retains the information of the previously tracked target in RAM 102 instead of deleting it.

[0165] In S207, CPU 101 calculates the distance between the coordinate position of the target subject of the main camera 500 and the coordinate position where the main camera 500 is positioned.

[0166] In S208, CPU 101 calculates the distance between the coordinate position of the target tracked by the secondary camera 400 and the coordinate position where the secondary camera 400 is positioned.

[0167] The CPU 101 reads the foot coordinates POSITION[n] stored in RAM 102 to obtain the coordinates of each subject. In other words, the CPU 101 obtains... Figure 8A The foot coordinates (XA, YA), foot coordinates (XB, YB), and foot coordinates (XC, YC) of subject A, subject B, and subject C are given.

[0168] The CPU 101 reads the coordinates (XM, YM) of the main camera from the ROM 103 to obtain the location information of the main camera 500. Similarly, the CPU 101 reads the coordinates (XS, YS) of the secondary camera from the ROM 103 to obtain the location information of the secondary camera 400.

[0169] CPU 101 determines the distance DISTANCE_MAIN between the target subject of the main camera 500 and the main camera 500 based on the obtained position information of the main camera 500 and the foot coordinates of each subject. Similarly, CPU 101 determines the distance DISTANCE_SUB between the tracking target of the secondary camera 400 and the secondary camera 400 based on the obtained position information of the secondary camera 400 and the foot coordinates of each subject.

[0170] In S209, the CPU 101, which serves as the pan / tilt value calculation unit 124, calculates the changes in pan and tilt angles required for the secondary camera 400 to track and capture the target determined in S206. The CPU 101, which serves as the zoom value calculation unit 125, calculates the zoom control value of the secondary camera 400 based on the change in the viewing angle of the main camera 500.

[0171] When multiple sub-cameras 400 are provided, the change amounts of the pan angle and the tilt angle, and the zoom control value can be calculated for each sub-camera 400.

[0172] First, the operation of the CPU 101 serving as the pan / tilt value calculation unit 124 will be described. Here, the following information is pre-stored in the ROM 403 of each sub-camera 400 as the default position information REF_POSI.

[0173] · The three-dimensional coordinates (values in the planar coordinate system) of the installation position of the sub-camera 400.

[0174] · The imaging direction corresponding to the initial values of the pan angle and the tilt angle of the drive unit 409.

[0175] · The controllable range of the pan angle and the tilt angle

[0176] The CPU 101 reads the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking target of the sub-camera 400 from the RAM 102. The CPU 101 first determines the pan angle based on the position information POSITION_OH and the installation position of the sub-camera 400.

[0177] Figure 11 FIG. is an example showing the positional relationship between the sub-camera 400 and the tracking target in the planar coordinate system. Here, the pan angle θ at which the optical axis of the sub-camera 400 points to the subject position is determined. The CPU 101 calculates the pan angle θ using Equation 2.

[0178] Mathematical Equation 2

[0179]

[0180] where px and py are the horizontal coordinate and the vertical coordinate of the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking target, and subx and suby are the horizontal coordinate and the vertical coordinate of the installation position of the sub-camera 400, respectively. Here, the current pan angle is the initial value of 0°, and the current optical axis direction is the vertical direction (Y-axis direction). If the current optical axis direction is not the vertical direction, the angle difference between the current optical axis direction and the vertical direction is reflected in the angle calculated using Equation 2. When subx > px, the pan direction is counterclockwise; when subx < px, the pan direction is clockwise.

[0181] Next, referring to Figure 12 the method for determining the tilt angle will be described. Figure 12Shows the state of the sub-camera 400 and its tracking target when viewed from the side. Assume that the current optical axis of the sub-camera 400 is horizontally directed at height h1, and the height of the surface of the tracking target pointed to by the optical axis is h2. Assume that the height-direction angle difference (pitch angle) between the current optical axis direction and the target optical axis direction is ρ. The CPU 101 calculates the pitch angle ρ using the following Equation 3 and Equation 4:

[0182] Mathematical Equation 3

[0183]

[0184] The coordinates used in Equation 4 are the same as those used in Equation 2. The values h1 and h2 are pre-input to the imaging control application and stored in the RAM 102. In this case, the identification number associated with h2 of each subject is set to be the same as the identification number assigned in the subject detection process. Alternatively, h2 can be a value measured in real time using a sensor (not shown).

[0185] Assume that the current pitch angle is the initial value of 0°, and the current optical axis direction is horizontal (constant height). When the current optical axis direction is not horizontal, the angle difference between the current optical axis direction and the horizontal direction is reflected in the angle calculated using Equation 4. When h1 > h2, the pitch direction is downward, and when h1 < h2, the pitch direction is upward.

[0186] The CPU 101 periodically communicates with the sub-camera 400 via the communication network 700 to obtain the current optical axis direction (the pan angle and pitch angle of the drive unit 409), and stores it in the RAM 102. The communication cycle can be set to, for example, be equal to or less than the reciprocal of the frame rate. Alternatively, the CPU 101 can store the total values of the pan angle and pitch angle of the sub-camera 400 from the initial state obtained by control in the RAM 102, and can use this total value as the current optical axis direction.

[0187] In this way, the CPU 101 calculates the change amounts of the pan angle and pitch angle of the sub-camera 400, and stores the change amounts in the RAM 102. In the case where multiple sub-cameras 400 are provided, the CPU 101 calculates the change amounts of the pan angle and pitch angle for each sub-camera 400.

[0188] The changes in pan and pitch angles can be set as angular velocities to cause the secondary camera 400 to rotate toward the tracked target. For example, the CPU 101 acquires the current pan and pitch angles from the secondary camera 400 via the communication network 700. The CPU 101 calculates the difference between the pan angle θ read from RAM 102 and the current pan angle. The CPU 101 also calculates the difference between the pitch angle ρ read from RAM 102 and the current pitch angle. The CPU 101 stores the calculated changes in pan and pitch angles in RAM 102.

[0189] The changes in pan and pitch angles can be calculated using video captured by the secondary camera 400 instead of the bird's-eye view camera 300. In this case, the CPU 101 can calculate the change in pan angle based on the horizontal difference between the current optical axis direction and the direction toward the tracked target in the coordinate system of the secondary camera 400, and can calculate the change in pitch angle based on the vertical difference between them. Alternatively, another camera system can be used where the camera direction used for tracking and capturing the tracked target is changed only in one of the pan or pitch directions. In such a camera system, only the change in pan or pitch angle can be calculated.

[0190] Next, the operation of the CPU 101, which serves as the zoom value calculation unit 125, will be described. Figure 13A This is an example diagram indicating the range of zoom control values ​​for the main camera and the secondary camera. Here, the main camera 500 and the secondary camera 400 optically change the field of view (the camera optical system has zoom functionality). However, a similar function can be achieved using digital zoom from image processing units 406 and 506.

[0191] A zoom control value is a parameter with a value corresponding to the angle of view. In this embodiment, the smaller (narrower) the angle of view, the smaller the zoom control value. The zoom control value on the telephoto side is smaller than the zoom control value on the wide-angle side. By sending a command specifying the zoom control value, the secondary camera 400 and the main camera 500 can control their imaging optical systems to the angle of view corresponding to the zoom control value. In other words, the zoom control value is information about the angle of view and indicates the zoom state. The zoom control value can be, for example, the focal length (mm) of the imaging optical system corresponding to a full-frame image sensor with a 35mm focal length. In this case, the zoom control value on the telephoto side is greater than the zoom control value on the wide-angle side.

[0192] exist Figure 13AIn the diagram, the zoom control value MAIN_ZOOM for the main camera 500 ranges from main_min to main_max. The zoom range for the secondary camera 400 ranges from sub_min to sub_max. The values ​​main_min and sub_min correspond to the telephoto ends of the main camera 500 and secondary camera 400, respectively. The values ​​main_max and sub_max correspond to the wide-angle ends of the main camera 500 and secondary camera 400, respectively. Figures 13A to 13C This example shows an example where the zoom control value range of the main camera 500 is greater than that of the secondary camera 400 at both the telephoto and wide-angle ends.

[0193] The CPU 101, which serves as the zoom value calculation unit 125, calculates the zoom control value of the secondary camera 400 corresponding to the change in the viewing angle of the main camera 500. Here, the CPU 101 considers information about the state of the main camera 500 or the video captured by the main camera 500, the role assigned to the secondary camera 400, and information about the distance between the positions of the main camera 500 and the secondary camera 400 and the respective subjects to be photographed, to determine the viewing angle of the secondary camera 400.

[0194] First, CPU 101 periodically acquires MAIN_ZOOM information indicating the viewing angle of the main camera 500 and stores it in RAM 102. CPU 101 can determine the zoom operation and phase of the main camera 500, for example, by detecting changes in the viewing angle of the video captured by the main camera 500. For instance, CPU 101 can detect changes in viewing angle based on the temporal change in the size of the subject area or the distance between subject areas.

[0195] When the information MAIN_ZOOM has changed, CPU 101 calculates the zoom control value Z_VALUE of the secondary camera 400 based on the control details CAMERA_ROLE corresponding to the role assigned to the secondary camera 400.

[0196] Figure 21A and Figure 21B The positions of the main camera 500, the secondary camera 400, and subjects A to C are shown, as well as the relationship between the target subject of the main camera 500 and the tracking target of the secondary camera 400.

[0197] Figure 21A An example is shown where the target subject of the main camera 500 and the secondary camera 400 is subject A. Figure 21B An example is shown where the target subject of the main camera 500 and the secondary camera 400 is subject C.

[0198] In this embodiment, the following processing is performed to enable the user to take pictures from a desired angle in cooperation with the perspective of the main camera 500.

[0199] Specifically, the CPU 101 considers the distance between the coordinate positions of the main camera 500 and the secondary camera 400 and the coordinate positions of the target subject of the main camera 500 and the tracking target of the secondary camera 400 for control purposes. The CPU 101 processes the data so that the size of the target subject within the field of view of the main camera 500 is equal to the size of the tracking target within the field of view of the secondary camera 400.

[0200] First, we will describe the control of the secondary camera 400 when its role is "primary follower".

[0201] In other words, the CPU 101 controls the subjects captured by the main camera 500 and the subjects captured by the secondary camera 400 to change in size in the same phase (equally).

[0202] Reference Figure 22A and Figure 22B The perspective control according to this embodiment is described. Figure 22A It is shown Figure 21A The example image shows the view from the main camera 500. Figure 22B It is shown in Figure 21A In the example, the diagram of the view of the secondary camera 400 is used to make the size of the tracked target in the view of the secondary camera 400 equal to the size of the target subject in the view of the main camera 500.

[0203] like Figure 22A As shown, the field of view Θm of the main camera 500 (=the zoom control value MAIN_ZOOM / 2 of the main camera 500), and the distance DISTANCE_MAIN between the main camera 500 and the target subject of the main camera 500 calculated in S207 are known values. Figure 22B As shown, the distance DISTANCE_SUB between the secondary camera 400 and the target tracked by the secondary camera 400, calculated in S208, is also a known value. The viewing angle Θs of the secondary camera 400 can be calculated based on those known values ​​using the following calculation expression. Figure 22B The camera range VIEW_MAIN of the main camera 500 and the camera range VIEW_SUB of the secondary camera 400 shown can be calculated using Equations 5 and 6.

[0204] VIEW_MAIN=tan(Θm)×DISTANCE_MAIN×2···Equation 5

[0205] VIEW_SUB=tan(Θs)×DISTANCE_SUB×2···Equation 6

[0206] Here, to make the size of the target subject within the field of view of the main camera 500 equal to the size of the tracked target within the field of view of the secondary camera 400, the following control is required. In other words, control is performed so that the imaging range VIEW_MAIN of the main camera 500 becomes equal to the imaging range VIEW_SUB of the secondary camera 400. For this purpose, the viewing angle Θs of the secondary camera 400 is calculated using Equation 7.

[0207] Θs=tan-1(tan(Θm)×(DISTANCE_MAIN / DISTANCE_SUB))···Equation 7

[0208] The CPU 101 stores the value obtained by doubling Θs calculated in Equation 7 in RAM 102 as the zoom control value SUB_ZOOM for shooting from the perspective of the secondary camera 400.

[0209] Next, the control will be described when the secondary camera 400 acts as a "master counter". In other words, the CPU 101 controls the camera to change the size of the subject captured by the master camera 500 and the size of the subject captured by the secondary camera 400 in opposite phases. The "master counter" role needs to be controlled so that the size of the target being tracked by the secondary camera 400 within its field of view changes in opposite phases to the size of the target being tracked by the master camera 500. Figure 13B and 13C An image showing the size of the target within its field of view, tracked using phase-inverse control. Figure 13B This example illustrates the value of SUB_ZOOM on the wide-angle side before the size within the target's field of view is converted to its inversion. In this case, control is applied to convert SUB_ZOOM to its value on the telephoto side.

[0210] Specifically, when the role of the secondary camera 400 is "master counter", the CPU 101 calculates SUB_ZOOM by substituting the SUB_ZOOM calculated based on Equation 7 when the role is "master follower" into the right side of Equation 8.

[0211] SUB_ZOOM=SUB_MAX-(SUB_ZOOM-SUB_MIN)···Equation 8

[0212] Figure 13C This illustrates an example of the SUB_ZOOM value on the telephoto side before the size of the tracked target within its field of view is converted to the inversion value. Similarly, in this case, Equation 8 is used for control such that the SUB_ZOOM on the telephoto side is converted to the SUB_ZOOM value on the wide-angle side.

[0213] Although this embodiment illustrates the role of the secondary camera 400 as a "primary follower" or "primary counter" for tracking the same subject as the primary camera 500, similar operations can be performed for other roles. For example, even when the role of the secondary camera 400 is a "secondary follower" or "secondary counter" for tracking a subject different from the subject of the primary camera 500, similar operations can be performed for field-of-view control that controls the zoom value when only the tracked subject changes.

[0214] Here, when the main camera 500 controls its field of view by cropping using digital zoom, the CPU 101 can determine the zoom control value SUB_ZOOM of the secondary camera 400 based on the size of the area cropped by the main camera 500. Specifically, the CPU 101 sets the zoom control value SUB_ZOOM to a smaller value (corresponding to a higher magnification) as the size of the area cropped by the main camera 500 decreases, and sets the zoom control value SUB_ZOOM to a larger value (corresponding to a lower magnification) as the size increases.

[0215] Details of the zoom control associated with the role of the secondary camera 400 are not limited to in-phase or out-of-phase control relative to the main camera 500. For example, zoom operations that change the viewpoint independently of the main camera 500 can be associated with the role. For example, auto-zoom operations such as keeping the size of the tracked target constant can be associated with the role. The viewpoint of the secondary camera 400 can be fixed to a specific viewpoint. By adding the role associated with this zoom control... Figure 4 The control details for each character are shown, or can be changed. Figure 4 Details of the zoom control for the shown character, including various zoom controls for the secondary camera 400.

[0216] CPU 101 reads the changes in pan and pitch angles calculated in S209, as well as the zoom control value, from RAM 102. CPU 101 generates a control command PT_VALUE instructing the secondary camera 400 to change the pan and pitch angles corresponding to the changes. CPU 101 also generates a control command Z_VALUE instructing the secondary camera 400 to change the viewing angle corresponding to the zoom control value. The format of the control commands is predetermined. CPU 101 stores the generated control commands PT_VALUE and Z_VALUE in RAM 102. Step S209 can be skipped when it is not necessary to generate control commands, such as when the tracked target remains stationary or when the viewing angle of the main camera 500 does not change.

[0217] CPU 101 reads control commands PT_VALUE and Z_VALUE from RAM 102 and sends the values ​​to communication network 700 via network I / F 105. Secondary camera 400 receives control commands PT_VALUE and Z_VALUE via network I / F 405.

[0218] CPU 101 performs processing on the next frame of the video captured by the bird's-eye view camera 300, starting from S201. It is not necessarily necessary to perform processing on each frame. Figure 6A The processing shown.

[0219] Although the case where the viewpoints of the main camera 500 and the secondary camera 400 are the same has been described, this embodiment is not limited to this case. The viewpoints of the main camera 500 and the secondary camera 400 can be different from each other and can be controlled to their respective predetermined viewpoints.

[0220] For example, when the subject occupies a small proportion of the field of view of the main camera 500, the CPU 101 controls the secondary camera 400 to make the subject occupy a large proportion of the field of view.

[0221] Specifically, CPU 101 controls the secondary camera 400 so that the field of view VIEW_SUB is smaller than the field of view VEIW_MAIN of the main camera 500.

[0222] When the subject occupies a large proportion of the field of view of the main camera 500, the CPU 101 controls the secondary camera 400 to make the proportion of the field of view occupied by the subject smaller.

[0223] Specifically, CPU 101 controls the secondary camera 400 so that its field of view (VIEW_SUB) is greater than the field of view (VEIW_MAIN) of the primary camera 500. Through this control, the secondary camera 400 and the primary camera 500 can each capture images from predetermined different angles.

[0224] When multiple secondary cameras 400 are provided, the viewing angle of the secondary cameras 400 can be controlled using Equations 7 to 8 based on the distance between the coordinate positions of the main camera 500 and the multiple secondary cameras 400 and the target being tracked by the secondary cameras 400.

[0225] Operation of Bird's-eye View Camera 300

[0226] Next, we will refer to Figure 6B Describe the operation of the bird's-eye view camera 300. The following operations are implemented by executing a program via CPU 301.

[0227] When power is applied to the bird's-eye view camera 300, the various functional blocks are initialized by the CPU 301 and enter the camera standby state. In the camera standby state, the CPU 301 can begin processing moving images for real-time framing and output the image data generated by the image processing unit 306 for display to the camera control device 100 via the network I / F 305.

[0228] In camera standby mode, CPU 301 waits to receive control commands via network I / F 305. Upon receiving a control command, CPU 301 executes the operation corresponding to the control command. Here, the operation when a camera command is received as a control command from camera control device 100 will be described.

[0229] In S301, CPU 301 receives camera commands from camera control device 100 via network I / F 305.

[0230] The camera commands may include camera parameters such as frame rate and resolution. The camera commands may also include settings for processing applied by the image processing unit 306.

[0231] In S302, in response to receiving a camera command, the CPU 301 begins processing to capture a moving image to be provided to the camera control device 100. In this moving image capture process, a higher quality moving image is captured compared to a moving image captured via live view moving image capture processing. For example, at least one of the resolution and frame rate of the moving image is higher than that of the live view moving image. The image processing unit 306 applies processing to the image based on the settings of the moving image to be provided to the camera control device 100. The image processing unit 306 sequentially stores the generated moving image data in RAM 302.

[0232] In S303, CPU 101 reads motion image data from RAM 302 and sends the motion image data to camera control device 100 via network I / F 305. From then on, the process from camera recording to providing motion image data is repeated until a control command to stop camera recording is received.

[0233] Operation of the main camera 500

[0234] Next, we will refer to Figure 6C Describe the operation of the main camera 500. The operations described below are implemented by executing a program through CPU 501.

[0235] When power is supplied to the main camera 500, the CPU 501 activates the various function blocks and begins processing to capture moving images to be provided to the camera control device 100. The image processing unit 506 processes the analog image signal obtained from the image sensor 507 based on the settings of the moving images to be provided to the camera control device 100. The image processing unit 506 sequentially stores the generated moving image data in RAM 502. The CPU 501 reads the moving image data from RAM 502 and provides the moving image data to the camera control device 100 via network I / F 505.

[0236] While providing motion image data to the camera control device 100, the CPU 501 waits to receive control commands via the network I / F 505. Upon receiving a control command, the CPU 501 executes the operation corresponding to the control command. Here, the operation when a camera direction acquisition command has been received will be described. When a pan / tilt control command PT_VALUE or a zoom control command Z_VALUE is received, the CPU 501 operates the drive unit 509 according to the command.

[0237] In S501, CPU 501 receives a camera orientation acquisition command via network I / F 505. CPU 501 stores the received camera orientation acquisition command in RAM 502.

[0238] In S502, in response to receiving the camera direction acquisition command, the CPU 501 acquires the current pan and tilt angles from the drive unit 509 via the drive I / F 508, and stores the current pan and tilt angles in the RAM 502.

[0239] In S503, CPU 501 reads the current pan and tilt angles from RAM 502 and sends them as information about the camera direction (ANGLE) to camera control device 100 via network I / F 305.

[0240] Operation of the secondary camera 400

[0241] Next, we will refer to Figure 6D Describe the operation of the secondary camera 400. The CPU 401 executes a program to implement the operations described below.

[0242] When power is supplied to the secondary camera 400, the CPU 401 activates the various function blocks and begins processing to capture moving images to be provided to the camera control device 100. The image processing unit 406 processes the analog image signal obtained from the image sensor 407 based on the settings of the moving images to be provided to the camera control device 100. The image processing unit 406 sequentially stores the generated moving image data in RAM 402. The CPU 401 reads the moving image data from RAM 402 and provides the moving image data to the camera control device 100 via network I / F 405.

[0243] While providing motion image data to the camera control device 100, the CPU 401 waits to receive control commands via the network I / F 405. Upon receiving a control command, the CPU 401 executes the operation corresponding to the control command. Here, the operation when a pan / tilt control command PT_VALUE or a zoom control command Z_VALUE has been received from the camera control device 100 will be described.

[0244] In S401, CPU 401 receives at least one of pan / tilt control command PT_VALUE and zoom control command Z_VALUE from camera control device 100 via network I / F 405. CPU 401 stores the received control command in RAM 402.

[0245] In S402, CPU 401 reads the operation direction and corresponding operation value from the control commands stored in RAM 402, and stores these values ​​in RAM 402. In the case of the pan / tilt control command PT_VALUE, the operation direction is the panning and / or tilting direction, and the operation value is the target angle. In the case of the zoom control command Z_VALUE, the operation value is the zoom control value, and the operation direction can be specified based on the zoom control value; therefore, reading and storing the operation direction is not required.

[0246] In S403, CPU 401 generates drive parameters for drive unit 409 based on the operation direction and operation amount read in S403. CPU 401 can obtain drive parameters corresponding to the combination of operation direction and operation amount from a table pre-stored in ROM 403. When the operation amount is given as a target value (target angle or zoom control value), CPU 401 obtains drive parameters based on the difference between the target value and the current value.

[0247] In S404, CPU 401 controls drive unit 409 via drive I / F 408 based on the drive parameters acquired in S404. Therefore, drive unit 409 changes the imaging direction of secondary camera 400 to the operating direction and angle specified by the pan / tilt control command PT_VALUE. Drive unit 409 also changes the viewing angle of the camera optical system to the zoom control value specified by the zoom control command Z_VALUE.

[0248] Next, we will refer to Figure 14 The flowchart shown describes in more detail the operation of the camera control device 100 in controlling the camera direction (pan and tilt) and viewing angle (zoom control value) of the secondary camera 400 according to the role assigned to the secondary camera 400. Figure 14 The operations shown in the flowchart are as Figure 6A It is executed as part of the operations from S205 to S207.

[0249] Step S601 corresponds to step S205, wherein CPU 101 reads from... Figure 5 The control details CAMERA_ROLE are stored in RAM 102 in S103.

[0250] For example, steps S602 to S607 are performed in S206 to S209.

[0251] In S602, CPU 101 determines whether the designation of the tracking target of the secondary camera 400 included in the control details CAMERA_ROLE indicates the tracking target (target subject) of the primary camera 500. For example, if the designation of the tracking target of the secondary camera 400 has a value indicating "same as the primary camera", then CPU 101 determines that the designation of the tracking target of the secondary camera 400 indicates the tracking target of the primary camera 500 and executes step S603. Conversely, if the designation of the tracking target of the secondary camera 400 has a value indicating "different from the primary camera (left side)", then CPU 101 determines that the designation of the tracking target of the secondary camera 400 does not indicate the tracking target of the primary camera 500 and executes step S604.

[0252] In S603, CPU 101 determines the shooting direction of secondary camera 400 to track the target (target subject) of main camera 500.

[0253] In S604, CPU 101 determines the shooting direction of secondary camera 400 to track a subject on the left side that is different from the target subject of main camera 500.

[0254] In S605, CPU 101 determines whether the zoom control specification for the secondary camera 400 included in the control details CAMERA_ROLE indicates in-phase control of the subject size between the target subject of the main camera 500 and the tracked target of the secondary camera. For example, when the specification of the target size control for the secondary camera 400 has a value indicating "in-phase with the main camera", CPU 101 determines that the zoom control specification for the secondary camera 400 indicates in-phase control with the main camera 500 and executes step S606. Conversely, when the specification of the target size control for the secondary camera 400 has a value indicating "out-of-phase with the main camera", CPU 101 determines that the zoom control specification for the secondary camera 400 does not indicate in-phase control with the main camera 500 and executes step S607.

[0255] In S606, CPU 101 determines the zoom control value (angle of view) of the secondary camera 400 to change size in phase with the target being tracked by the main camera 500.

[0256] Reference Figure 15A and Figure 15B The description assigned to the secondary camera 400 is an example of "master follower". Figure 15A and Figure 15B This diagram schematically illustrates how the camera control unit 100 controls the camera direction and angle of the secondary camera 400 as the target subject and viewing angle of the main camera 500 change over time during recording. In the diagram, time is represented from left to right. Figure 15A and Figure 15B In the diagram, the zoom state is represented by three stages: "telephoto," "intermediate," and "wide-angle." This is because the zoom control value can vary between the secondary camera's 400 and the main camera's 500. Figures 13A to 13C As shown. "Telephoto" corresponds to the camera zoomed to the telephoto end, "wide-angle" corresponds to the camera zoomed to the wide-angle end, and "intermediate" corresponds to the zoom state between the telephoto and wide-angle ends; however, the actual zoom control value can differ between the secondary camera 400 and the main camera 500. For example, when the zoom state is "telephoto," the zoom control value for the main camera 500 is main_min, and the zoom control value for the secondary camera 400 is sub_min.

[0257] First, refer to Figure 15A Describe the process of controlling the zoom state while shooting the same subject B using the main camera 500 and the secondary camera 400.

[0258] First, the target subject (tracking target) of the main camera 500 is subject B, and its zoom state is "center". Therefore, the CPU 101 identifies subject B as the tracking target of the secondary camera 400 and controls the camera direction so that the secondary camera 400 tracks subject B. The CPU 101 also controls the secondary camera 400 so that the size of its tracking target becomes equal to the size of the target subject of the main camera 500. Figure 15A In this case, CPU 101 controls the zoom state to "intermediate".

[0259] Subsequently, while keeping subject B as the target of the main camera 500, the zoom state of the main camera 500 was changed from "intermediate" to "telephoto". As a result, the size of subject B within the field of view of the main camera 500 became larger than its size before the zoom state change.

[0260] In response, the CPU 101, while keeping the tracking target of the secondary camera 400 as subject B, controls the zoom state of the secondary camera 400 from "intermediate" to "telephoto". As a result, the size of subject B in the field of view of the secondary camera 400 becomes larger than its size before the zoom state change, and becomes equal to the size of subject B captured by the main camera 500.

[0261] Subsequently, while keeping subject B as the target of the main camera 500, the zoom state of the main camera 500 changes from "telephoto" to "wide-angle". In response, the CPU 101, while keeping subject B as the tracking target of the secondary camera 400, controls the zoom state of the secondary camera 400 from "telephoto" to "wide-angle". As a result, the size of subject B within the field of view of the secondary camera 400 becomes smaller than its size before the zoom state change, and equal to the size of subject B captured by the main camera 500.

[0262] This control allows the target subject of the main camera 500 and the tracked target of the secondary camera 400 to be photographed at the same size.

[0263] Next, refer to Figure 15B The following example will be described: While the main camera 500 and the secondary camera 400 are taking pictures of subject B, subject A and subject C in sequence, their zoom states are controlled so that the size of subject B, the size of subject A and the size of subject C become equal.

[0264] First, the target subject (tracking target) of the main camera 500 is subject B, and its zoom state is "center". Then, the CPU 101 identifies subject B as the tracking target of the secondary camera 400 and controls the shooting direction of the secondary camera 400 to track subject B. The CPU 101 also controls the secondary camera 400 so that the size of its tracking target becomes equal to the size of the target subject of the main camera 500. Figure 15B In this case, CPU 101 controls the zoom state to "intermediate".

[0265] Subsequently, the target subject of the main camera 500 changes from subject B to subject A. When the target subject changes from subject B to subject A, the zoom state of the main camera 500 changes from "center" to "wide-angle" to keep the size of the subject to be photographed unchanged.

[0266] In response, CPU 101 changes the tracking target of secondary camera 400 to subject A.

[0267] At this point, since the distance from the secondary camera 400 to subject A is greater than the distance to subject B, the size of the subject to be photographed in the current zoom state becomes smaller. Therefore, the CPU 101 controls the zoom state of the secondary camera 400 to change from "intermediate" to "telephoto" so that the size of the subject is equal to the size of the subject photographed by the main camera 500. As a result, by controlling the size of subject A within the field of view of the secondary camera 400 to be larger than its size before the zoom state change, the size of subject A photographed by the secondary camera 400 can be made equal to the size of subject A photographed by the main camera 500.

[0268] Next, the target subject of the main camera 500 changes from subject A to subject C. When the target subject changes from subject A to subject C, the zoom state of the main camera 500 changes from "wide-angle end" to "telephoto end" so that the size of the subject to be photographed remains unchanged.

[0269] In response, CPU 101 changes the tracking target of secondary camera 400 to subject C.

[0270] At this point, since the distance from the secondary camera 400 to the subject C is less than the distance to the subject C itself, the size of the subject to be photographed in the current zoom state becomes larger. Therefore, the CPU 101 controls the zoom state of the secondary camera 400 to change from "telephoto" to "wide-angle" so that the size of the subject is equal to the size of the subject photographed by the main camera 500. As a result, by controlling the size of the subject C within the field of view of the secondary camera 400 to be smaller than its size before the zoom state change, the size of the subject C photographed by the secondary camera 400 can be made equal to the size of the subject C photographed by the main camera 500.

[0271] This control allows the target subject of the main camera 500 and the tracked target of the secondary camera 400 to be photographed at the same size.

[0272] Therefore, when the secondary camera 400 is in the role of "master follower", the CPU 101 automatically changes the tracking target and zoom control value of the secondary camera 400 in order to follow the changes in the target subject and viewing angle (zoom control value) of the main camera 500.

[0273] exist Figure 15A In the example shown, the zoom state of the secondary camera 400 is controlled to change to the same extent as the target being tracked by the primary camera 500. This is just an example; the actual zoom control values ​​may differ, as long as the zoom control is performed so that the size changes in tandem with the target being tracked by the primary camera 500. For example, the zoom state of the secondary camera 400 does not need to be at the telephoto end even when the primary camera 500 is zoomed at the telephoto end. Role setting information can be used to configure whether the zoom control values ​​of the secondary camera 400 should match the zoom control values ​​of the primary camera 500.

[0274] exist Figure 15A and Figure 15B In the example shown, although the zoom state of the secondary camera 400 is controlled analogously to make the size of the subject captured by the main camera 500 equal to the size of the subject captured by the secondary camera 400, the zoom state can be controlled digitally. For example, it can be controlled so that the size of the captured subject is changed to a predetermined size, such as a full-body shot, a waist shot, or a chest shot. Such control allows for automatic control of the perspective based on composition. Character setting information can be used to configure whether the zoom state of the secondary camera 400 should match the zoom state of the main camera 500.

[0275] In S607, CPU 101 determines the zoom control value (angle of view) of the secondary camera 400, so that the size of the tracked target changes in opposite directions to the size of the tracked target of the main camera 500.

[0276] Reference Figure 16 This describes an example of secondary camera control when the role assigned to secondary camera 400 is "master counter". First, refer to... Figure 16 Describe the process of controlling the zoom state of the main camera 500 and the secondary camera 400 while shooting subject B.

[0277] First, the target subject (tracking target) of the main camera 500 is subject B, and the zoom state is "center". Therefore, the CPU 101 identifies subject B as the tracking target of the secondary camera 400 and controls the shooting direction of the secondary camera 400 to track subject B. The CPU 101 controls the size of the tracking target of the secondary camera 400 to change in opposite phase to the size of the tracking target of the main camera 500. Figure 16 In this case, since the zoom state of the main camera 500 is "intermediate", the CPU 101 controls the zoom state of the secondary camera 400 to "intermediate".

[0278] Subsequently, while keeping subject B as the target of the main camera 500, the zoom state of the main camera 500 was changed from "intermediate" to "telephoto". As a result, the size of subject B within the field of view of the main camera 500 became larger than its size before the zoom state change.

[0279] In response, CPU 101 controls the zoom state from "middle" to "wide-angle" to reverse the zoom state of the secondary camera 400 while keeping the tracking target of the secondary camera 400 as subject B. As a result, the size of subject B in the field of view of the secondary camera 400 becomes smaller than its size before the zoom state change, and subject B is photographed at a size different from that captured by the main camera 500.

[0280] Subsequently, while the target subject of the main camera 500 remains subject B, the zoom state of the main camera 500 changes from "telephoto" to "wide-angle". In response, the CPU 101 controls the zoom state of the secondary camera 400 from "wide-angle" to "telephoto", thus maintaining the tracking target of the secondary camera 400 as subject B while controlling the zoom state of the secondary camera 400 in reverse phase. As a result, the size of subject B within the field of view of the secondary camera 400 becomes larger than its size before the zoom state change, and subject B is captured at a size different from that captured by the main camera 500.

[0281] This control allows the target subject of the main camera 500 and the tracked target of the secondary camera 400 to be captured at different sizes. The subject size of the secondary camera 400 can be set relative to the subject size of the main camera 500 using character setting information.

[0282] Therefore, when the secondary camera 400 acts as a "master counter," the CPU 101 responds to changes in the target subject of the main camera 500 and automatically changes the tracking target of the secondary camera 400 based on the relationship between the target subjects of the secondary camera 400 and the main camera 500. The CPU 101 also automatically changes the zoom control value of the secondary camera 400 in the opposite direction to the change in the viewing angle (zoom control value) of the main camera 500.

[0283] exist Figure 16 In the example shown, CPU 101 controls the zoom state of the secondary camera 400 to change it to the same extent as the zoom state of the primary camera 500; however, the amount of change in the zoom control value can differ when the direction of change is opposite. For example, when the primary camera 500 is zoomed to the telephoto end, the secondary camera 400 does not necessarily have to be zoomed to the wide-angle end. The degree of change in the zoom state of the secondary camera 400 relative to the zoom state of the primary camera 500 can be configured using character setting information.

[0284] Although the embodiments illustrate the role of the secondary camera 400 as an example of "primary follower" and "primary counter" for capturing a subject identical to the subject of the primary camera 500, this disclosure is applicable to any other role. For example, when the role is "auxiliary follower," the secondary camera 400 tracks a subject different from the target subject of the primary camera 500, and its zoom state is controlled in tandem with respect to the subject size. In this case, the CPU 101 can... Figure 15B The zoom state is controlled in the same manner as described in the previous section. When the role is "auxiliary counter," the secondary camera 400 tracks a subject different from the target subject of the main camera 500, and its zoom state is controlled to be inverse relative to the subject size. In this case, the CPU 101 can control the zoom state in the same way as described in the previous section. Figure 16 The zoom state is controlled in the same way as described in the text.

[0285] Variation Example 1

[0286] So far, examples have been described of the automatic control of the secondary camera 400 to track the target and zoom control values ​​based on the target subject and zoom control values ​​of the main camera 500. In the examples described above, the secondary camera 400 is automatically controlled to track a single subject; however, the secondary camera 400 can also be automatically controlled to track multiple subjects within the field of view.

[0287] Reference Figure 17 This section describes examples of secondary camera 400 tracking a single subject and multiple subjects when its assigned role is "assistant tracking". For ease of understanding and illustration, the scenario where the main camera 500's viewpoint remains unchanged and only controls the tracked target will be described. The secondary camera 400's viewpoint allows it to always capture all subjects within the camera's field of view 20. Figure 17 The shooting direction of the secondary camera 400 shown at the top center indicates the shooting direction when tracking a single subject.

[0288] like Figure 16As shown, firstly, the target subject (tracking target) of the main camera 500 is subject B. Therefore, the CPU 101 identifies subject A on the left of subjects A and C (excluding subject B) as the tracking target of the secondary camera 400, and controls the camera direction of the secondary camera 400 to track subject A. When the camera direction is controlled so that the tracking target is in the center of the screen, as shown in the second row from the bottom, the image captured by the secondary camera 400 becomes unbalanced as subjects A to C move to the right. Therefore, when the image captured by the secondary camera 400 includes multiple subjects containing the tracking target, the camera direction can be controlled to track multiple subjects. For example, the CPU 101 can control the camera direction to track the center of gravity of multiple subjects A to C included in the image captured by the secondary camera 400. As a result, the secondary camera 400 can capture an image as shown in the bottom row.

[0289] exist Figure 17 In the example shown, even if the secondary camera 400 tracks any of the subjects A to C, subjects A to C are all photographed. Therefore, even if the target subject of the main camera 500 changes, the shooting direction of the secondary camera 400 remains basically constant.

[0290] Variation Example 2

[0291] After determining the target being tracked by the secondary camera 400, the CPU 101 can control the secondary camera 400 to focus on the target. Essentially, the CPU 401 continuously controls the focal length of the secondary camera 400 to focus on the designated target; however, the autofocus (AF) frame of the secondary camera 400 can be set to the position of the target via the camera control device 100. This allows the secondary camera 400 to focus on the target quickly and reliably. When the AF frame is set at a low panning speed (equal to or less than a threshold), the target is more likely to be centered on the screen, thus reducing the time required for focusing.

[0292] Variation Example 3

[0293] The secondary camera 400 can be controlled even without using the bird's-eye view camera 300. In this case, the shooting direction of the secondary camera 400 can be determined based on the installation positions of the main camera 500 and the secondary camera 400, as well as the shooting direction of the main camera 500 (the orientation of the main camera 500 relative to the tracked target). By utilizing the main camera 500 to perform subject detection processing, and by acquiring and using the results through the CPU 101, the secondary camera 400 can be controlled. For example, the CPU 101 acquires an image of the subject region from the main camera 500 as the result of the subject detection processing. The CPU 101 can then control the secondary camera 400 to perform subject tracking processing using the acquired image as a template. Alternatively, the CPU 101 can use the acquired image as a template and control the secondary camera 400 to track subject regions with low relevance to the template.

[0294] Variation Example 4

[0295] The camera control device 100 and the character control device 600 have been described as devices independent of the main camera 500, and the functions of the camera control device 100 and the character control device 600 can be integrated into the main camera 500. In this case, the video captured by the bird's-eye view camera 300 is provided to the main camera 500. This configuration allows for a reduction in the number of devices required to implement a multi-camera system.

[0296] As described above, according to this embodiment, the operation of the secondary camera is automatically controlled based on the role assigned to the secondary camera, the state of the primary camera, and the video captured by the primary camera. This enables the camera control device according to this embodiment to achieve more flexible automatic camera control while saving labor.

[0297] Second Embodiment

[0298] Next, a second embodiment of this disclosure will be described. In this embodiment, in addition to considering the state of the main camera or the video captured by the main camera and the role assigned to the secondary camera, the tracking target of the secondary camera is also determined based on information from the video captured by the secondary camera.

[0299] Figure 18 This is a schematic diagram illustrating an example of the construction of the camera system 10' according to this embodiment. Figure 18 In the process, components having the same structure as the camera system 10 according to the first embodiment are composed of... Figure 1 The same reference numerals are used in the accompanying drawings, and their descriptions will be omitted. The camera system 10' of this embodiment includes two secondary cameras, A800 and B900. Since the functional configuration of the secondary cameras A800 and B900 is the same as that of the secondary camera 400 described in the first embodiment, their descriptions will be omitted.

[0300] In this embodiment, a role is set from the role control device 600 to the camera control device 100, and the role setting information includes control details that differ for each secondary camera. The CPU 101 controls the operation of each secondary camera according to the role set in the camera control device 100.

[0301] Figure 19 This diagram illustrates an example of character setting information according to this embodiment. Here, control details corresponding only to the "assistant follow" character of each secondary camera are shown as an example. However, the types of characters that can be set in the camera control device 100 and the control details corresponding to those types are not limited to... Figure 19 The example shown.

[0302] Here, for the secondary camera A800, among subjects different from the tracking target of the main camera 500, the subject on the left is set as the tracking target, and control is applied to focus on the tracking target. For the secondary camera B 900, among subjects different from the tracking target of the main camera 500, the subject on the right is set as the tracking target, and control is applied to focus on the tracking target. In this case, since it is known in advance that the total number of subjects is three, "the subject on the left (or right) of the two subjects" is specified; however, it can be simply specified as "on the left (or right)". Although zoom control can be defined as in the first embodiment, zoom control will be omitted here for the sake of simplicity.

[0303] Figure 19 The character setting information shown is pre-stored in the ROM 103 of the camera control device 100. Alternatively, the character setting information can be provided to the camera control device 100 from the character control device 600, and the CPU 101 can store the character setting information in the RAM 102.

[0304] Reference Figure 20A and Figure 20B This describes how the camera control device 100 controls the secondary camera A800 and the secondary camera B900 based on the character setting information.

[0305] Figure 20A The image shows the positional relationship between the main camera 500, secondary camera A800 and secondary camera B900 within the camera range 20 at the start of the recording, and the subjects A to C. Figure 20B This shows the state after filming begins, where subject A moves from the left side to the right side of subject B, and subject C moves from the right side to the left side of subject B.

[0306] exist Figure 20A At any given moment, the CPU 101 of the camera control device 100 is based on Figure 19The role setting information shown controls the operation of secondary cameras A800 and B900. In other words, among subjects A and C other than the target subject (subject B) of the main camera 500, the CPU 101 will identify subject A on the left as the tracking target of secondary camera A800, and subject C on the right as the tracking target of secondary camera B900.

[0307] CPU 101 controls the camera orientation of secondary cameras A800 and B900 to track the identified target. CPU 101 controls secondary cameras A800 and B900 to focus on the identified target.

[0308] When reached Figure 20B In the state shown, subject A tracked by secondary camera A800 does not meet the condition "subject on the left" (other than the target subject of main camera 500). Therefore, CPU 101 changes the tracking target of secondary camera A800 to subject C on the left, among subjects A and C other than the target subject (subject B) of main camera 500. Similarly, subject C tracked by secondary camera B 900 does not meet the condition "subject on the right". Therefore, CPU 101 changes the tracking target of secondary camera B 900 to subject A on the right, among subjects A and C other than the target subject (subject B) of main camera 500. CPU 101 controls secondary cameras A800 and B 900 to focus on the determined tracking targets.

[0309] When the subject is like Figure 20B When significant movement is observed, there is a high probability that another subject is present in the foreground; therefore, the tracked target subject may be hidden. By setting the secondary camera positioned on the left relative to the camera range 20 to track a subject on the left, the tracking target of the secondary camera can be effectively changed when the tracked target has moved significantly to the right. This also applies to the secondary camera positioned on the right relative to the camera range 20.

[0310] In this embodiment, control details are defined such that secondary cameras assigned the same role track different subjects. This enables the camera control device 100 according to this embodiment to automatically control the secondary cameras based on information obtained from the state of the main camera and the video captured by the main camera, so as to capture video tracking various subjects.

[0311] Variation Example 1

[0312] In this embodiment, the shooting direction of the secondary camera used to track a specific subject is estimated by converting the position of the subject area detected from the video captured by the bird's-eye view camera 300 into coordinates. However, the same estimation can be performed based on the video captured by the secondary cameras A800 and B900. In this case, although the processing load on the camera control device 100 increases, no coordinate transformation is required, thereby improving the control accuracy of the shooting direction of the secondary cameras A800 and B900.

[0313] Variation Example 2

[0314] In this embodiment, the pan / pitch angles of the secondary cameras A 800 and B 900 are calculated to control the tracking target automatically. However, automatic tracking is not required. For example, character setting information can be defined such that the pan angles of the secondary cameras A 800 and B 900 are controlled based on the zoom control value of the main camera 500.

[0315] In one example, when the main camera 500 is zoomed to the "wide-angle" position, the secondary camera A 800 can be controlled to face 45 degrees to the left, and the secondary camera B 900 can be controlled to face 45 degrees to the right. Conversely, when the main camera 500 is zoomed to the "telephoto" position, both the secondary cameras A 800 and B 900 can be controlled to face 0 degrees to the center.

[0316] This control allows for the synchronized control of the camera orientation of multiple secondary cameras with the zoom state of the main camera 500, thereby enhancing the visual effect. For example, when the main camera 500 zooms in on a specific subject, secondary cameras A 800 and B 900 can also change their camera orientation while cooperating with the main camera 500 to zoom in on the same subject. Such control enables the simultaneous viewing of multiple images captured by multiple synchronously controlled cameras in a presentation of images captured by multiple cameras displayed on multiple monitors. Furthermore, compared to manually operating each camera, by assigning the same role to each camera and automatically controlling them, the variation in the field of view between cameras over time is reduced, resulting in a stronger sense of unity when the field of view changes, and providing a more realistic visual effect.

[0317] Variation Example 3

[0318] Although this embodiment illustrates an example where the main camera 500 is controlled as the main camera and the secondary cameras A800 and B900 are controlled as secondary cameras, the master-slave relationship between the main camera 500 and the secondary camera A800 can be dynamically changed. For example, among the main camera 500, secondary camera A800, and secondary camera B900, the camera capturing the main video can be controlled as the main camera, while the other cameras can be controlled as secondary cameras. In this case, the camera control device 100 only needs to obtain information from an external device, such as a switcher for selecting video, indicating which camera's video is selected as the main video, or determine this information based on a counting signal. Similar control can be performed not only when selecting the main video, but also when selecting videos related to the viewer's viewing or recording (such as videos used for recording).

[0319] Furthermore, when manually operating either the secondary camera A800 or the secondary camera B900, the master-slave relationship between the cameras can be switched. In this case, when the user manually operates one of the secondary cameras A800 and B900, the other secondary camera is controlled, thus reclassifying the operated camera as the master camera, thereby enhancing operability.

[0320] Other embodiments

[0321] The embodiments of this disclosure can also be implemented by one or more processors of a computer in a system or device that reads and executes computer-executable instructions recorded on a storage medium to perform one or more functions in the above embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions in the above embodiments.

[0322] According to embodiments of the present disclosure, a camera control device and a camera control method are provided, which are configured to enable the secondary camera to capture images from a desired angle when automatically controlling the imaging performed by the secondary camera in cooperation with a primary camera.

[0323] Although this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such variations and equivalent structures and functions.

Claims

1. A camera control apparatus comprising: at least one processor and / or circuitry configured to function as: an acquisition unit configured to acquire information from a state of a master camera and / or a video captured by the master camera among a plurality of cameras; and a control unit configured to control an operation of a sub camera among the plurality of cameras based on a role assigned to the sub camera and the information, wherein the acquisition unit acquires, as the information, information on a view angle of the master camera, information on a distance between the master camera and a target subject of the master camera, and information on a distance between the sub camera and a tracking target of the sub camera, and wherein the control unit controls a view angle of the sub camera based on the role assigned to the sub camera and the information. The control unit determines, as the tracking target of the sub camera, the target subject or a subject different from the target subject, in accordance with the role assigned to the sub camera.

2. The image pickup control device according to claim 1, wherein The control unit controls the sub camera to track a plurality of subjects including the tracking target.

3. The image pickup control device according to claim 2, wherein In a case where the sub camera includes a plurality of sub cameras, the control unit determines different tracking targets for the plurality of sub cameras.

4. The image pickup control device according to claim 1, wherein A master-slave relationship between the master camera and the sub camera is dynamically switchable.

5. The image pickup control device according to claim 1, wherein The master-slave relationship between the master camera and the sub camera is switched when the sub camera is manually operated.

6. The image pickup control device according to claim 5, wherein Among the plurality of cameras, a camera whose video is selected by an external device is used as the master camera.

7. The image pickup control device according to claim 5, wherein The control unit determines a camera direction of the sub camera based on a video captured by a camera different from the plurality of cameras, the camera being configured to capture an entire camera range of the plurality of cameras.

8. The image pickup control device according to claim 1, wherein The sub camera is capable of externally controlling a pan operation and a tilt operation.

9. The image pickup control device according to claim 1, wherein The control unit controls the view angle of the sub camera to be in phase or out of phase with a change in the view angle of the master camera, in accordance with the role assigned to the sub camera.

10. The image pickup control device according to claim 1, wherein 11.A camera system comprising: the camera control apparatus according to any one of claims 1 to 10; the plurality of cameras; and a communication network configured to communicably connect the camera control apparatus and the plurality of cameras. 12.A camera control method performed by a camera control apparatus, the method comprising: acquiring information from a state of a master camera and / or a video captured by the master camera among a plurality of cameras; and controlling an operation of a sub camera among the plurality of cameras based on a role assigned to the sub camera and the information, wherein the acquiring includes acquiring, as the information, information on a view angle of the master camera, information on a distance between the master camera and a target subject of the master camera, and information on a distance between the sub camera and a tracking target of the sub camera, and wherein the controlling includes: controlling a view angle of the sub camera based on the role assigned to the sub camera and the information. 13.A computer-readable storage medium storing a program for causing a computer to execute the camera control method according to claim 12. ​ ​ ​

Images