Image capture control device and its control method, and multi-camera imaging system

Abstract

To provide a shooting control device that can appropriately control the coordination of multiple cameras in a multi-camera imaging system. [Solution] This is a shooting control device that automatically controls at least one of the shooting direction and field of view of a sub-camera among a plurality of cameras including a main camera and one or more sub-cameras. The shooting control device selectively performs a first control that automatically controls at least one of the shooting direction and field of view of the sub-camera based on information about the main camera, based on the state or image of the main camera, and a second control that automatically controls at least one of the shooting direction and field of view of the sub-camera without being based on the state or image of the main camera.

Description

Technical Field

[0001] The present invention relates to a shooting control device, its control method, and a multi-camera imaging system, and particularly to a technique for automatically controlling the shooting operation of an imaging device.

Background Art

[0002] In live distribution and recording, a multi-camera imaging system is used in which video (main-line video) used for broadcasting or recording is dynamically selected or switched and output using a video switcher from videos captured in parallel by a plurality of cameras (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In such a multi-camera imaging system, for example, when attempting to automatically control the shooting operation of another camera (sub-camera) based on information obtained from one camera (main camera), problems may occur. Specifically, when the shooting direction or angle of view of the main camera changes while the video of the sub-camera is selected by the video switcher, if the shooting direction or angle of view of the sub-camera changes in conjunction with this change, the video being selected by the video switcher may change to an unintended one.

[0005] In view of such problems, in one aspect of the present invention, there is provided a shooting control device and its control method capable of appropriately controlling the interlocking of a plurality of cameras in a multi-camera imaging system.

Means for Solving the Problems

[0006] In one embodiment, the present invention provides a shooting control device for controlling the shooting operation of a sub-camera among a plurality of cameras including a main camera and one or more sub-cameras, wherein the device has a control means for automatically controlling at least one of the shooting direction and field of view of the sub-camera, and the control means selectively performs a first control that automatically controls at least one of the shooting direction and field of view of the sub-camera based on information about the main camera, based on the state or image of the main camera, and a second control that automatically controls at least one of the shooting direction and field of view of the sub-camera without being based on the state or image of the main camera. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an imaging control device and a control method that can appropriately control the coordination of multiple cameras in a multi-camera imaging system. [Brief explanation of the drawing]

[0008] [Figure 1] Schematic diagram of the imaging system according to the first embodiment [Figure 2] Block diagram showing an example of the functional configuration of each device in the imaging system according to the first embodiment. [Figure 3] This diagram shows the image capture control device according to the first embodiment, focusing on its main operations and signal flow. [Figure 4] This figure shows examples of roles and control contents that can be set for the sub-camera according to the first embodiment. [Figure 5] Flowchart relating to the role determination process according to the first embodiment [Figure 6] Flowchart relating to the operation of each device in the imaging system according to the first embodiment [Figure 7] A diagram illustrating the coordinate transformation according to the first embodiment. [Figure 8] Figure relating to subject detection and coordinate transformation according to the first embodiment. [Figure 9] Schematic diagram of the operation control of the sub-camera according to the first embodiment [Figure 10]Schematic diagram of another operation control of the sub-camera according to the first embodiment [Figure 11] Diagram for explaining the calculation of the pan value according to the first embodiment [Figure 12] Diagram for explaining the calculation of the tilt value according to the first embodiment [Figure 13] Diagram showing an example of mapping of zoom values between the main camera and the sub-camera according to the first embodiment [Figure 14] Flowchart regarding the characteristic operation of the imaging control device according to the first embodiment [Figure 15] Flowchart regarding the characteristic operation of the imaging control device according to the second embodiment [Figure 16] Timing chart showing the interlocking status of the imaging control device according to the second embodiment [Figure 17] Flowchart regarding the characteristic operation of the imaging control device according to the third embodiment [Figure 18] Schematic diagram for explaining the composition selection operation according to the third embodiment [Figure 19] Flowchart regarding the characteristic operation of the imaging control device according to the fourth embodiment [Figure 20] Timing chart showing the interlocking status of the imaging control device according to the fourth embodiment

Embodiments for Carrying Out the Invention

[0009] Hereinafter, the present invention will be described in detail based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Also, although a plurality of features are described in the embodiments, not all of them are essential to the invention, and a plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] ●<First Embodiment> (Overview of the multi-camera imaging system) FIG. 1 is a schematic diagram showing a configuration example of a multi-camera imaging system 10 (hereinafter simply referred to as an imaging system) according to the present embodiment. The imaging system 10 includes a plurality of cameras 300, 400a to 400c, 500, a shooting control device 100, a role control device 600, and a video switcher 1000 (hereinafter simply referred to as a switcher 1000). The plurality of cameras 300, 400a to 400c, 500, the shooting control device 100, the role control device 600, and the switcher 1000 are communicably connected to each other through a communication network 700.

[0011] The communication network 700 complies with known wired or wireless communication standards such as the IEEE802.3 series and the 1EEE802.11 series. In addition, each of the plurality of cameras 300, 400a to 400c, 500, the shooting control device 100, the role control device 600, and the switcher 1000 has a communication interface compliant with the standard of the communication network 700.

[0012] Among the plurality of cameras 300, 400a to 400c, 500, the camera 300 captures the entire predetermined shooting range. The shooting range is set, for example, as a range where a subject to be photographed may exist in a studio. Therefore, all the subjects within the shooting range are captured in the video of the camera 300.

[0013] The purpose of the camera 300 is to capture a video for detecting a subject (for example, a person) to be photographed within the shooting range. Therefore, the shooting direction and the angle of view of the camera 300 are determined according to the position of the camera 300 and the shooting range, and are basically fixed during shooting. In addition, the camera 300 preferably captures the entire shooting range so as not to be hidden by an object outside the shooting range. In order to distinguish it from other cameras 400a to 400c and 500 whose shooting direction and angle of view during shooting are not basically fixed, hereinafter, the camera 300 is referred to as an overhead camera. However, the installation position of the overhead camera 300 is not limited to a position overlooking the shooting range. The operation of the overhead camera 300 can be controlled from the shooting control device 100.

[0014] Cameras 400a-400c and 500 are, for example, PTZ cameras, and their operation, including shooting direction (pan and tilt angles) and field of view (zoom), can be controlled from an external device. Here, the operation of camera 500 is controlled by the user of the imaging system, and the operation of cameras 400a-400c is controlled by the shooting control device 100. In the following, since the shooting control device 100 controls the operation of cameras 400a-400c based on the state of camera 500, camera 500 will be referred to as the main camera, and cameras 400a-400c as sub-cameras. For ease of explanation and understanding, Figure 1 shows an example with three sub-cameras 400a-400c, but the number of sub-cameras can be one or more. Hereafter, sub-cameras 400a-400c will be collectively referred to as sub-camera 400, and matters and operations common to sub-cameras 400a-400c will be described as matters and operations of sub-camera 400.

[0015] The main camera 500 may be operated remotely by an operator or automatically from the shooting control device 100 or another remote control device, or there may be a photographer who directly operates the main camera 500. Furthermore, the sub-camera 400 and the main camera 500 may be configured so that the shooting direction (pan and tilt angle) can be controlled by mounting the camera body on a tripod head. Additionally, the sub-camera 400 and the main camera 500 may be configured so that a zoomable interchangeable lens is attached to the camera body.

[0016] In this embodiment, it is assumed that the role control device 600 and the switcher 1000 have operators (users). The shooting control device 100 may also have an operator (user), but this is not required. Furthermore, the same user may operate two or more of the role control device 600, the shooting control device 100, and the switcher 1000. Since the shooting control device 100 controls the shooting of the overhead camera 300 and the sub-camera 400, a photographer is not required. The main camera 500 is assumed to have an operator or photographer. In this way, labor savings can be achieved by configuring several devices without requiring operators or photographers.

[0017] Although Figure 1 shows all signals being communicated via the communication network 700, video signals and control signals may be communicated in different ways. For example, each of the multiple cameras 300, 400, and 500 may directly supply its video signal to the shooting control device 100 via a cable. The cameras 300, 400, 500 and the shooting control device 100 have communication circuits corresponding to the video signal standard. Video signal standards include, but are not limited to, the SDI (Serial Digital Interface) standard and HDMI (High-Definition Multimedia Interface) (registered trademark).

[0018] The shooting control device 100 detects a subject from the video signal received from the overhead camera 300. The shooting control device 100 also determines whether or not to automatically control the shooting direction and field of view in conjunction with the main camera 500 for each of the sub-cameras 400a to 400c. For sub-cameras 400 whose shooting direction and field of view are to be automatically controlled in conjunction with the main camera 500, the shooting control device 100 determines the shooting direction and field of view based on the assigned role, the subject detection result, and the status of the main camera 500. The shooting control device 100 transmits a control command including the determined shooting direction and field of view to the target sub-camera 400. By assigning a role to each of the sub-cameras 400a to 400c, the method for determining the shooting direction and field of view of each sub-camera 400 can be set individually, increasing the degree of freedom in controlling the operation of sub-cameras 400a to 400c.

[0019] (Examples of functional configurations for each device) Figure 2 is a block diagram showing an example of the functional configuration of each device constituting the imaging system 10 shown in Figure 1. However, sub-cameras 400a to 400c are assumed to have the same configuration, and only one sub-camera 400 is shown in Figure 2. The configurations represented as functional blocks in the drawings can be realized by integrated circuits such as ASICs and FPGAs, by discrete circuits, or by a combination of memory and a processor that executes the program stored in memory. Furthermore, one functional block may be realized by multiple integrated circuit packages, or multiple functional blocks may be realized by one integrated circuit package. In addition, the same functional block may be implemented in different configurations depending on the operating environment and required capabilities.

[0020] (Shooting control device 100) First, an example of the functional configuration of the imaging control device 100 will be described. The imaging control device 100 may be a general-purpose computer device such as a personal computer or a workstation. The imaging control device 100 has a configuration in which a CPU 101, RAM 102, ROM 103, inference unit 104, network interface (I / F) 105, user input unit 106, and display unit 108 are interconnected via an internal bus 110.

[0021] The CPU 101 is a microprocessor capable of executing programmed instructions. For example, the CPU 101 realizes the functions of the imaging control device 100, which will be described later, by reading a program stored in the ROM 103 into the RAM 102 and executing it. The CPU 101 can also realize the functions of the imaging control device 100 by executing an imaging control application that runs on the operating system (OS), for example.

[0022] RAM 102 is used to load programs executed by CPU 101, and to temporarily store data processed by CPU 101, data being processed, etc. A portion of RAM 102 may also be used as video memory for the display unit 108.

[0023] ROM103 is a rewritable, non-volatile memory that stores programs executed by CPU101 (OS and applications), user data, and other similar information.

[0024] The inference unit 104 performs object region detection processing using a machine learning model on the video feed from the overhead camera 300. The inference unit 104 can be implemented using hardware circuits capable of high-speed execution of machine learning model calculations, such as a GPU (Graphics Processing Unit) or an NPU (Neural Network Processing Unit). Alternatively, the inference unit 104 may be implemented using reconfigurable logic circuits such as an FPGA (Field-Programmable Gate Array). The CPU 101 may execute a program to implement the functions of the inference unit 104.

[0025] The machine learning model may be a convolutional neural network (CNN) trained according to the type of subject to be detected. Here, the inference unit 104 detects human body regions or human face regions as subject regions from the input image. The inference unit 104 also outputs the position and size of the rectangular region inscribed in the subject region, as well as the detection confidence level, for each detected subject region. Multiple machine learning models may be used to perform detection processing of different types of subject regions on the same input image. The inference unit 104 may also perform subject region detection processing using known methods that do not use machine learning models. For example, the inference unit 104 can detect subject regions using methods that utilize local features such as SIFT or SURF, or methods that utilize pattern matching.

[0026] The network interface 105 is an interface for connecting the image capture control device 100 to the communication network 700. The image capture control device 100 (CPU 101) can communicate with external devices on the communication network 700, such as the overhead camera 300, sub-camera 400, main camera 500, role control device 600, and switcher 1000, via the network interface 105. The image capture control device 100 may also communicate with external devices via other communication interfaces (not shown, such as USB or Bluetooth®).

[0027] The CPU 101 communicates with each device on the communication network 700 (overhead camera 300, sub-camera 400, main camera 500, role control device 600, switcher 1000). Therefore, it obtains the network address of each device at any time and stores it in RAM 102. The CPU 101 also obtains information about each device (device type, model name, etc.) at any time (for example, during the initial communication) and stores it in RAM 102. Thus, the CPU 101 is assumed to know at least the identification information and device type of the overhead camera 300, sub-camera 400, main camera 500, role control device 600, and switcher 1000. The user may be allowed to assign arbitrary names to each device.

[0028] The user input unit 106 is an input device (not shown) such as a mouse, keyboard, or touch panel. The shooting control device 100 receives user instructions through the user input unit 106.

[0029] The display unit 108 is a display device such as a liquid crystal display (LCD). The display unit 108 displays a GUI screen provided by the OS or shooting control application.

[0030] (Overhead camera 300) Next, we will explain an example of the functional configuration of the overhead camera 300. The CPU 301 is a microprocessor capable of executing programmed instructions. For example, the CPU 301 controls the operation of each functional block by loading a program stored in the ROM 303 into the RAM 302 and executing it, thereby realizing the functions of the overhead camera 300 described later.

[0031] RAM302 is used to load programs executed by CPU301, and to temporarily store data processed by CPU301, data being processed, etc. RAM302 may also be used as a buffer for video signals obtained during capture.

[0032] ROM308 is a rewritable, non-volatile memory. ROM308 stores programs executed by the CPU301, settings for the overhead camera 300, user data, and other similar information. ROM308 can also be used as a recording destination for video signals. ROM308 may include both internal memory and a removable memory card.

[0033] The image sensor 307 comprises an imaging optical system and an image sensor. The image sensor may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The image sensor has a pixel array in which multiple pixels are arranged in two dimensions, and peripheral circuits for reading signals from each pixel. Each pixel accumulates charge according to the amount of incident light by photoelectric conversion. By reading signals with a voltage corresponding to the amount of charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface is obtained.

[0034] The image processing unit 306 applies predetermined signal processing and image processing to the analog image signal output by the image sensor 307 to generate signals and image data according to the application, and to acquire and / or generate various types of information.

[0035] The processing applied by the image processing unit 306 may include, for example, preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effects processing. Preprocessing may include A / D conversion, signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is performed when a color filter is provided on the image sensor 307, and is a process that interpolates the values ​​of color components not included in the individual pixel data that make up the image data. Color interpolation is also called demosaicing. Correction processing may include white balance adjustment, gradation correction, correction of image degradation caused by optical aberrations of the imaging optical system (image recovery), correction of the effect of peripheral vignetting of the imaging optical system, and color correction. Data processing may include region cropping, synthesis, scaling, encoding and decoding, and header information generation (data file generation). The generation of video signals to be output externally and video data to be recorded in the ROM 308 is also included in data processing. The evaluation value calculation process may include generating signals and evaluation values ​​used for autofocus detection (AF), and generating evaluation values ​​used for automatic exposure control (AE). AF and AE are executed by the CPU 301. Special effect processing may include adding bokeh effects, changing color tones, and relighting. These are examples of processes that the image processing unit 306 can apply, and do not limit the processes that the image processing unit 306 can apply. The image processing unit 306 outputs the acquired or generated information and data to the CPU 301, RAM 302, etc., depending on the application.

[0036] Furthermore, the type and settings of processing applied by the image processing unit 306 can be controlled by sending commands from the shooting control device 100 to the overhead camera 300.

[0037] The network interface 305 is an interface for connecting the overhead camera 300 to the communication network 700. The overhead camera 300 (CPU 301) can communicate with external devices on the communication network 700, such as the shooting control device 100, sub-camera 400, main camera 500, and role control device 600, via the network interface 305. The overhead camera 300 may also communicate with external devices via other communication interfaces (not shown, such as USB or Bluetooth).

[0038] (Sub-camera 400) Next, we will explain an example of the functional configuration of the sub-camera 400. Functional blocks with the same name in the sub-camera 400 and the overhead camera 300 are assumed to have the same function, and their explanation will be omitted.

[0039] As described above, the sub-camera 400 is a PTZ camera, and its shooting direction and field of view can be controlled externally. Therefore, the sub-camera 400 has a drive unit 409 that can perform pan and tilt movements and zoom movements, and a drive I / F 408. The drive I / F 408 is a communication interface between the drive unit 409 and the CPU 401.

[0040] The drive unit 409 includes a pan / tilt mechanism that supports the sub-camera 400 so that it can be panned and tilted, a zoom mechanism that changes the field of view of the imaging optical system, and motors that drive these mechanisms. The zoom mechanism may use image enlargement and reduction by the image processing unit 406. The drive unit 409 drives the motors according to instructions received from the CPU 401 via the drive I / F 408 to adjust the optical axis direction (shooting direction) and field of view of the imaging optical system.

[0041] (Main camera 500) Next, we will describe an example of the functional configuration of the main camera 500. Functional blocks with the same name in the main camera 500 and the sub-camera 400 are assumed to have the same function, and their explanation will be omitted. The main camera 500 is operated by the user. Here, it is assumed that the user remotely operates the main camera 500 by sending commands through the communication network 700. However, if the main camera 500 is not a PTZ camera, the photographer may operate the main camera 500 directly.

[0042] The shooting control device 100 (CPU 101) can acquire information on the shooting direction and field of view of the sub-camera 400 and the main camera 500 from the sub-camera 400 and the main camera 500 via the network I / F 505. The shooting direction may be the pan and tilt angles of the drive units 409 and 509, with a predetermined reference direction set at 0°. The reference direction may be the direction directly facing the shooting range.

[0043] The sub-camera 400 and the main camera 500 are equipped with light-emitting elements (lamps, LEDs, etc.) for tally display. The tally display indicates whether the captured video is being streamed or previewed. The tally display may, for example, use a red light to indicate streaming and a green light to indicate preview, but is not limited to these. The tally display can be performed by the camera's CPU 401 or 501 based on the camera status information STREAMING supplied from the switcher 1000. Thus, the camera status information STREAMING is used as a control signal for the tally display. Further details will be described later.

[0044] (Role control device 600) Next, an example of the functional configuration of the role control device 600 will be described. The CPU 601 is a microprocessor capable of executing programmed instructions. For example, the CPU 601 controls the operation of each functional block and realizes the functions of the role control device 600 by reading the role setting program stored in the ROM 603 into the RAM 602 and executing it.

[0045] RAM 602 is used to load programs executed by CPU 601, and to temporarily store data processed by CPU 601, data being processed, etc. A portion of RAM 602 may also be used as video memory for the display unit 608.

[0046] ROM603 is a rewritable non-volatile memory that stores programs executed by CPU601, settings of role control unit600, user data, and other similar information.

[0047] The user input unit 611 is an input device such as a button, dial, joystick, or touch panel. The role control device 600 receives user instructions regarding the setting of the role of the sub-camera 400 through the user input unit 611.

[0048] The network interface 605 is an interface for connecting the role control device 600 to the communication network 700. The role control device 600 (CPU 601) can communicate with external devices on the communication network 700, such as the overhead camera 300, sub-camera 400, and shooting control device 100, via the network interface 605. The role control device 600 may also communicate with external devices via other communication interfaces (not shown, such as USB or Bluetooth).

[0049] The display unit 608 is a display device such as a liquid crystal display (LCD). The display unit 608 displays a GUI screen provided by the OS or a role setting application.

[0050] The role control device 600 stores role setting information, for example, in the ROM 603. The role setting information is information that associates the identification information of the sub-camera 400 with information indicating the role it has been assigned. The CPU 601 displays the role setting screen on the display unit 608 by executing a role setting application. The role setting screen displays, for example, the identification information of the sub-camera 400 (network address, user-set name, etc.) and the name of the currently assigned role, in association with each other. The initial value of the currently assigned role may be a pre-set default role. The user can change the current role displayed in association with the desired sub-camera 400 by operating the user input unit 611.

[0051] When the CPU 601 detects a user action indicating the completion of a setting operation, such as pressing the OK button on the role setting screen, it updates the role setting information stored in the ROM 103 according to the contents of the role setting screen.

[0052] When CPU601 receives a role acquisition command via network I / F605, it reads the role setting information stored in ROM103 and sends it to the source of the role acquisition command.

[0053] Although Figures 1 and 2 depict the role control device 600 as an independent device, for example, a shooting control application executed by the shooting control device 100 may provide the same functionality as the role control device 600. Alternatively, roles may be directly assigned to the sub-camera 400, and the shooting control device 100 may acquire the roles assigned to the sub-camera 400 from the sub-camera 400.

[0054] The roles that can be assigned to the sub-camera 400 are predetermined to determine how the information obtained from the main camera 500 will be used to control the operation of the sub-camera 400. Here, as an example, the information from the main camera will be used to control the subject tracking and zoom operation of the sub-camera 400.

[0055] (Switcher 1000) Next, an example of the functional configuration of switcher 1000 will be described. Switcher 1000 has a CPU 1001, RAM 1002, video input unit 1011, video switch control unit 1004, video output unit 1005, user input I / F 1006, and network I / F 1007. In addition, 1008 is an internal bus that connects each of the above blocks.

[0056] The CPU 1001 controls the operation of each functional block and realizes the functions of the switcher 1000 by, for example, loading a program stored in the ROM 1003 into the RAM 1002 and executing it.

[0057] RAM 1002 is used to load programs executed by CPU 1001, and to temporarily store data processed by CPU 1001, data being processed, etc. Additionally, a portion of RAM 1002 may be used as video memory for an external display device connected to monitor output unit 1009, or as a buffer for input video signals.

[0058] ROM1003 is a rewritable, non-volatile memory that stores programs executed by CPU1001, settings for switcher1000, user data, and other similar information.

[0059] The video input units 1011a to 1011d are interfaces for receiving video from sub-cameras 400a to 400c and the main camera 500, and consist of receivers compliant with standards such as SDI and HDMI. In this embodiment, there are four video input units 1011a to 1011d for receiving video from sub-cameras 400a to 400c and the main camera 500, but the number of video input units 1011 only needs to be two or more. Here, video input unit 1011a receives video from sub-camera 400a, video input unit 1011b receives video from sub-camera 400b, video input unit 1011c receives video from sub-camera 400c, and video input unit 1011d receives video from the main camera 500.

[0060] Instead of inputting video from sub-cameras 400a to 400c and the main camera 500 into video input units 1011a to 1011d, the video may be input via the communication network 700. When video is input via the communication network 700, the video transmitted by sub-camera 400 from network I / F 405 and the video transmitted by main camera 500 from network I / F 505 are received from network I / F 1007. The video from sub-cameras 400a to 400c and the main camera 500 may be a mix of video input from the video input unit 1011 and video input from network I / F 1007. In addition, five or more video streams may be input via the communication network 700.

[0061] The video switch control unit 1004 outputs one of the multiple video signals input to the video input units 1011a to 1011d from each of the sub-cameras 400a to 400c and the main camera 500 to the video output unit 1005, which will be described later. The video switch control unit 1004 outputs the video signal selected by the user input interface 1006, which will be described later. The video signal output by the video switch control unit 1004 is used for purposes such as broadcasting, distribution, and recording, and is referred to as the "distribution video" in this specification for convenience, but is sometimes called the "main video." The video switch control unit 1004 is configured to select and output one video signal from all the video signals input to the switcher 1000, regardless of the input path.

[0062] The video output unit 1005 is an interface for outputting the video output by the video switch control unit 1004 to live streaming equipment (not shown), program recording equipment, etc., and is composed of a transmitter compliant with standards such as SDI and HDMI.

[0063] The monitor output unit 1009 generates a video that displays a list of multiple video inputs to the switcher 1000 via the video input units 1011a to 1011d and the communication network 700. The monitor output unit 1009 also outputs the generated video to an external display device (not shown). The monitor output unit also outputs data from the switcher 1000's settings screen to the external display device.

[0064] The user input interface 1006 is an interface for connecting input devices (not shown) such as buttons, dials, joysticks, and touch panels. The switcher 1000 receives user instructions via the user input interface 1006 regarding the selection of the video being streamed and the preview video, as well as the settings of the switcher 1000. The preview video is the video that is scheduled to be streamed next.

[0065] The CPU 1001 assigns information (status information) to each camera receiving selectable video input from the video switch control unit 1004, indicating whether the captured video is being streamed, previewed, or something else. Specifically, the CPU 1001 determines the status information of cameras capturing video being output from the video output unit 1005 (streaming video) to be "streaming." The CPU 1001 also determines the status information of cameras capturing preview video to be "previewing." Furthermore, the CPU 1001 determines the status information of cameras capturing the remaining video to be "standby." The CPU 1001 associates the status information with the identification information of each camera and stores it in the RAM 1002 as camera status information STREAMING. The identification information of the sub-camera 400 may be any information that can identify each individual sub-camera, such as a unique name assigned by the user, a serial number, or a network address. The CPU 1001 updates the camera status information STREAMING stored in the RAM 1002 whenever, for example, the selection of the video signal is changed.

[0066] The tally signal output unit 1010 transmits control signals (tally information) via the network interface 1007 for cameras whose status information is "streaming" or "previewing" to display a tally. Therefore, the tally information may also be used as the camera status information STREAMING.

[0067] Network I / F 1007 is an interface for connecting switcher 1000 to communication network 700. Switcher 1000 (CPU 1001) can communicate with external devices on communication network 700, such as the overhead camera 300, sub-cameras 400a to 400c, main camera 500, and shooting control device 100, via network I / F 1007. Switcher 1000 may also communicate with external devices via other communication interfaces (not shown, such as USB or Bluetooth).

[0068] Figure 4 shows an example of the types of roles that can be set for the sub-camera 400 and the control content associated with those roles. The control content for each role can be stored in the ROM 603 of the role control device 600 and the ROM 103 of the shooting control device 100 in a table format, for example, as shown in Figure 4. Here, it is assumed that one of the following roles can be set: "Main Follow," "Main Counter," "Assist Follow," or "Assist Counter." If there are multiple sub-cameras 400, the role can be set for each sub-camera.

[0069] For sub-camera 400 whose role is "main follow," the shooting control device 100 (CPU 101) sets the same tracking subject as the main camera 500, and when the main camera 500 is zoomed, it also performs in-phase zoom control on the sub-camera 400. Here, in-phase means that the direction of zoom (telephoto or wide-angle) is the same, that is, the direction of the angle of view change is the same. On the other hand, opposite phase means that the direction of zoom (telephoto or wide-angle) is the opposite, that is, the direction of the angle of view change is the opposite. Note that even if the zoom direction is in-phase, the angle of view does not have to be the same as that of the main camera 500, and the degree of zoom change (such as the speed of change or rate of change) does not have to be the same as that of the main camera 500, regardless of whether it is in-phase or opposite phase.

[0070] For sub-camera 400 whose role is "main counter," the shooting control device 100 (CPU 101) sets the same tracking subject as the main camera 500, and when the main camera 500 is zoomed, it performs a zoom control on the sub-camera 400 in the opposite phase. Therefore, when the main camera 500 is zoomed in, the shooting control device 100 (CPU 101) controls the sub-camera 400 with this role to zoom out. Note that zooming in means changing the zoom in the telephoto direction (telephoto end direction), and zooming out means changing the zoom in the wide-angle direction (wide-angle end direction). When zoom control is performed by the image processing unit 406, zooming in means making the area to be cut out from the image smaller and increasing the magnification of the cut-out area compared to before the area change. On the other hand, zooming down means making the area to be cut out from the image larger and decreasing the magnification of the cut-out area compared to before the area change.

[0071] For sub-camera 400 whose role is "assist follow," the shooting control device 100 (CPU 101) sets a different tracking subject from the main camera 500, and when the main camera 500 is zoomed, it also performs the same phase zoom control on sub-camera 400.

[0072] For the sub-camera 400, whose role is "assist counter," the shooting control device 100 (CPU 101) sets a different tracking subject from the main camera 500. Furthermore, when the main camera 500 is zoomed, the shooting control device 100 applies a zoom control in the opposite phase to the sub-camera 400.

[0073] Here, for sub-camera 400 with the roles "Assist Follow" and "Assist Counter," the subject located to the left of the main camera 500's focus in the image is set as the subject to be tracked by sub-camera 400. Note that the subject to be tracked by sub-camera 400 may be set according to other conditions. For example, the subject to be tracked by sub-camera 400 may be located to the right, above, or below the main camera 500's focus in the image. Alternatively, the subject located closest to or furthest from the main camera 500's focus may be set as the subject to be tracked by sub-camera 400.

[0074] Alternatively, you can choose to perform only one of the following: setting the subject to be tracked or controlling the zoom, or you can add other control items.

[0075] The role setting information stored in the ROM 603 by the role control device 600 includes information indicating the role (such as the name of the type or the number assigned to the type) associated with the identification information of the sub-camera 400. The CPU 101 of the shooting control device 100 obtains the role setting information from the role control device 600 and performs operation control of the sub-camera 400 according to the type of role set for the sub-camera 400.

[0076] Furthermore, the role control device 600 may notify an external device (e.g., the shooting control device 100) if there is a change in the role setting for the sub-camera 400. This allows the change in the role setting to be immediately reflected in the operation control of the sub-camera 400.

[0077] <Explanation of the operation of each device> Next, the operation of each device in the multi-camera imaging system will be explained. Here, the shooting control device 100 automatically controls the shooting operation of the sub-camera 400 based on the video from the overhead camera 300, information obtained from the main camera 500, the camera status information STREAMING, and the role set for the sub-camera 400.

[0078] Figure 3 is a diagram illustrating the series of processes performed by the shooting control device 100 when controlling the operation of the sub-camera 400, focusing on the main operations and signal flows. The functional blocks shown within the shooting control device 100 schematically represent the main operations and correspond to the main functions provided by the shooting control application. Each functional block in Figure 3 is realized by a combination of the CPU 101 that executes the shooting control application and one or more of the functional blocks of the shooting control device 100 shown in Figure 2.

[0079] Figure 5 is a flowchart showing the operation of the CPU 101 as the role determination unit 120. Figures 6(a) to 6(d) are flowcharts relating to the operation of the shooting control device 100, the overhead camera 300, the main camera 500, and the sub-camera 400, respectively.

[0080] In the following description, it is assumed that the 3D coordinate values ​​of the viewpoint position and the shooting direction (optical axis direction) of the overhead camera 300 are known to the shooting control device 100. Furthermore, it is assumed that known positional information, such as the 3D coordinate values ​​of the viewpoint positions of the sub-camera 400 and the main camera 500, and the coordinate values ​​of markers placed within the shooting range, are pre-stored in the ROM 103 as default positional information REF_POSI. Note that the coordinate system of the position is predetermined according to the type of position.

[0081] (Operation of the role determination unit 120) First, the operation of the CPU 101 as the role determination unit 120 in Figure 3 will be explained with reference to the flowchart shown in Figure 5. The operation described below is achieved by the CPU 101 executing the shooting control application.

[0082] There are no particular restrictions on the timing of initiating the operations shown in the flowchart of Figure 5, but they should be performed at least before control of the shooting operation of the sub-camera 400 begins. Furthermore, the operations should also be performed when the role control device 600 receives notification from the network I / F 105 via the communication network 700 that the role setting for the sub-camera 400 has been changed.

[0083] In S101, the CPU 101, acting as the role determination unit 120, acquires the role (role setting information) corresponding to the sub-camera 400 from the role control device 600. The CPU 101, acting as the role determination unit 120, can acquire the aforementioned role setting information from the role control device 600 by, for example, sending a role acquisition command to the role control device 600 via the communication network 700 from the network I / F 105. The CPU 101 stores the acquired role setting information in the RAM 102.

[0084] In S102, the CPU 101, acting as the role determination unit 120, refers to the role setting information stored in the RAM 102 based on the identification information of the sub-camera 400 and determines the operation control content CAMERA_ROLE for the sub-camera 400. Then, in S103, the CPU 101, acting as the role determination unit 120, transmits the acquired operation control content CAMERA_ROLE to the tracking subject determination unit 123. In practice, the CPU 101 stores the operation control content CAMERA_ROLE in a specific area of ​​the RAM 102 and refers to it when functioning as the tracking subject determination unit 123.

[0085] In S104, the CPU 101, acting as the role determination unit 120, transmits the acquired operation control content CAMERA_ROLE to the zoom value calculation unit 125. In practice, the CPU 101 stores the operation control content CAMERA_ROLE in a specific area of ​​RAM 102 and refers to it when functioning as the zoom value calculation unit 125.

[0086] (Operation of the shooting control device 100) Next, the operation of the shooting control device 100 in controlling shooting by the sub-camera 400 will be explained with reference to Figures 3 and 6(a). The operation described below corresponds to the operation of the CPU 101 as the recognition unit 121, the focus subject determination unit 122, the tracking subject determination unit 123, the pan / tilt value calculation unit 124, and the zoom value calculation unit 125 in Figure 3. Note that the operation described below is realized by the CPU 101 executing the shooting control application.

[0087] In S201, the CPU 101 sends a shooting command to the overhead camera 300 via the communication network 700 from the network I / F 105 using a predetermined protocol. In response to this command, the overhead camera 300 begins supplying a video signal (video data) IMG to the network I / F 105 via the communication network 700. After the CPU 101 begins storing the video signal IMG received by the network I / F 105 into the RAM 102, it executes S202.

[0088] In S202, the CPU 101 acquires information ANGLE indicating the shooting direction from the main camera 500. Specifically, the CPU 101 sends a shooting direction acquisition command to the main camera 500 via the communication network 700 from the network I / F 105 using a predetermined protocol. In response to the shooting direction acquisition command, the CPU 501 of the main camera 500 sends information ANGLE indicating the current shooting direction of the main camera 500 to the shooting control device 100. Information ANGLE may be, for example, the pan and tilt angles of the drive unit 509. The CPU 101 stores the acquired information ANGLE in the RAM 102.

[0089] In S203, the recognition unit 121 performs the following process. (1) Apply subject area detection processing to the input frame image and store the detection result. (2) For each detected subject area, the position information (image coordinates) of the subject is transformed. (3) Apply identification processing to each detected subject area to identify identification information (add information for identification processing in the case of a new subject). (4) Store the identification information ID[n] and location information POSITION[n] associated with each detected subject area.

[0090] The recognition unit 121 is mainly implemented by the CPU 101 and the inference unit 104. The CPU 101 reads one frame of video received from the overhead camera 300 from the RAM 102 and inputs it to the inference unit 104.

[0091] The operation of the recognition unit 121 will be explained step by step below. (1) First, the inference unit 104 inputs the frame image to the machine learning model and detects the subject area. The inference unit 104 stores the position and size of each detected subject area, and the detection confidence level, which are output by the machine learning model as detection results, in the RAM 102. The position and size of the subject area may be any information that can identify the position and size of the rectangular area inscribed in the subject area. Here, the coordinates of the center of the bottom edge of the rectangular area, as well as its width and height, are used as the position and size of the subject area.

[0092] Furthermore, the inference unit 104 stores the detection result for the first frame image in RAM 102 in association with the subject identification information ID[n]. Here, n is the subject number and is an integer that takes values ​​from 1 to the total number of detected subject regions. In addition, the inference unit 104 stores the subject regions detected from the first frame image in RAM 102 in association with the subject identification information ID[n] as a template for identifying individual subjects. If template matching is not used for subject identification, it is not necessary to store the template.

[0093] Figure 8(a) shows an example of the results of subject detection processing by the inference unit 104 on the image from the overhead camera 300 shown in Figure 7(a). Here, the areas of human subjects A to C that are within the shooting range 20 are detected, and the coordinates of the center of the bottom edge of the rectangular area inscribed with the subject area (foot coordinates) are output as the position.

[0094] Furthermore, for coordinate transformations described later, if markers are placed at known positions within the shooting range 20, as shown in Figure 7(b), the CPU 101 detects the marker images included in the frame image (Figure 7(a)) and stores their positions in the RAM 102. The detection of marker images may also be configured to be performed by the inference unit 104. Marker image detection can be performed by any known method, such as pattern matching using a marker template. Marker images may also be detected using a pre-stored machine learning model for marker detection.

[0095] (2) Next, the coordinate transformation performed by the inference unit 104 will be explained. Figure 7(a) schematically shows the image from the overhead camera 300, and Figure 7(b) schematically shows the shooting range 20 as viewed from directly above its center. The inference unit 104 transforms the position of the subject area in the coordinate system of the overhead camera to the coordinate system (plane coordinate system) when the shooting range 20 is viewed from directly above its center.

[0096] The reason for converting the coordinates to a planar coordinate system here is that it is convenient for calculating the pan value (angle of movement in the horizontal plane) required to photograph a specific subject with the sub-camera 400. It is assumed here that the sub-camera 400 is positioned so that the drive unit 409 pans within a horizontal plane parallel to the floor of the shooting range 20.

[0097] Coordinate transformation can be performed in various ways, but here, markers are placed at multiple known positions on the floor within the shooting range 20, and the coordinates are transformed from the overhead camera coordinate system to the planar coordinate system based on the marker positions in the image obtained from the overhead camera 300. Alternatively, the coordinate transformation may be performed without using markers, such as by using the viewpoint position and shooting direction of the overhead camera 300.

[0098] The coordinate transformation can be performed using the homography transformation matrix H, according to Equation 1 below.

number

[0099] The homography transformation matrix can be calculated by substituting the coordinates of the four markers detected from the video and the coordinates (known) of the four markers placed in the shooting range 20 into Equation 1 and solving the system of equations. If the positional relationship between the shooting range 20 and the overhead camera 300 is fixed, the homography transformation matrix H can be calculated in advance during test shooting and saved, for example, in ROM 103.

[0100] The CPU 101 sequentially reads the position of the subject area from the RAM 102 and transforms the coordinates to values ​​in a planar coordinate system. Figure 8(b) schematically shows the state after the foot coordinates (x,y) of each subject area detected in the overhead camera 300 video shown in Figure 8(a) have been transformed to coordinate values ​​(X,Y) in a planar coordinate system using Equation 1 and the homography transformation matrix H stored in ROM 103. The CPU 101 stores the transformed foot coordinates as POSITION[n] in the RAM 102.

[0101] (3) Next, the operation of the inference unit 104 in identifying the subject identification information ID[n] will be described. Here, the subject will be identified using template matching. Subject identification will be performed on the processing results of the second and subsequent subject detections. For the first processing result, a new identification information ID[n] will be assigned to the subject region.

[0102] The inference unit 104 identifies the identification information ID[n] of the detected subject area by template matching using templates stored in RAM 102. This identifies the subject within the shooting range. For example, the inference unit 104 calculates an evaluation value representing the correlation of individual templates for each detected subject area. Then, the inference unit 104 identifies the identification information ID[n] corresponding to the template with the highest correlation that has a certain level of correlation as the identification information ID[n] of the subject area. The evaluation value can be a known value, such as the sum of the absolute differences of pixel values.

[0103] Furthermore, the inference unit 104 assigns a new identification information ID[n] to any subject area that does not have a certain level of correlation with all templates, and adds the image of the subject area to the template.

[0104] Furthermore, the inference unit 104 may update existing templates using subject regions detected in the most recent frame image, or delete templates for which no subject regions with a certain level of correlation have been found for a certain period of time. In addition, the inference unit 104 may store templates corresponding to frequently appearing identification information ID[n] in the ROM 103.

[0105] Furthermore, subjects may be identified by methods other than template matching. For example, at least one of the previously detected position and size may be identified as having the same identification information ID [n] as the nearest subject region. Alternatively, the position in the current frame image may be predicted using a Kalman filter or the like based on the position changes in multiple past detection results associated with the same identification information, and the same identification information ID may be identified as the subject region closest to the predicted position. These methods may also be combined. By not using template matching, the accuracy of identifying different subjects that look similar can be improved.

[0106] (4) The inference unit 104 associates the identified identification information ID[n] with the corresponding position (plane coordinate system) POSITION[n] of the subject area and stores it in the RAM 102.

[0107] Note that, of the processes (1) to (4), the CPU 101 may execute the processes other than subject detection instead of the inference unit 104.

[0108] Here, the image from the overhead camera 300 was used to determine the identification information ID[n] and position[n] of the subject within the shooting range 20. However, the image from the sub-camera 400 may also be used. If there are multiple sub-cameras 400, the CPU 101 performs the operations shown in the flowchart in Figure 6(a) for each sub-camera 400. The position of the subject area is output as a value in the coordinate system of each sub-camera 400. Thus, although the overhead camera 300 is not essential, it is considered that using the overhead camera 300 improves the accuracy of subject detection.

[0109] Returning to the explanation of Figure 6(a), in S204, the CPU 101, which acts as the subject of interest determination unit 122 in Figure 3, determines the subject of interest to be tracked by the main camera 500. The CPU 101 can determine the subject of interest of the main camera 500 from among the subjects detected in S203, based on the shooting direction of the main camera 500 acquired in S202. The CPU 101 stores the identification information ID[n] corresponding to the subject area determined to be the subject of interest of the main camera 500 in the RAM 102 as the identification information of the subject of interest MAIN_SUBJECT.

[0110] For example, the CPU 101 can determine the subject closest to the shooting direction of the main camera 500 in a planar coordinate system as the subject of interest for the main camera 500. If there are multiple subjects whose distance from the shooting direction of the main camera 500 is below a threshold, the user may be allowed to select the subject of interest from among them.

[0111] When prompting the user to select a subject of interest, the CPU 101 displays the frame image to which subject detection processing has been applied in S202 on the display unit 108 or an external display device, along with an indicator showing the shooting direction and an indicator showing the subject area that is a candidate for the subject of interest. The subject area indicator may be a rectangular frame indicating the outer edge of the subject area, as shown in Figure 8(a), but other indicators may also be used. The CPU 101 may also display messages on the display unit 108 prompting the user to select a subject of interest in the image.

[0112] The user can select a subject area corresponding to a desired subject of interest by operating the user input unit 106 (input device). There are no particular restrictions on the selection method, but it may be an operation to specify the desired subject area by using a mouse or keyboard.

[0113] When the CPU 101 detects a user operation specifying a subject area, it stores the identification information ID[n] corresponding to the specified subject area in the RAM 102 as the identification information MAIN_SUBJECT of the subject of interest.

[0114] Next, in S205, the CPU 101, acting as the tracking subject determination unit 123 in Figure 3, obtains the operation control content CAMERA_ROLE corresponding to the role set for the sub-camera 400. Specifically, the CPU 101 reads the operation control content CAMERA_ROLE obtained in the role determination process explained using Figure 5 and stored in the RAM 102. If there are multiple sub-cameras 400, the CPU 101 executes the processes in S205 to S209 for each sub-camera.

[0115] In S206, the CPU 101, acting as the linkage execution decision unit 130, decides whether or not to link the sub-camera 400 with the main camera 500. Details of the decision method will be described later. In addition, a linkage execution flag is stored in RAM 102 as a flag to switch whether or not to link. The linkage execution flag takes the values ​​ON and OFF, and the default is ON. If in S206 it is decided to link the sub-camera 400 with the main camera 500, the CPU 101 leaves the linkage execution flag for the sub-camera 400 stored in RAM 102 as ON. On the other hand, if it is decided not to link, the CPU 101 updates the linkage execution flag for the sub-camera 400 stored in RAM 102 to OFF.

[0116] In S207, the CPU 101, acting as the subject tracking unit 123, determines the subject to be tracked and photographed by the sub-camera 400. If the linked execution flag is OFF, the CPU 101 does not change the subject being tracked by the sub-camera 400, even if the subject of interest of the main camera 500 changes.

[0117] On the other hand, if the linked operation flag is ON, the CPU 101 determines the subject to be tracked and photographed by the sub-camera 400 according to the subject of interest of the main camera 500, in accordance with the operation control content CAMERA_ROLE. The CPU 101 determines the subject to be tracked by the sub-camera 400 according to the definition of the tracked subject included in the operation control content CAMERA_ROLE (Figure 4).

[0118] If the subject being tracked by sub-camera 400 is to be the same as the subject of interest of main camera 500, CPU 101 sets the identification information of the subject of interest determined in S203, MAIN_SUBJECT, as the identification information of the subject being tracked by sub-camera 400, SUBJECT_ID.

[0119] If the subject to be tracked by the sub-camera 400 is to be a subject located to the left of the subject of interest of the main camera 500, the CPU 101 detects the subject region located at the leftmost edge of the subject region other than the subject of interest among the subject regions detected in S203. Then, the CPU 101 sets the identification information ID[n] corresponding to the detected subject region as the identification information SUBJECT_ID of the subject to be tracked by the sub-camera 400.

[0120] The CPU 101 writes the identification information SUBJECT_ID of the determined tracked subject to RAM 102. If the tracked subject may differ depending on the sub-camera, the CPU 101 stores the identification information SUBJECT_ID of the tracked subject in association with the identification information of the sub-camera. If the tracked subject changes, the CPU 101 retains the information of the previous tracked subject in RAM 102 without erasing it.

[0121] Here, we will explain the operation when the role set for the sub-camera 400 is "main follow," using Figure 9. When the role "main follow" is set for the sub-camera 400, the shooting control device 100 controls it to track the subject of interest of the main camera 500.

[0122] Therefore, as shown in Figure 9(a), if the main camera 500 determines that the subject of interest is subject B, the CPU 101 decides that subject B will be the subject to be tracked by the sub-camera 400. Subsequently, as shown in Figure 9(b), if the main camera 500 determines that the subject of interest has changed to subject A, the CPU 101 changes the subject to be tracked by the sub-camera 400 to subject A. Similarly, as shown in Figure 9(c), if the main camera 500 determines that the subject of interest has changed to subject C, the CPU 101 changes the subject to be tracked by the sub-camera 400 to subject C.

[0123] The operation of the sub-camera 400 when its assigned role is "Assist Follow" will be explained using Figure 10. When the sub-camera 400 is assigned the role "Assist Follow," the shooting control device 100 controls it to track a subject located to the left of the main camera 500, which is a subject other than the subject of interest of the main camera 500.

[0124] Therefore, as shown in Figure 10(a), if the main camera 500 determines that the subject of interest is subject B, the CPU 101 decides that subject A, the leftmost of subjects A and C, will be the subject tracked by the sub-camera 400. Subsequently, as shown in Figure 10(b), if the main camera 500 determines that the subject of interest has changed to subject A, the CPU 101 changes the subject tracked by the sub-camera 400 to subject B, the leftmost of subjects B and C. Also, as shown in Figure 10(c), if the main camera 500 determines that the subject of interest has changed to subject C, the CPU 101 changes the subject tracked by the sub-camera 400 to subject A, the leftmost of subjects A and B.

[0125] The role control device 600 dynamically changes the role assigned to the sub-camera 400, thereby changing the subject tracked by the sub-camera 400 and enabling flexible automatic shooting.

[0126] Returning to Figure 6(a), in S208, the CPU 101, acting as the pan / tilt value calculation unit 124, calculates the amount of change in pan and tilt angles necessary for the sub-camera 400 to track and photograph the subject determined in S207. The CPU 101, acting as the zoom value calculation unit 125, also calculates the zoom value of the sub-camera 400 in accordance with the change in the field of view of the main camera 500. The following describes the case where there is one sub-camera 400, but if there are multiple sub-cameras 400, the calculation of the pan and tilt angle changes and the zoom value are performed for each sub-camera.

[0127] First, the operation of the CPU 101 as the pan-tilt value calculation unit 124 will be described. Here, it is assumed that the following information is stored in advance in the ROM 103 as the default position information REF_POSI for each sub-camera 400. · 3D coordinates of the installation position (values in a planar coordinate system) · Shooting direction corresponding to the initial values of the pan angle and tilt angle of the drive unit · Controllable range of the pan and tilt angles

[0128] The CPU 101 reads out the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking subject of the sub-camera 400 from the RAM 102. Then, the CPU 101 determines the pan angle first from the position information POSITION_OH and the installation position of the sub-camera 400.

[0129] FIG. 11 is a diagram showing an example of the positional relationship between the sub-camera 400 and the tracking subject in a planar coordinate system. Here, it is assumed that the pan angle θ that directs the optical axis direction of the sub-camera 400 toward the subject position is determined. The CPU 101 calculates the pan angle θ using the following Equation 2.

Equation

[0130] In Equation 2, px and py are the horizontal and vertical coordinates of the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking subject. Also, subx and suby are the horizontal and vertical coordinates of the installation position of the sub-camera. Here, it is assumed that the current pan angle is the initial value of 0° and the optical axis direction is the vertical direction (Y-axis direction). If the current optical axis direction is not the vertical direction, the angle difference between the current optical axis direction and the vertical direction may be reflected in the angle obtained by Equation 2. Also, the direction of the pan is counterclockwise if subx > px, and clockwise if subx < px.

[0131] Next, the method for determining the tilt angle will be described using FIG. 12. FIG. 12 shows a state in which the sub-camera and the tracked subject are viewed from the side. Assume that the current optical axis of the sub-camera 400 is in the horizontal direction and has a height of h1, and the height of the face of the tracked subject facing the optical axis is h2. Let the angle difference (tilt angle) in the height direction between the current optical axis direction and the target optical axis direction be ρ. The CPU 101 calculates the tilt angle ρ using the following equations 3 and 4. [Number]

[0132] The coordinate values used in Equation 4 are the same as those used in Equation 2. Assume that h1 and h2 are input in advance to the shooting control application and stored in the RAM 102. In this case, the identification number associated with h2 for each subject is made equal to the identification number assigned in the subject detection process. Alternatively, h2 may be a value measured in real time using a sensor (not shown).

[0133] Here, assume that the current tilt angle is the initial value of 0°, and the optical axis direction is the horizontal direction (constant height). If the current optical axis direction is not the horizontal direction, the angle difference between the current optical axis direction and the horizontal direction may be reflected in the angle obtained by Equation 4. Also, the tilt direction is downward if h1 > h2, and upward if h1 < h2.

[0134] The CPU 101 communicates with the sub-camera 400 periodically through the communication network 700, acquires the current optical axis direction (the pan angle and tilt angle of the drive unit), and stores it in the RAM 102. Note that the communication cycle can be, for example, less than or equal to the reciprocal of the frame rate. Alternatively, the CPU 101 may hold the total values of the pan angle and tilt angle controlled from the initial state for the sub-camera 400 in the RAM 102 and use them as the current optical axis direction.

[0135] The CPU 101 calculates the amount of change in the pan angle and tilt angle of the sub-camera 400 in this manner and stores it in the RAM 102. If there are multiple sub-cameras 400, the CPU 101 calculates the amount of change in the pan angle and tilt angle for each sub-camera.

[0136] The changes in the pan and tilt angles may be the angular velocity required for the sub-camera 400 to rotate towards the tracked subject. For example, the CPU 101 obtains the current pan and tilt angles from the sub-camera 400 via the communication network 700. The CPU 101 then calculates the angular velocity of the pan, which is proportional to the difference between the pan angle θ read from the RAM 102 and the current pan angle. The CPU 101 also calculates the angular velocity of the tilt, which is proportional to the difference between the tilt angle ρ read from the RAM 102 and the current tilt angle. The CPU 101 stores these calculated angular velocities in the RAM 102.

[0137] Alternatively, the change in pan angle and tilt angle may be calculated using the image from the sub-camera 400 instead of the image from the overhead camera 300. In this case, the CPU 101 may calculate the change in pan angle from the horizontal difference between the current optical axis direction and the direction of the tracked subject in the coordinate system of the sub-camera 400, and the change in tilt angle from the vertical difference. Furthermore, the imaging system may change the shooting direction for tracking the tracked subject by only changing either the pan direction or the tilt direction, and in such an imaging system, only the change in either the pan angle or the tilt angle may be calculated.

[0138] Next, the operation of the CPU 101 as the zoom value calculation unit 125 will be described. The CPU 101 as the zoom value calculation unit 125 periodically acquires information MAIN_ZOOM indicating the field of view of the main camera 500 and stores it in RAM 102. When the information MAIN_ZOOM changes, the CPU 101 calculates the zoom value Z_VALUE for the sub-camera 400 according to the operation control content CAMERA_ROLE corresponding to the role set for the sub-camera 400. The format of the control command is predetermined. The CPU 101 stores the generated control commands PT_VALUE and Z_VALUE in RAM 102. Note that if the tracked subject is stationary, if the field of view of the main camera 500 does not change, or if there is no need to generate a control command, S208 may be skipped. For the sub-camera 400 that is not linked to the main camera 500, a field of view control command may be generated to maintain the size of the tracked subject, or the field of view may be fixed without generating a field of view control command.

[0139] The CPU 101 can determine the zoom operation of the main camera 500 and its phase, for example, by detecting changes in the field of view of the image from the main camera 500. For example, changes in the field of view may be detected from changes in the size and spacing of the subject area over time.

[0140] Figure 13 shows an example of mapping the zoom values ​​of the main camera and the sub-camera. Here, the main camera 500 and the sub-camera 400 are assumed to optically change their field of view (the imaging optical system has a zoom function). However, a similar function may be achieved with digital zoom using the image processing units 406 and 506.

[0141] The zoom value is a parameter whose value corresponds to the field of view. In this embodiment, the smaller (narrower) the field of view, the smaller the zoom value, and the zoom value on the telephoto side is smaller than the zoom value on the wide-angle side. The sub-camera 400 and the main camera 500 can control the imaging optical system to the field of view corresponding to the zoom value by sending a command that specifies the zoom value. In other words, the zoom value is information about the field of view and represents the zoom state. The zoom value may also be, for example, the focal length (mm) of the imaging optical system corresponding to a 35mm full-frame image sensor, in which case the zoom value on the telephoto side will be larger than the zoom value on the wide-angle side.

[0142] In Figure 13, the zoom range of the main camera 500, MAIN_ZOOM, is main_min to main_max. The zoom range of the sub-camera 400 is sub_min to sub_max. main_min and sub_min correspond to the telephoto ends of the main camera 500 and sub-camera 400, respectively, while main_max and sub_max correspond to the wide-angle ends of the main camera 500 and sub-camera 400, respectively. Figure 13 shows an example where the zoom range of the main camera 500 is wider than the zoom range of the sub-camera 400 at both the telephoto and wide-angle ends.

[0143] When controlling the zoom value SUB_ZOOM of sub-camera 400 to be in phase with the zoom value MAIN_ZOOM of main camera 500, CPU 101 calculates the SUB_ZOOM corresponding to the current MAIN_ZOOM using the following equation 5.

number

[0144] On the other hand, when controlling the zoom value SUB_ZOOM of sub-camera 400 in the opposite phase to the zoom value MAIN_ZOOM of main camera 500, the SUB_ZOOM corresponding to the current MAIN_ZOOM is calculated using the following equation 6. Specifically, the CPU 101 calculates the SUB_ZOOM corresponding to the current MAIN_ZOOM by substituting the SUB_ZOOM calculated in equation 5 into the right-hand side of the following equation 6. SUB_ZOOM=sub_max-(SUB_ZOOM-sub_min) (Formula 6)

[0145] When the main camera 500 performs digital zoom and controls the field of view by cropping, the CPU 101 can determine the zoom value SUB_ZOOM of the sub-camera 400 according to the size of the area to be cropped by the main camera 500. Specifically, the CPU 101 sets the zoom value SUB_ZOOM to a smaller size (higher magnification) when the area to be cropped by the main camera 500 is small, and to a larger size (lower magnification) when the area to be cropped is large.

[0146] Furthermore, the zoom control associated with the role of the sub-camera 400 is not limited to in-phase or out-of-phase control with the main camera 500. For example, a zoom operation independent of the angle of view change of the main camera 500 may be associated with the role. For example, an auto-zoom operation that maintains a constant size of the tracked subject may be associated with the role. Alternatively, the angle of view of the sub-camera 400 may be fixed to a specific angle of view. By adding roles associated with these zoom controls to the control content for each role shown in Figure 4, or by changing the zoom control content of the roles shown in Figure 4, various zoom controls for the sub-camera 400 become possible.

[0147] Returning to Figure 6(a), in S209, the CPU 101, acting as the linked execution decision unit 130, transmits the control command calculated in S208 to the sub-camera 400. Here, the CPU 101 reads the control commands PT_VALUE and Z_VALUE from RAM 102 and transmits them to the communication network 700 via the network I / F 105. The sub-camera 400 receives the control commands PT_VALUE and Z_VALUE from the network I / F 405 via the communication network 700.

[0148] The CPU 101 executes the processing from S201 on the next frame image of the video from the overhead camera 300. Note that the processing shown in Figure 6(a) does not necessarily have to be executed every frame.

[0149] (Operation of overhead camera 300) Next, the operation of the overhead camera 300 will be explained with reference to Figure 6(b). The operation described below is achieved by the CPU 301 executing a program.

[0150] When the overhead camera 300 is powered on, the CPU 301 initializes each functional block, and then the camera enters a shooting standby state. In the shooting standby state, the CPU 301 may start video recording processing for live view display and output the display image data generated by the image processing unit 306 to the shooting control device 100 via the network I / F 305.

[0151] In the shooting standby state, the CPU 301 waits for control commands to be received from the network I / F 305. When the CPU 301 receives a control command, it executes an action corresponding to the control command. This section describes the operation when a shooting command is received as a control command from the shooting control device 100.

[0152] In S301, the CPU301 receives the imaging command transmitted from the imaging control device 100 via the network I / F305 through the communication network 700.

[0153] The shooting command may also specify shooting parameters such as frame rate and resolution. Furthermore, it may include settings related to the processing to be applied by the image processing unit 306.

[0154] In S302, the CPU 301 responds to the reception of a shooting command and starts video recording processing to be supplied to the shooting control device 100. This video recording processing captures video with higher image quality than the video recording processing for live view display. For example, at least one of the video resolution and the shooting frame rate is higher than that of the video for live view display. The image processing unit 306 applies processing to the image based on the settings for the video to be supplied to the shooting control device 100. The image processing unit 306 sequentially stores the generated video data in the RAM 302.

[0155] In step S303, the CPU 101 reads the video data from the RAM 302 and transmits it to the shooting control device 100 via the communication network 700 through the network I / F 305. From this point onward, the processing from shooting to supplying video data continues until a control command to stop shooting is received.

[0156] (Operation of Main Camera 500) Next, the operation of the main 500 will be explained with reference to Figure 6(c). The operations described below are achieved by the CPU 501 executing a program.

[0157] When the main camera 500 is powered on, the CPU 501 initializes each functional block, and then starts video recording processing to supply to the shooting control device 100. The image processing unit 506 applies processing to the analog image signal obtained from the image sensor 507 based on the settings for video to be supplied to the shooting control device 100. The image processing unit 506 sequentially stores the generated video data in the RAM 502. The CPU 501 reads the video data from the RAM 502 and supplies it to the shooting control device 100 via the communication network 700 through the network I / F 505.

[0158] The CPU 501 supplies video data to the shooting control device 100 while waiting for control commands to be received from the network interface 305. When the CPU 501 receives a control command, it executes an action corresponding to the control command. This section describes the operation when a shooting direction acquisition command is received. If the CPU 501 receives the pan / tilt control command PT_VALUE or the zoom control command Z_VALUE, it drives the drive unit 509 according to the command.

[0159] In S501, CPU501 receives a shooting direction acquisition command from network I / F505 via communication network 700. CPU501 stores the received shooting direction acquisition command in RAM502.

[0160] In S502, the CPU 501, in response to receiving a command to acquire the shooting direction, acquires the current pan angle and tilt angle from the drive unit 509 via the drive I / F 508 and stores them in the RAM 502.

[0161] In S503, the CPU 501 reads the current pan angle and tilt angle from RAM 502 and transmits the shooting direction information ANGLE to the shooting control device 100 via the communication network 700 from the network I / F 305.

[0162] (Operation of sub-camera 400) Next, the operation of the sub-camera 400 will be explained with reference to Figure 6(d). The operation described below is achieved by the CPU 401 executing a program.

[0163] When the sub-camera 400 is powered on, the CPU 401 initializes each functional block and then starts video recording processing to supply to the shooting control device 100. The image processing unit 406 applies processing to the analog image signal obtained from the image sensor 407 based on the settings for video to be supplied to the shooting control device 100. The image processing unit 406 sequentially stores the generated video data in the RAM 402. The CPU 401 reads the video data from the RAM 402 and supplies it to the shooting control device 100 via the communication network 700 through the network I / F 405.

[0164] The CPU 401 supplies video data to the shooting control device 100 while waiting for control commands to be received from the network I / F 305. When the CPU 401 receives a control command, it executes an action corresponding to the control command. This section describes the operation when the CPU 401 receives the pan / tilt control command PT_VALUE and the zoom control command Z_VALUE from the shooting control device 100.

[0165] In S401, the CPU401 receives at least one of the pan / tilt control command PT_VALUE and the zoom control command Z_VALUE transmitted from the imaging control device 100 via the communication network 700 through the network I / F405. The CPU401 stores the received control commands in the RAM402.

[0166] In S402, CPU401 reads the operating direction and corresponding manipulated amount from the control command stored in RAM402 and stores them in RAM402. Here, in the case of the pan / tilt control command PT_VALUE, the operating direction is the direction of pan and / or tilt, and the manipulated amount is the target angle. Also, in the case of the zoom control command Z_VALUE, the manipulated amount is the zoom value, and since the operating direction can be determined from the zoom value, reading and storing the operating direction is unnecessary.

[0167] In S403, the CPU 401 generates drive parameters for the drive unit 409 based on the operating direction and operating amount read in S403. The CPU 401 may, for example, obtain drive parameters corresponding to a combination of operating direction and operating amount using a table previously stored in ROM 403. If the operating amount is given as a target value (target angle or zoom value), the CPU 410 obtains the drive parameters from the difference between that value and the current value.

[0168] In S404, CPU401 controls drive unit 409 via drive I / F408 based on the drive parameters acquired in S404. This causes drive unit 409 to change the shooting direction of sub-camera 400 to the operating direction and angle specified by the pan / tilt control command PT_VALUE. Additionally, drive unit 409 changes the field of view of the shooting optical system to the zoom value specified by the zoom control command Z_VALUE.

[0169] Next, in S206 of Figure 6(a), as an example of a method for determining whether or not to link the sub-camera 400 with the main camera 500, we will explain a method for making this determination based on the distribution status of the main camera 500.

[0170] Figure 14 is a flowchart showing the operation of the CPU 101 as the linked execution decision unit 130. In this embodiment, the shooting control device 100 basically automatically controls the shooting direction and field of view of the sub-camera 400 in conjunction with the movement of the main camera 500. However, if the image selected by the switcher 1000 is switched from the image of the main camera 500 to the image of the sub-camera 400, undesirable situations may occur with the automatic control of the sub-camera 400.

[0171] For example, if a photographer operating the main camera 500 loses track of the main camera 500's footage on the switcher 1000, they might drastically change the shooting direction or field of view to find the next subject to film. If the sub-camera 400 is linked to this operation of the main camera 500, which is based on the assumption that the footage is not being used, the footage from the sub-camera 400 selected by the switcher 1000 may become unsuitable for use in streaming or other applications. This issue can occur not only when the photographer directly operates the main camera 500, but also when it is operated remotely.

[0172] On the other hand, when the main camera 500's video feed is selected on the switcher 1000, the operator of the main camera 500 understands that the video being filmed is being used for streaming, etc., and therefore refrains from making significant changes to the filming direction or field of view. In this case, there is no problem with linking the sub-camera 400 to the main camera 500.

[0173] Therefore, in S206, the CPU 101 can decide whether or not to automatically control the sub-camera 400 to work in conjunction with the main camera 500, depending on whether or not the video from the main camera 500 is currently being streamed.

[0174] In S1401, the CPU 101, acting as the linked execution decision unit 130, acquires status information for the main camera 500. The CPU 101 obtains the current camera status information, STREAMING, stored in RAM 1002, from the switcher 1000 and extracts the status information for the main camera 500. Alternatively, the CPU 101 may directly acquire status information from the main camera 500 to reduce the amount of communication between the switcher 1000 and the shooting control device 100. Furthermore, as described above, tally information for the main camera 500 may be acquired from the switcher 1000 or the main camera 500 as camera status information for the main camera 500. The CPU 101 stores the acquired status information in RAM 102.

[0175] In S1402, the CPU 101 determines whether the status information (tally information) of the main camera 500 acquired in S1401 is "streaming". If it is determined to be "streaming", it executes S1403; otherwise, it executes S1401. Note that if tally information is acquired in S1401, the CPU 101 determines whether the tally information for the main camera 500 is "streaming".

[0176] In S1403, the CPU 101 turns on the linkage execution flag stored in RAM 102. This corresponds to the decision to link the sub-camera 400 with the main camera 500. When the linkage execution flag is ON, a control command is sent to the sub-camera 400 to automatically control it in conjunction with the main camera 500 according to its role.

[0177] In S1404, the CPU 101 turns OFF the linkage activation flag stored in RAM 102. This corresponds to the decision not to link the sub-camera 400 to the main camera 500. When the linkage activation flag is OFF, the CPU 101 sends a control command to the sub-camera 400 to continue tracking and shooting the subject it is currently tracking while maintaining its current field of view.

[0178] Thus, in this embodiment, the decision of whether or not to link the sub-camera 400 to the main camera 500 is made based on whether or not the video from the main camera 500 is selected by the switcher 1000. Then, automatic control is selectively performed for cases where the sub-camera 400 is linked to the main camera 500 and for cases where it is not linked.

[0179] If the main camera 500's video feed is not selected by the switcher 1000, the main camera 500's shooting direction and field of view may change abruptly. Therefore, the CPU 101 controls the sub-camera 400 to continue tracking the currently tracked subject without linking it to the main camera 500. On the other hand, if the main camera 500's video feed is selected by the switcher 1000, the possibility of the main camera 500's shooting direction and field of view changing abruptly is considered to be sufficiently low. Therefore, the CPU 101 controls the sub-camera 400's shooting operation to be linked to the main camera 500. This allows for the unattended operation of the sub-camera 400 while still providing stable, high-quality video even when the switcher 1000 selects the sub-camera 400's video feed.

[0180] Furthermore, even if the status information of the main camera 500 is "Previewing," the sub-camera 400 may not be linked to the main camera 500, similar to the case where it is "Streaming." Alternatively, instead of the camera status information STREAMING, the tally information output by the tally signal output unit 1010 may be used to determine whether or not the video from the main camera 500 is selected by the switcher 1000.

[0181] ●<Second Embodiment> Next, a second embodiment of the present invention will be described. In this embodiment, as another example of a method for determining whether or not to link the sub-camera 400 with the main camera 500 in S206 of Figure 6(a), a determination method based on the PTZ speed information of the main camera 500 will be described.

[0182] Figure 15 is a flowchart showing the operation of the CPU 101 as the linked execution decision unit 130 in this embodiment. As described above, the CPU 101, acting as the linked execution decision unit 130, periodically acquires the information ANGLE and the information MAIN_ZOOM from the main camera 500. In S1501, the CPU 101 calculates the pan, tilt, and zoom speeds (PTZ speeds) based on this information, the information acquisition period, and the amount of change, and stores them in the RAM 102.

[0183] In S1502, the CPU 101 reads the PTZ speed of the main camera 500 from RAM 102 and determines whether it exceeds a predetermined threshold. The predetermined threshold is set to a value faster than the speed of zoom, pan, and tilt operations used in typical video effects. If the CPU 101 determines that the speeds of pan, tilt, and zoom do not exceed the predetermined threshold, it executes S1503; otherwise, it executes S1504.

[0184] In addition, in S1502, instead of a predetermined threshold, the average speed or the number of rapid changes in speed within a predetermined period may be used. A rapid change may be, for example, the difference between the minimum and maximum values ​​of speed in a predetermined unit time shorter than a certain period, or the time change of speed exceeding a predetermined threshold. This makes it possible to control, for example, the sub-camera 400 not to be linked with the main camera 500 if the number of rapid PTZ operations in the main camera 500 exceeds a threshold. In S1503, CPU101 turns OFF the linked execution flag stored in RAM102 and terminates the process.

[0185] In S1504, CPU 101 determines whether the current linkage execution flag is OFF and whether a predetermined time has elapsed since the last time the linkage execution flag was turned OFF. If these conditions are met, it is determined that the PTZ speed of the main camera 500 is below the threshold, but the operation is not yet stable, and the sub-camera 400 is not linked to the main camera 500. Therefore, if CPU 101 determines that these conditions are met, it leaves the last linkage execution flag OFF and terminates the process. On the other hand, if it is determined that the current linkage execution flag is not OFF, or that a predetermined time has elapsed since the last time the linkage execution flag was turned OFF, CPU 101 executes S1505.

[0186] In S1505, CPU101 turns on the linkage execution flag if it is OFF, and terminates processing. In other words, in S1505, CPU101 decides to link subcamera 400 with main camera 500. Even if the linkage execution flag is OFF, if the elapsed time since the PTZ speed of main camera 500 fell below the threshold exceeds a predetermined time, there is a high probability that the operation is stable. Therefore, CPU101 decides to link subcamera 400 with main camera 500.

[0187] Figure 16 is a timing chart showing an example of a decision in S206 based on the PTZ speed of the main camera 500.

[0188] Figure 16(a) shows an example of a timing chart when the main camera 500 performs a gradual zoom-down and pan operation. As a visual effect, for example, a subject may be gradually zoomed down, or a gradual pan operation may be performed to film another subject located in the subject's line of sight. In this case, even if the sub-camera 400 is linked to the main camera 500, the shooting direction and field of view of the sub-camera 400 are controlled to change gradually, similar to the PTZ speed of the main camera 500. Therefore, there is no problem with the quality of the sub-camera 400's image. As a result, the user of the switcher 1000 can select the image from the sub-camera 400 as the preview image. Thus, in the example in Figure 16(a), by linking the sub-camera 400 with the main camera 50, the operator of the sub-camera 400 can be eliminated.

[0189] The relationship between the timing chart in Figure 16(a) and each step in the flowchart in Figure 15 will be explained. Here, we assume that initially, the sub-camera 400 is automatically controlled to work in conjunction with the main camera 500 (the linkage flag is ON).

[0190] At time 1600, the operator of main camera 500 begins a gradual zoom-down operation. In response to this operation, S1502 in Figure 15 determines that the PTZ speed does not exceed a predetermined threshold, and S1504 is executed. Since the linkage execution flag is ON, S1505 is executed, and the linkage execution flag remains ON without being changed. As a result, automatic control of sub-camera 400 continues in conjunction with main camera 500.

[0191] At time 1601, the zoom-down is completed, and the photographer of main camera 500 maintains the shooting direction and field of view. Because the PTZ speed is 0, S1502 determines that the PTZ speed does not exceed a predetermined threshold, and S1504 is executed. Since the linkage execution flag is ON, S1505 is executed, and the linkage execution flag remains ON without change. As a result, automatic control of sub-camera 400 in conjunction with main camera 500 continues.

[0192] At time 1602, the operator of main camera 500 begins a slow panning motion. At S1502, it is determined that the PTZ speed does not exceed a predetermined threshold, and S1504 is executed. Since the linkage execution flag is ON, S1505 is executed, and the linkage execution flag remains ON without being changed. As a result, automatic control of sub-camera 400 continues in conjunction with main camera 500.

[0193] At time 16:03, the panning operation is completed, and the camera operator of main camera 500 maintains the shooting direction and field of view. Because the PTZ speed is 0, S1502 determines that the PTZ speed does not exceed a predetermined threshold, and S1504 is executed. Since the linkage execution flag is ON, S1505 is executed, and the linkage execution flag remains ON without change. As a result, automatic control of sub-camera 400 in conjunction with main camera 500 continues.

[0194] Figure 16(b) shows an example of a timing chart for rapid zoom-down and zoom-up operations performed on the main camera 500. For example, such zoom operations may occur when the operator of the main camera 500 is trying to quickly find another subject of interest.

[0195] The relationship between the timing chart in Figure 16(b) and each step in the flowchart in Figure 15 will be explained. Here, we assume that initially, the sub-camera 400 is automatically controlled to work in conjunction with the main camera 500 (the linkage flag is ON).

[0196] At time 1610, the operator of main camera 500 begins a rapid zoom-down operation. In response to this operation, S1502 determines that the PTZ speed exceeds a predetermined threshold, and in S1503 the linkage execution flag is turned OFF. As a result, sub-camera 400 does not link with main camera 500 and starts automatic control to continue tracking the currently tracked subject.

[0197] At time 16:11, the operator of main camera 500 interrupts the operation to determine the next subject to focus on from the wide-angle image. As a result, S1502 determines that the PTZ speed does not exceed a predetermined threshold, and S1504 is executed. Since the elapsed time since the linkage execution flag was turned OFF is within the predetermined time, S1505 is not executed, and the linkage execution flag remains OFF and is not changed.

[0198] At time 1612, the photographer of main camera 500 begins a rapid zoom operation on the next potential subject of interest. In S1502, it is determined that the PTZ speed exceeds a predetermined threshold, and S1503 is executed, but since the linked execution flag is OFF, no change is made.

[0199] At time 1613, the zoom-up operation is completed, and the photographer of main camera 500 maintains the shooting direction and field of view. Because the PTZ speed is 0, S1502 determines that the PTZ speed does not exceed a predetermined threshold, and S1504 is executed. Since the elapsed time since the linkage execution flag was turned OFF is within the predetermined time, S1505 is not executed, and the linkage execution flag remains OFF and is not changed.

[0200] The shooting direction and field of view of the main camera 500 are maintained, and at time 1614, the elapsed time since the linkage execution flag was turned OFF exceeds a predetermined time. In S1502, it is determined that the PTZ speed does not exceed a predetermined threshold, and S1504 is executed. In S1504, since the elapsed time since the linkage execution flag was turned OFF exceeds a predetermined time, S1505 is executed, and the linkage execution flag is turned ON. As a result, automatic control is started to perform shooting operations for the sub-camera 400 in conjunction with the main camera 500.

[0201] Figure 16(c) shows an example of a timing chart when rapid panning is repeatedly performed with the main camera 500. For example, such panning may occur when the operator of the main camera 500 is trying to quickly find another subject of interest.

[0202] The relationship between the timing chart in Figure 16(c) and each step in the flowchart in Figure 15 will be explained. Here, we assume that initially, the sub-camera 400 is automatically controlled to work in conjunction with the main camera 500 (the linkage flag is ON).

[0203] At time 1620, the operator of main camera 500 repeatedly performs rapid panning operations. In response to this operation, S1502 determines that the PTZ speed exceeds a predetermined threshold, and in S1503 the synchronization flag is turned OFF. As a result, sub-camera 400 does not synchronize with main camera 500 and starts automatic control to continue tracking the currently tracked subject.

[0204] At time 16:21, the operator of main camera 500 interrupts the operation because they have identified the next subject to focus on. As a result, S1502 determines that the PTZ speed does not exceed a predetermined threshold, and S1504 is executed. Since the elapsed time since the linkage execution flag was turned OFF is within the predetermined time, S1505 is not executed, and the linkage execution flag remains OFF and is not changed.

[0205] The shooting direction and field of view of the main camera 500 are maintained, and at time 1622, the elapsed time since the linkage execution flag was turned OFF exceeds a predetermined time. In S1502, it is determined that the PTZ speed does not exceed a predetermined threshold, and S1504 is executed. In S1504, since the elapsed time since the linkage execution flag was turned OFF exceeds a predetermined time, S1505 is executed and the linkage execution flag is turned ON. As a result, automatic control is started to perform shooting operations for the sub-camera 400 in conjunction with the main camera 500.

[0206] In this embodiment, the CPU 101 determines whether or not to synchronize the sub-camera with the main camera based on the PTZ speed of the main camera 500. For example, if the PTZ speed of the main camera 500 exceeds a predetermined threshold, it may be searching for the next subject of interest, for instance, without considering the quality of the image. Therefore, the CPU 101 controls the sub-camera 400 to continue tracking and shooting the currently tracked subject without synchronizing its shooting operation with the main camera 500. On the other hand, if the PTZ speed of the main camera 500 does not exceed a predetermined threshold, the CPU 101 controls the sub-camera 400 to synchronize its shooting operation with the main camera 500. This makes it possible to stably provide high-quality images while operating the sub-camera 400 unattended.

[0207] ●<Third Embodiment> Next, a third embodiment of the present invention will be described. In this embodiment, as yet another example of a method for determining whether or not to link the sub-camera 400 with the main camera 500 in S206 of Figure 6(a), a determination method based on the location being captured by the main camera 500 will be described.

[0208] Figure 17 is a flowchart showing the operation of the CPU 101 as the linked execution decision unit 130 in this embodiment.

[0209] In this embodiment, it is assumed that the user has previously specified a specific range (here referred to as the shooting range) for the image from the overhead camera 300 via the user input unit 106. The shooting range information is stored in the RAM 102.

[0210] In S1601, the CPU 101, acting as the linked execution decision unit 130, acquires information ANGLE, which indicates the shooting direction, from the main camera 500 and stores it in the RAM 102.

[0211] In S1602, the CPU 101 reads the shooting range information and the information ANGLE from the RAM 102 and determines whether the shooting direction of the main camera 500 is within the shooting range.

[0212] Figure 18 is a schematic diagram illustrating an example where the shooting direction of the main camera 500 is outside the shooting range. In this example, the main camera 500 is shooting the audience seats. In live broadcasts, the operator of the main camera 500 may intentionally include footage of the audience in the shots of the stage. In such cases, the subject of focus and field of view of the main camera 500 change frequently, so it is better not to link the sub-camera 400 to the main camera 500.

[0213] Returning to Figure 17, in S1602, the CPU 101 executes S1603 if it determines that the shooting direction of the main camera 500 is facing within the shooting range, and executes S1604 otherwise.

[0214] In S1603, the CPU 101 turns on the linkage execution flag stored in RAM 102. That is, the CPU 101 decides to link the sub-camera 400 to the main camera 500.

[0215] In S1604, the CPU 101 turns OFF the linkage execution flag stored in RAM 102. In other words, the CPU 101 decides not to link the sub-camera 400 to the main camera 500.

[0216] In this embodiment, whether or not the main camera 500 is capturing the shooting range is determined based on the shooting range information and the shooting direction of the main camera 500. However, other methods may be used for this determination. For example, the main camera 500 may be determined by performing a shooting range detection process using a machine learning model or the like on the video feed from the main camera 500. This method eliminates the need to set the shooting range in advance.

[0217] In this embodiment, if the main camera 500 is not capturing a predetermined shooting range, the sub-camera 400 is controlled to continue tracking and shooting the currently tracked subject without being linked to the main camera 500. On the other hand, if the main camera 500 is capturing the shooting range, the sub-camera 400 is linked to the main camera 500. This makes it possible to provide high-quality video stably while operating the sub-camera 400 unattended.

[0218] ●<Fourth Embodiment> Next, a fourth embodiment of the present invention will be described. This embodiment relates to yet another example of a method for determining whether or not to link the sub-camera 400 with the main camera 500 in S206 of Figure 6(a). Specifically, it relates to a method for determining whether or not to link the sub-camera 400 with the main camera 500 for a predetermined period of time.

[0219] Figure 19 is a flowchart showing the operation of the CPU 101 as the linked execution decision unit 130 in this embodiment.

[0220] In S1901, the CPU 101, acting as the linked execution decision unit 130, obtains the identification information of the subject of interest, MAIN_SUBJECT, from the subject of interest determination unit 122 and stores it in the RAM 102. As mentioned above, the identification information MAIN_SUBJECT stores information that allows for the identification of the subject using a template matching method.

[0221] In S1902, the CPU 101 reads the identification information MAIN_SUBJECT from RAM 102 and determines whether or not there is a subject of interest. For example, if the main camera 500 is shooting with a wide-angle lens and all subjects are below a predetermined size, then there is no subject of interest. Alternatively, even if a subject of interest is detected, if it is not in a predetermined position (e.g., the center) on the screen, it may be determined that there is no subject of interest. For example, if the photographer of the main camera 500 is trying to shoot so that the subject of interest is in the center of the screen, and the subject of interest is not in the center of the screen, the sub-camera 400 may not be activated in conjunction with the camera, as the composition has not yet been determined. If the CPU 101 determines that there is a subject of interest, it executes S1903; otherwise, it executes S1905.

[0222] In S1905, the CPU 101 turns OFF the linkage execution flag stored in RAM 102. In other words, the CPU 101 decides not to link the sub-camera 400 to the main camera 500.

[0223] In S1903, the CPU 101 determines whether the main camera 500 has been continuously photographing the same subject for a predetermined period of time or longer. Specifically, the CPU 101 reads the identification information MAIN_SUBJECT from the RAM 102 and determines whether the subject of interest indicated by the identification information MAIN_SUBJECT has not changed for a predetermined period of time or longer. This predetermined period is set as the time at which the photographer of the main camera 500 can be considered to have confirmed the subject of interest.

[0224] In Figure 6(a), at S204, each time a subject of interest is determined, the CPU 101 stores the history of the identification information MAIN_SUBJECT in the RAM 102. At S1903, the CPU 101 determines that the main camera 500 has been continuously photographing the same subject for a predetermined period of time or longer, for example, if the same identification information MAIN_SUBJECT has been stored in the RAM 102 for a predetermined number of consecutive times or more. Alternatively, the CPU 101 may store the identification information MAIN_SUBJECT in association with the current time, and if the subject of interest does not change over a predetermined period of time, it may determine that the main camera 500 has been continuously photographing the same subject for a predetermined period of time or longer.

[0225] These are examples of determination methods, and other methods may be used. Similar to S1902, the position of the subject in the frame may be considered in the determination. That is, if the same subject is continuously present in a predetermined position in the frame for a predetermined time or longer, the main camera 500 may be determined to have been continuously photographing the same subject for a predetermined time or longer. Even if the same subject is continuously present in the frame for a predetermined time or longer, if its position in the frame is changing, the composition may be considered not to be determined, and the sub-camera 400 may not be activated.

[0226] The CPU 101 executes S1904 if it determines that the main camera 500 has been continuously photographing the same subject for a predetermined period of time or longer, and executes S1905 otherwise.

[0227] In S1904, the CPU 101 turns on the linkage execution flag stored in RAM 102. That is, the CPU 101 decides to link the sub-camera 400 to the main camera 500.

[0228] In S1905, the CPU 101 turns OFF the linkage execution flag stored in RAM 102. In other words, the CPU 101 decides not to link the sub-camera 400 to the main camera 500.

[0229] Figure 20 is a timing chart showing an example of a decision made in S206 based on whether or not there is a change in the subject of interest of the main camera 500. The relationship between the timing chart in Figure 20 and each step in the flowchart in Figure 19 will be explained below. Here, we assume that initially the sub-camera 400 is automatically controlled to work in conjunction with the main camera 500 (the linkage flag is ON).

[0230] Initially, the main camera 500 is recording subject A as the subject of focus. The switcher 1000 selects and outputs the video from the main camera 500. In other words, the status information (tally information) of the main camera 500 is "streaming". The sub-camera 400 is automatically controlled to perform shooting in conjunction with the main camera 500 according to its role.

[0231] At time 2000, switcher 1000 selects the image from sub-camera 400. Also, since the image from main camera 500 is no longer selected, the operator of main camera 500 begins operations to change the subject of focus. For example, the operator of main camera 500 rapidly zooms down to photograph the entire stage and, while viewing the wide-angle image, decides on the next subject of focus.

[0232] If the CPU 101 determines that the PTZ speed of the main camera 500 exceeds a predetermined threshold, as described in the second embodiment, for example, in S1902, it determines that there is no subject of interest. In S1902, the CPU 101 may also use other methods to determine that there is no subject of interest, such as determining that there is no subject of interest based on the fact that the subject of interest up to time 2000 is no longer detected in the image from the main camera 500. As a result, in S1905, the linkage execution flag is turned OFF, and the automatic control content of the sub-camera 400 is changed to perform shooting without linkage with the main camera 500.

[0233] At time 2001, the photographer of main camera 500 rapidly zooms in on subject B, and then begins to photograph subject B in close-up without any rapid operation. As a result, CPU 101 determines in S1902 that there is a subject of interest (subject B). However, since the predetermined time has not elapsed since subject B became the subject of interest, the determination in S1903 is No, and CPU 101 executes S1905. Consequently, the linked operation flag remains OFF.

[0234] If the photographer of the main camera 500 performs another rapid zoom-down operation at time 2002, before the predetermined time has elapsed from time 2001, the CPU 101 determines in S1902 that there is no subject of interest, and therefore the linkage operation flag remains OFF. Since there is no subject of interest before the predetermined time has elapsed since subject B became the subject of interest at time 2001, the sub-camera 400 continues shooting independently of the main camera 500.

[0235] At time 2003, the photographer of main camera 500 rapidly zooms in on subject C, and then begins to photograph subject C in close-up without any rapid operation. As a result, CPU 101 determines in S1902 that there is a subject of interest (subject C). However, since the predetermined time has not elapsed since subject C became the subject of interest, the determination in S1903 is No, and CPU 101 executes S1905. Consequently, the linked operation flag remains OFF.

[0236] At time 2004, after a predetermined time has elapsed from time 2003, the CPU 101 determines in S1903 that the main camera 500 is still focusing on subject C, and that the main camera 500 has been continuously photographing the same subject for a predetermined time or longer. As a result, the CPU 101 turns on the linked operation flag in S1904. Consequently, the automatic control is changed so that the sub-camera 400 performs shooting in conjunction with the main camera 500.

[0237] In this embodiment, whether or not the main camera 500 is continuously photographing the same subject of interest for a predetermined period of time is determined based on the identification information MAIN_SUBJECT. The identification information MAIN_SUBJECT indicates the subject of interest of the main camera 500, determined based on the image from the overhead camera 300 and the shooting direction of the main camera 500. Alternatively, the presence or absence of the main camera 500 may be determined by other methods. For example, a subject of interest detection process may be performed on the image from the main camera 500 using a machine learning model, and the presence or absence of the subject of interest may be determined based on the period during which the detection result is the same subject. In this case, the presence or absence of the subject of interest can also be determined using the results of the subject of interest detection process. In this case, the subject of interest can be detected from the image from the main camera 500 even if the shooting direction of the main camera 500 is not available.

[0238] In this embodiment, if the main camera 500 is not continuously capturing the same subject of interest for a predetermined period of time, the sub-camera 400 is controlled to continue tracking and capturing the currently tracked subject without being linked to the main camera 500. On the other hand, if the main camera 500 is continuously capturing the same subject of interest for a predetermined period of time, the sub-camera 400 is linked to the main camera 500. This makes it possible to provide high-quality video stably while operating the sub-camera 400 unattended.

[0239] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0240] This embodiment includes the following imaging control device and its control method, a multi-camera imaging system, and a program. (Item 1) A shooting control device that controls the shooting operation of a sub-camera among a plurality of cameras, including a main camera and one or more sub-cameras, The sub-camera has a control means for automatically controlling at least one of the shooting direction and field of view, Based on the information relating to the main camera, the control means A first control that automatically controls at least one of the shooting direction and field of view of the sub-camera based on the state or image of the main camera, A second control that automatically controls at least one of the shooting direction and field of view of the sub-camera, without relying on the state or image of the main camera, is selectively performed. A photographic control device characterized by the following features. (Item 2) The information regarding the main camera is whether or not the image from the main camera is selected for the external device. The control means executes the first control when the image from the main camera is selected for the external device, and executes the second control when the image from the main camera is not selected for the external device. The photographic control device according to item 1, characterized in that it is a photographic control device. (Item 3) The shooting control device according to item 2, characterized in that the external device is a video switcher that outputs a video selected from the videos captured by the plurality of cameras. (Item 4) The shooting control device according to item 3, characterized in that the control means determines whether or not the image from the main camera has been selected for an external device based on tally information supplied by the video switcher to the plurality of cameras. (Item 5) The information relating to the main camera is the pan, tilt, or zoom speed of the main camera. The control means executes the second control when the speed exceeds a predetermined threshold, and executes the first control when the speed does not exceed the predetermined threshold. The photographic control device according to item 1, characterized in that it is a photographic control device. (Item 6) The information relating to the main camera is one or more speeds of the pan, tilt, and zoom of the main camera. The control means executes the second control if the number of times the time change of one or more speeds exceeds a predetermined threshold within a predetermined period exceeds the threshold, and executes the first control if the number of times the time change of one or more speeds exceeds a predetermined threshold within the predetermined period does not exceed the threshold. The photographic control device according to item 1, characterized in that it is a photographic control device. (Item 7) The information relating to the main camera is the shooting direction of the main camera, The control means executes the first control when the shooting direction is directed within a predetermined shooting range, and executes the second control when the shooting direction is not directed within the shooting range. The photographic control device according to item 1, characterized in that it is a photographic control device. (Item 8) The information relating to the main camera is information relating to the subject of interest of the main camera, The control means executes the first control when it is determined that there is a subject of interest for the main camera, and executes the second control when it is determined that there is no subject of interest. The photographic control device according to item 1, characterized in that it is a photographic control device. (Item 9) The shooting control device according to item 8, characterized in that the case in which it is determined that there is no subject of interest is when the size of the subjects in the video captured by the main camera is all less than or equal to a predetermined size. (Item 10) The shooting control device according to item 8 or item 9, characterized in that the case in which it is not determined that there is a subject of interest is when the subject of interest is not in a predetermined position in the image captured by the main camera. (Item 11) The information relating to the main camera is information relating to the subject of interest of the main camera, The control means executes the first control if the main camera has been continuously photographing the same subject of interest for a predetermined time or longer, and executes the second control if the main camera has not been continuously photographing the same subject of interest for a predetermined time or longer. The photographic control device according to item 1, characterized in that it is a photographic control device. (Item 12) The shooting control device according to item 11, characterized in that the control means executes the second control if the subject of interest is not in the predetermined position for a predetermined time or longer in the image captured by the main camera. (Item 13) The shooting control device according to any one of items 1 to 12, characterized in that when the control means changes the automatic control of the sub-camera from the first control to the second control, in the second control, the shooting operation of the sub-camera is automatically controlled so as to continue tracking the subject that was being tracked when the first control was being performed. (Item 14) The shooting control device according to any one of items 1 to 13, characterized in that when the control means performs the first control, it automatically controls the shooting operation of the sub-camera according to the role set for the sub-camera. (Item 15) The shooting control device according to item 14, characterized in that the aforementioned role defines at least one of the relationship between the subject tracked and photographed by the sub-camera and the subject of interest of the main camera, and the relationship between the zoom operation of the sub-camera and the zoom operation of the main camera. (Item 16) Multiple cameras, including a main camera and one or more sub-cameras, A multi-camera imaging system comprising an imaging control device described in any one of items 1 to 15. (Item 17) A control method performed by a shooting control device that controls the shooting operation of a sub-camera, among a plurality of cameras including a main camera and one or more sub-cameras, The system includes the ability to automatically control at least one of the shooting direction and field of view of the sub-camera. The aforementioned automatic control is performed based on information regarding the main camera, A first control that automatically controls at least one of the shooting direction and field of view of the sub-camera based on the state or image of the main camera, This includes selectively performing a second control that automatically controls at least one of the shooting direction and field of view of the sub-camera, without relying on the state or image of the main camera. A control method for a shooting control device characterized by the above. (Item 18) A program for causing a computer to function as a control means of a photographic control device described in any one of items 1 to 15.

[0241] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0242] 100...Shooting control device, 300...Overhead camera, 400...Sub-camera, 500...Main camera, 600...Role control device, 1000...Video switcher, 101...CPU, 102...RAM, 103...ROM

Claims

1. A shooting control device that controls the shooting operation of a sub-camera among a plurality of cameras including a main camera and one or more sub-cameras, The sub-camera has a control means for automatically controlling at least one of the shooting direction and field of view, Based on the information relating to the main camera, the control means A first control that automatically controls at least one of the shooting direction and field of view of the sub-camera based on the state or image of the main camera, A second control that automatically controls at least one of the shooting direction and field of view of the sub-camera, without relying on the state or image of the main camera, is selectively performed. A photographic control device characterized by the following features.

2. The information regarding the main camera is whether or not the image from the main camera is selected for the external device. The control means executes the first control when the image from the main camera is selected for the external device, and executes the second control when the image from the main camera is not selected for the external device. The photographic control device according to claim 1.

3. The shooting control device according to claim 2, characterized in that the external device is a video switcher that outputs a video selected from the videos captured by the plurality of cameras.

4. The shooting control device according to claim 3, characterized in that the control means determines whether or not the image from the main camera has been selected for an external device based on tally information supplied by the video switcher to the plurality of cameras.

5. The information relating to the main camera is the pan, tilt, or zoom speed of the main camera. The control means executes the second control when the speed exceeds a predetermined threshold, and executes the first control when the speed does not exceed the predetermined threshold. The photographic control device according to claim 1.

6. The information relating to the main camera is one or more speeds of the pan, tilt, and zoom of the main camera. The control means executes the second control if the number of times the time change of one or more speeds exceeds a predetermined threshold within a predetermined period exceeds the threshold, and executes the first control if the number of times the time change of one or more speeds exceeds a predetermined threshold within the predetermined period does not exceed the threshold. The photographic control device according to claim 1.

7. The information relating to the main camera is the shooting direction of the main camera, The control means executes the first control when the shooting direction is directed within a predetermined shooting range, and executes the second control when the shooting direction is not directed within the shooting range. The photographic control device according to claim 1.

8. The information relating to the main camera is information relating to the subject of interest of the main camera, The control means executes the first control when it is determined that there is a subject of interest for the main camera, and executes the second control when it is determined that there is no subject of interest. The photographic control device according to claim 1.

9. The shooting control device according to claim 8, characterized in that the case in which it is determined that there is no subject of interest is when the size of the subjects in the video captured by the main camera is all less than or equal to a predetermined size.

10. The shooting control device according to claim 8, characterized in that the case in which it is not determined that there is a subject of interest is when the subject of interest is not in a predetermined position in the image captured by the main camera.

11. The information relating to the main camera is information relating to the subject of interest of the main camera, The control means executes the first control if the main camera has been continuously photographing the same subject of interest for a predetermined time or longer, and executes the second control if the main camera has not been continuously photographing the same subject of interest for a predetermined time or longer. The photographic control device according to claim 1.

12. The shooting control device according to claim 11, wherein the control means executes the second control if the subject of interest is not in the predetermined position for a predetermined time or longer in the image captured by the main camera.

13. The shooting control device according to claim 1, characterized in that when the control means changes the automatic control of the sub-camera from the first control to the second control, in the second control, it automatically controls the shooting operation of the sub-camera so as to continue tracking the subject that was being tracked when the first control was being performed.

14. The shooting control device according to claim 1, characterized in that when the control means performs the first control, it automatically controls the shooting operation of the sub-camera according to the role set for the sub-camera.

15. The shooting control device according to claim 14, characterized in that the aforementioned role defines at least one of the relationship between the subject tracked and photographed by the sub-camera and the subject of interest of the main camera, and the relationship between the zoom operation of the sub-camera and the zoom operation of the main camera.

16. Multiple cameras, including a main camera and one or more sub-cameras, A multi-camera imaging system comprising: an imaging control device according to any one of claims 1 to 15.

17. A control method executed by a shooting control device that controls the shooting operation of a sub-camera, among a plurality of cameras including a main camera and one or more sub-cameras, The system includes the ability to automatically control at least one of the shooting direction and field of view of the sub-camera. The aforementioned automatic control is performed based on information regarding the main camera, A first control that automatically controls at least one of the shooting direction and field of view of the sub-camera based on the state or image of the main camera, This includes selectively performing a second control that automatically controls at least one of the shooting direction and field of view of the sub-camera, without relying on the state or image of the main camera. A control method for a shooting control device characterized by the above.

18. A program for causing a computer to function as a control means of the imaging control device described in any one of claims 1 to 15.

Images