Imaging control device and imaging control method, and imaging system

Abstract

To provide an imaging control device and imaging control method that enable more intuitive remote control of the imaging direction of an imaging device. [Solution] This is a shooting control device that controls the operation of a second imaging device, which is one of a plurality of imaging devices including a first imaging device and a second imaging device, based on information about the first imaging device and set control content. When the first imaging device is changed, the shooting control device determines whether the difference between the position of the operating member for remotely controlling the shooting direction of the first imaging device and the position of the first imaging device after the change is below a threshold. If the shooting control device determines that the difference is not below the threshold, it disables remote control by the operating member until it determines that the difference is below the threshold.

Description

Technical Field

[0001] The present invention relates to a shooting control device, a shooting control method, and an imaging system, and particularly to a technique for controlling a plurality of imaging devices.

Background Art

[0002] Patent Document 1 describes an imaging system that divides a plurality of cameras into a main camera and a sub-camera and controls the sub-camera to photograph the same subject as the subject of the main camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the imaging system as described in Patent Document 1, the pan and tilt operations of the main camera are remotely controlled through operations of a joystick or a GUI displayed on the screen. However, such remote operations feel different from the case of directly operating the pan / tilt head on which the main camera is mounted. Therefore, it was necessary to get used to it in order to control the shooting direction as intended. Such a difference in the operation feeling can be particularly problematic when trying to realize shooting in an environment where the camera is originally operated on site, such as studio shooting, by remote operation.

[0005] In view of such problems, in one aspect, the present invention provides a shooting control device and a shooting control method that can remotely operate the shooting direction of an imaging device more intuitively.

Means for Solving the Problems

[0006] In one embodiment, the present invention provides an imaging control device comprising: a control means for controlling the operation of a second imaging device among a plurality of imaging devices including a first imaging device and a second imaging device, based on information relating to the first imaging device and set control content; and a determination means for determining whether the difference between the orientation of an operating member for remotely controlling the shooting direction of the first imaging device and the orientation of the first imaging device after the change is less than or equal to a threshold when the first imaging device is changed, wherein if the control means determines that the difference is not less than or equal to a threshold, it disables remote control by the operating member until it is determined that the difference is less than or equal to a threshold. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to provide an imaging control device and an imaging control method that enable more intuitive remote control of the imaging direction of an imaging device. [Brief explanation of the drawing]

[0008] [Figure 1] Schematic diagram of the imaging system according to the first embodiment [Figure 2] Block diagram showing an example of the functional configuration of each device in the imaging system according to the first embodiment. [Figure 3] This diagram illustrates the imaging control device according to the embodiment, focusing on its main operations and signal flow. [Figure 4] This figure shows examples of roles and control contents that can be set for a sub-camera in this embodiment. [Figure 5] Flowchart relating to the role determination process in the first embodiment [Figure 6] Flowchart relating to the operation of each device in the imaging system according to the embodiment [Figure 7] A diagram illustrating the coordinate transformation in the embodiment. [Figure 8] Figure relating to subject detection and coordinate transformation in the embodiment [Figure 9] Schematic diagram of the operation control of the sub-camera in the first embodiment. [Figure 10] Schematic diagram of another operation control of the sub-camera in the first embodiment [Figure 11] Figure for explaining the calculation of the bread angle in the embodiment [Figure 12] Figure for explaining the calculation of the tilt angle in the embodiment [Figure 13] Figure showing an example of the mapping of the zoom values between the main camera and the sub - camera in the first embodiment [Figure 14] Flowchart regarding the determination process of the control content according to the role of the sub - camera in the first embodiment [Figure 15] Schematic diagram of the control according to the role of the sub - camera in the first embodiment [Figure 16] Schematic diagram of the control according to the role of the sub - camera in the first embodiment [Figure 17] Schematic diagram showing an example of the appearance of the remote control device in the first embodiment [Figure 18] Figure showing an example of the directions of each camera before and after the role change [Figure 19] Figure for explaining the inconsistency between the postures of the main camera and the operation unit [Figure 20] Flowchart regarding the operation when changing the main camera in the first embodiment [Figure 21] Timing chart regarding the operation when changing the main camera in the first embodiment [Figure 22] Flowchart regarding the operation during the invalid period of the remote control device in the second embodiment [Figure 23] Figure showing an example of the display in the second embodiment

Mode for Carrying Out the Invention

[0009] Hereinafter, the present invention will be described in detail based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Also, although a plurality of features are described in the embodiments, not all of them are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant explanations are omitted.

[0010] ● <First Embodiment> (Overview of Multi-Camera Imaging System) FIG. 1 is a schematic diagram showing a configuration example of a multi-camera imaging system 10 (hereinafter simply referred to as an imaging system) according to the present embodiment. The imaging system 10 includes a plurality of cameras 300, 400, 500, a shooting control device 100, a role control device 600, and a remote control device 800. The plurality of cameras 300, 400, 500, the shooting control device 100, the role control device 600, and the remote control device 800 are communicably connected through a communication network 700.

[0011] The communication network 700 complies with known wired or wireless communication standards such as the IEEE802.3 series and the 1EEE802.11 series. Each of the plurality of cameras 300, 400, 500, the shooting control device 100, the role control device 600, and the remote control device 800 has a communication interface compliant with the standard of the communication network 700.

[0012] Among the plurality of cameras 300, 400, 500, the camera 300 captures the entire predetermined shooting range. The shooting range is set, for example, as a range in which a subject to be photographed may exist in a studio. Therefore, all the subjects within the shooting range are captured in the video of the camera 300.

[0013] The purpose of camera 300 is to capture images for detecting subjects within the shooting range. Therefore, the shooting direction and field of view of camera 300 are determined according to the position of camera 300 and the shooting range, and are basically fixed during shooting. Furthermore, it is preferable that camera 300 captures the entire shooting range without it being obscured by objects outside the shooting range. For this reason, camera 300 is positioned to overlook the entire shooting range. To distinguish it from other cameras 400 and 500, whose shooting direction and field of view are not basically fixed during shooting, camera 300 will be referred to as the overhead camera below. However, the installation position of camera 300 is not limited to a position that overlooks the shooting range. The operation of the overhead camera 300 can be controlled from the shooting control device 100.

[0014] Cameras 400 and 500 are, for example, PTZ cameras, and their operation, including shooting direction (pan and tilt angles) and field of view (zoom), can be controlled from an external device. Alternatively, cameras 400 and 500 may be configured as, for example, video cameras capable of remote operation, including zoom, mounted on an electric pan / tilt head capable of remote control of pan and tilt angles.

[0015] This example shows a configuration with two cameras other than the overhead camera 300 (cameras 400 and 500), but there may be three or more cameras. Of the multiple cameras other than the overhead camera 300, one is set as the main camera (first imaging device) to be remotely controlled by the remote control device 800. The remaining cameras are set as sub-cameras (second imaging devices) whose operation is controlled by the shooting control device 100 based on the state of the remotely controlled camera. Therefore, each of the multiple cameras other than the overhead camera 300 can be remotely controlled by either the remote control device 800 or the shooting control device 100.

[0016] Cameras 400 and 500 can both be set as the main camera. Furthermore, the main camera setting can be changed dynamically. Therefore, in the following, cameras 400 and 500 will be referred to as either the main camera or the sub-camera, depending on their current settings.

[0017] For multiple cameras other than the overhead camera 300, the main camera will be configured by the role control device 600. The main camera configuration may be performed by the operator of the role control device 600, or it may be performed automatically by the role control device 600 according to predetermined conditions, such as sequentially switching the main camera at regular intervals. The main camera configuration (e.g., unique information of the main camera, IP address, etc.) is notified from the role control device 600 to the remote control device 800 and the shooting control device 100, respectively.

[0018] For the purposes of the following explanation, it will be assumed that camera 500 is set as the main camera and camera 400 is set as the sub-camera.

[0019] In this embodiment, the role control device 600 and the remote control device 800 are assumed to have operators. The shooting control device 100 may also have an operator, but is not required. The same operator may be responsible for operating multiple devices. The shooting of the overhead camera 300 and the sub-camera 400 is controlled by the shooting control device 100, so a photographer is not required. The shooting of the main camera 500 is remotely controlled by the operator of the remote control device 800.

[0020] Although Figure 1 shows all signals being communicated via the communication network 700, video signals and control signals may be communicated in different ways. For example, each of the multiple cameras 300, 400, and 500 may directly supply video signals to the shooting control device 100 and the remote control device 800 via cables. In this case, the cameras 300, 400, 500, the shooting control device 100, and the remote control device 800 have communication circuits corresponding to the video signal standard. Video signal standards include, but are not limited to, the SDI (Serial Digital Interface) standard and HDMI (High-Definition Multimedia Interface) (registered trademark).

[0021] The remote control device 800 remotely controls the shooting direction (pan and tilt angles) of the main camera (camera 500 in this case), as set by the role control device 600, via the shooting control device 100. The remote control device 800 may also remotely control the zoom operation of the main camera.

[0022] The shooting control device 100 detects a subject from the video signal received from the overhead camera 300. Based on the subject detection result, the status of the main camera 500, and the role set for the sub-camera, the shooting control device 100 determines the shooting direction (pan and tilt angles) and field of view (zoom value) of the sub-camera. The shooting control device 100 transmits a control command including the determined shooting direction and field of view to the sub-camera 400. By changing the role setting, the method for determining the shooting direction and field of view of the sub-camera 400 can be changed, thereby increasing the degree of freedom in controlling the operation of the sub-camera 400.

[0023] The shooting control device 100 also acquires attitude information from the remote control device 800 and controls the shooting direction of the main camera. It also acquires zoom operation information from the remote control device 800 and controls the field of view of the main camera.

[0024] (Examples of functional configurations for each device) Figure 2 is a block diagram showing an example of the functional configuration of each component constituting the multi-camera imaging system shown in Figure 1. The configurations represented as functional blocks in the diagram can be implemented using integrated circuits such as ASICs and FPGAs, discrete circuits, or a combination of memory and a processor that executes the program stored in memory. Furthermore, one functional block may be implemented using multiple integrated circuit packages, or multiple functional blocks may be implemented using a single integrated circuit package. Additionally, the same functional block may be implemented in different configurations depending on the operating environment and required capabilities.

[0025] Note that the functional configurations of cameras 400 and 500 are identical, so only camera 400 is shown in Figure 2, but camera 500 also exists in reality.

[0026] (Shooting control device 100) First, an example of the functional configuration of the imaging control device 100 will be described. The imaging control device 100 may be a general-purpose computer device such as a personal computer or a workstation. The imaging control device 100 has a configuration in which a CPU 101, RAM 102, ROM 103, inference unit 104, network interface (I / F) 105, user input unit 106, and display unit 108 are interconnected via an internal bus 110.

[0027] The CPU 101 is a microprocessor capable of executing programmed instructions. For example, the CPU 101 realizes the functions of the imaging control device 100, which will be described later, by reading a program stored in the ROM 103 into the RAM 102 and executing it. The CPU 101 can also realize the functions of the imaging control device 100 by executing an imaging control application that runs on the operating system (OS), for example.

[0028] RAM 102 is used to load programs executed by CPU 101, and to temporarily store data processed by CPU 101, data being processed, etc. A portion of RAM 102 may also be used as video memory for the display unit 108.

[0029] ROM103 is a rewritable, non-volatile memory that stores programs executed by the CPU101 (OS and applications), user data, and other similar information.

[0030] The inference unit 104 performs object region detection processing using a machine learning model on the video from the overhead camera 300. The inference unit 104 can be implemented using hardware circuits capable of high-speed execution of machine learning model calculations, such as a GPU (Graphics Processing Unit) or an NPU (Neural Network Processing Unit). Alternatively, the inference unit 104 may be implemented using reconfigurable logic circuits such as an FPGA (Field-Programmable Gate Array). The CPU 101 may execute a program to implement the functions of the inference unit 104.

[0031] The machine learning model may be a convolutional neural network (CNN) trained according to the type of subject to be detected. Here, the inference unit 104 detects human body regions or human face regions as subject regions from the input image. The inference unit 104 also outputs the position and size of the rectangular region inscribed in the subject region, and the detection confidence level for each detected subject region. Multiple types of machine learning models may be used to perform detection processing of different types of subject regions on the same input image. The inference unit 104 may also perform subject region detection processing using known methods that do not use machine learning models. For example, the inference unit 104 can detect subject regions using methods that use local features or methods that use pattern matching. Examples of methods that use local features include SIFT (Scale Invariant Feature Transform), SURF (Speed-Upped Robust Feature), ORB (Oriented-BRIEF), and AKAZE (Accelerated KAZE).

[0032] The network interface 105 is an interface for connecting the image capture control device 100 to the communication network 700. The image capture control device 100 (CPU 101) can communicate with external devices on the communication network 700, such as cameras 300-500, role control device 600, and remote control device 800, via the network interface 105. The image capture control device 100 may also communicate with any external devices, including the devices shown in Figure 1, via other communication interfaces (USB, Bluetooth®) not shown.

[0033] The CPU 101 communicates with each device on the communication network 700 (cameras 300-500, role control device 600, remote control device 800) by acquiring the address information (e.g., IP address) of each device at any time and storing it in RAM 102. The CPU 101 also acquires information about each device (device type, model name, etc.) at any time (e.g., during the initial communication) and stores it in RAM 102. Thus, the CPU 101 assumes that at least the identification information, communication information, and device type of each camera 300-500, role control device 600, and remote control device 800 are known. The user may be allowed to assign arbitrary names to each device.

[0034] The user input unit 106 is an input device (not shown) such as a mouse, keyboard, or touch panel. The shooting control device 100 receives user instructions through the user input unit 106.

[0035] The display unit 108 is a display device such as a liquid crystal display (LCD). The display unit 108 displays a GUI screen provided by the OS or shooting control application.

[0036] (Overhead camera 300) Next, we will explain an example of the functional configuration of the overhead camera 300. The CPU 301 is a microprocessor capable of executing programmed instructions. For example, the CPU 301 controls the operation of each functional block by loading a program stored in the ROM 303 into the RAM 302 and executing it, thereby realizing the functions of the overhead camera 300 described later.

[0037] RAM302 is used to load programs executed by CPU301, and to temporarily store data processed by CPU301, data being processed, etc. RAM302 may also be used as a buffer for video signals obtained during capture.

[0038] ROM303 is a rewritable, non-volatile memory. ROM303 stores programs executed by the CPU301, settings for the overhead camera 300, user data, and other similar information. ROM303 can also be used as a recording destination for video signals. ROM303 may include both internal memory and a removable memory card.

[0039] The image sensor 307 comprises an imaging optical system and an image sensor. The image sensor may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The image sensor has a pixel array in which multiple pixels are arranged in two dimensions, and peripheral circuits for reading signals from each pixel. Each pixel accumulates charge according to the amount of incident light by photoelectric conversion. By reading signals with a voltage corresponding to the amount of charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface is obtained.

[0040] The image processing unit 306 applies predetermined signal processing and image processing to the analog image signal output by the image sensor 307 to generate signals and image data according to the application, and to acquire and / or generate various types of information.

[0041] The processing applied by the image processing unit 306 may include, for example, preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effects processing. Preprocessing may include A / D conversion, signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is performed when a color filter is provided on the image sensor 307, and it is a process that interpolates the values ​​of color components that are not included in the individual pixel data that make up the image data. Color interpolation is also called demosaicing. Correction processing may include white balance adjustment, gradation correction, correction of image degradation caused by optical aberrations in the imaging optical system (image recovery), correction of the effects of vignetting in the imaging optical system, and color correction. Data processing may include processes such as region extraction (trimming), merging, scaling, encoding and decoding, and header information generation (data file generation). The generation of video signals to be output externally and video data to be recorded in ROM308 are also included in data processing. The evaluation value calculation process may include generating signals and evaluation values ​​used for autofocus detection (AF), and generating evaluation values ​​used for automatic exposure control (AE). AF and AE are executed by CPU 301. Special effects processing may include adding blur effects, changing color tones, and relighting. These are merely examples of processes that the image processing unit 306 can apply, and do not limit the processes that the image processing unit 306 can apply. The image processing unit 306 outputs the acquired or generated information and data to the CPU 301, RAM 302, etc., depending on the application.

[0042] Furthermore, the type and settings of processing applied by the image processing unit 306 can be controlled by sending commands from the shooting control device 100 to the overhead camera 300.

[0043] The network interface 305 is an interface for connecting the overhead camera 300 to the communication network 700. The overhead camera 300 (CPU 301) can communicate with external devices on the communication network 700, such as the shooting control device 100, cameras 400-500, role control device 600, and remote control device 800, via the network interface 305. The overhead camera 300 may also communicate with any external devices, including the devices shown in Figure 1, via other communication interfaces (not shown, such as USB or Bluetooth).

[0044] (Camera 400) Next, an example of the functional configuration of camera 400 will be described. As mentioned above, cameras 400 and 500 may have the same configuration. Therefore, the configuration of camera 400 will be described as representative. In this specification, reference numerals 501 to 510 indicate the components of camera 500 that correspond to components 401 to 410 of camera 400. Among the components of camera 400, functional blocks with the same name as those of camera 300 are assumed to have the same function and their explanation will be omitted.

[0045] Camera 400 is a camera whose shooting direction (pan and tilt angles) and field of view (zoom value) can be remotely controlled. Therefore, camera 400 has a drive unit 409 that can perform pan and tilt operations and zoom operations, and a drive I / F 408. The drive I / F 408 is a communication interface between the drive unit 409 and the CPU 401.

[0046] The drive unit 409 includes a pan / tilt mechanism that supports the sub-camera 400 so that it can be panned and tilted, a zoom mechanism that changes the angle of view of the imaging optical system, and motors that drive these mechanisms. The zoom mechanism may use image enlargement and reduction by the image processing unit 406. The drive unit 409 drives the motors according to instructions received from the CPU 401 via the drive I / F 408 to adjust the optical axis direction (shooting direction) and angle of view of the imaging optical system. The pan / tilt mechanism of the drive unit 409 may be an electric pan / tilt head on which the camera 400 is mounted.

[0047] The CPU 401 controls the pan / tilt mechanism of the drive unit 409 via the drive I / F 408 according to pan and / or tilt commands received via the network I / F 405. The CPU 401 also controls the zoom mechanism of the drive unit 409 via the drive I / F 408 according to zoom commands received via the network I / F 405.

[0048] The shooting control device 100 (CPU 101) can acquire information on the shooting direction and field of view of cameras 400 and 500 from cameras 400 and 500 via the network interface 105. The shooting direction may be the pan and tilt angles of the drive units 409 and 509, with a predetermined reference direction set at 0°. The reference direction may be the direction directly facing the shooting range.

[0049] Cameras 400 and 500 output the captured video to an external device (not shown), such as a switcher. The switcher selects and outputs one of the video feeds from cameras 400 and 500.

[0050] In addition, cameras 400 and 500 output video for live view display to the shooting control device 100 and the remote control device 800, separate from the video output to the switcher. If cameras 400 and 500 are configured to operate according to whether they are set as the main camera or the sub-camera, only the main camera needs to output video for live view display to the remote control device 800.

[0051] (Role control device 600) Next, we will describe an example of the functional configuration of the role control device 600. The CPU 601 is a microprocessor capable of executing programmed instructions. For example, the CPU 601 controls the operation of each functional block and realizes the functions of the role control device 600 by reading a role setting program stored in the ROM 603 into the RAM 602 and executing it.

[0052] RAM 602 is used to load programs executed by CPU 601, and to temporarily store data processed by CPU 601, data being processed, etc. A portion of RAM 602 may also be used as video memory for the display unit 608.

[0053] ROM603 is a rewritable non-volatile memory that stores programs executed by CPU601, settings of role control unit600, user data, and other similar information.

[0054] The user input unit 611 is an input device such as a button, dial, joystick, or touch panel. The role control device 600 receives user instructions regarding the setting of the role of the sub-camera 400 through the user input unit 611.

[0055] The network interface 605 is an interface for connecting the role control device 600 to the communication network 700. The role control device 600 (CPU 601) can communicate with external devices on the communication network 700, such as cameras 300-500, the shooting control device 100, and the remote control device 800, via the network interface 605. The role control device 600 may also communicate with any external devices, including the devices shown in Figure 1, via other communication interfaces (USB, Bluetooth, etc.) not shown.

[0056] The display unit 608 is a display device such as a liquid crystal display (LCD). The display unit 608 displays a GUI screen provided by the OS or a role setting application.

[0057] The role control device 600 stores role setting information, for example, in the ROM 603. The role setting information is information that associates the identification information of cameras 400 and 500 (address information, user-set names, etc.) with information indicating the assigned role. The CPU 601 displays the role setting screen on the display unit 608 by executing a role setting application. The role setting screen displays, for example, the identification information of cameras 400 and 500 and the name of the currently assigned role in association with each other. The initial value of the currently assigned role may be a pre-set default role. The user can change the currently assigned role displayed for cameras 400 and 500 by operating the user input unit 611.

[0058] When the CPU 601 detects a user action indicating the completion of a setting operation, such as pressing the OK button on the role setting screen, it updates the role setting information stored in the ROM 103 according to the contents of the role setting screen. The roles that can be set for cameras 400 and 500 include the types of main cameras and sub-cameras, and for sub-cameras, it further includes the type of operation control by the control device 100. Details of the roles will be described later.

[0059] When CPU601 receives a role acquisition command via network I / F605, it reads the role setting information stored in ROM103 and sends it to the source of the role acquisition command.

[0060] Although Figures 1 and 2 depict the role control device 600 as an independent device, a shooting control application executed by the shooting control device 100 may provide similar functionality to the role control device 600. Alternatively, roles may be directly assigned to cameras 400 and 500, and the shooting control device 100 may acquire the roles assigned to cameras 400 and 500 from the cameras themselves.

[0061] Next, referring to Figure 4, we will explain the roles that can be set for cameras other than the overhead camera 300 (cameras 400 and 500). The role indicates whether it is a main camera or a sub-camera, and in the case of a sub-camera, what kind of control the shooting control device 100 will perform (automatic control content). Figure 4 shows an example in which the shooting control device 100 controls the tracking subject and zoom operation of the sub-camera. However, the content that the shooting control device 100 automatically controls is not limited to tracking subjects and zoom operation.

[0062] For example, the table shown in Figure 4 can be stored in the ROM 603 of the role control device 600 and the ROM 103 of the image capture control device 100. Note that the image capture control device 100 is not involved in the operation of the main camera. Therefore, the role control device 600 and the image capture control device 100 only need to store information related to the sub-camera.

[0063] Here, the control content (CAMERA_ROLE) for the sub-camera can be set to one of the following: "Main Follow," "Main Counter," "Assist Follow," or "Assist Counter." If there are multiple sub-cameras, the control content can be set for each sub-camera. In the following, a sub-camera with the control content "Main Follow" set will be referred to as the camera with the role sub (main follow) set. Other control content will be described similarly.

[0064] A camera whose role is set to "main" (in this case, camera 500) is treated as the main camera by the shooting control device 100 and the remote control device 800.

[0065] For cameras with the role "Sub (Main Follow)" set, the shooting control device 100 (CPU 101) sets the same tracking subject as the main camera. Furthermore, if the angle of view of the main camera changes, the shooting control device 100 applies zoom control to the controlled camera that is in phase with the main camera. Here, "in phase" means that the direction of zoom (telephoto direction or wide-angle direction) is the same, that is, the direction of the angle of view change is the same. Therefore, when the shooting control device 100 detects that the main camera has zoomed in, it controls the controlled camera to zoom in. Zooming in means changing the angle of view in the telephoto direction (telephoto end direction).

[0066] On the other hand, opposite phase means that the direction of zoom (telephoto or wide-angle) is opposite, that is, the direction of the angle of view change is opposite. Therefore, when the shooting control device 100 detects that the main camera is zooming up, it controls the camera being controlled to zoom down. Zooming down means changing the angle of view towards the wide-angle direction (wide-angle end direction).

[0067] Furthermore, for zoom control of the sub-camera, only the phase is specified, and the field of view of the main camera and the sub-camera do not need to be equal. Also, in both in-phase and out-of-phase zoom control, the degree of change in the field of view of the controlled camera (such as the rate of change) does not need to match the degree of change in the field of view of the main camera.

[0068] Furthermore, when zoom control is performed by scaling the image using the image processing unit 406, zooming in can be achieved by reducing the area to be cut out from the image and increasing the magnification of the cut-out area compared to before the area was changed. Similarly, zooming down can be achieved by increasing the area to be cut out from the image and decreasing the magnification of the cut-out area compared to before the area was changed.

[0069] For cameras with the role "Sub (Main Counter)" set, the shooting control device 100 (CPU 101) sets the same tracking subject as the main camera. In addition, if the angle of view of the main camera changes, the shooting control device 100 applies zoom control that is in the opposite phase to that of the main camera to the controlled camera.

[0070] For cameras with the role "Sub (Assist Follow)" set, the shooting control device 100 (CPU 101) sets a different tracking subject from the main camera 500. Furthermore, if the angle of view of the main camera changes, the shooting control device 100 applies zoom control to the controlled camera that is in the same phase as the main camera.

[0071] For cameras with the role "Sub (Assist Counter)" set, the shooting control device 100 (CPU 101) sets a different tracking subject from the main camera 500. Furthermore, if the angle of view of the main camera changes, the shooting control device 100 applies a zoom control to the controlled camera that is in the opposite phase to that of the main camera.

[0072] Here, sub-cameras with the roles "Sub (Assist Follow)" and "Sub (Assist Counter)" set will track subjects located to the left of the main camera's subject of interest in the image. Note that the tracked subjects for sub-cameras with these roles may be set according to other conditions. For example, subjects located to the right, above, or below the main camera's subject of interest in the image may be set as the tracked subjects. Alternatively, the subject that is closest or furthest from the main camera's subject of interest may be set as the tracked subject.

[0073] Furthermore, the shooting control device 100 may perform only one of the following: setting the tracked subject or zoom control, or it may control other items.

[0074] The role setting information stored in the ROM 603 by the role control device 600 includes information indicating the role (such as the name of the type or the number assigned to the type), which is associated with the identification information of each camera other than the overhead camera 300. The CPU 101 of the shooting control device 100 obtains the role setting information from the role control device 600 and executes operation control for the sub-camera according to the set control content.

[0075] As described above, the role control device 600 notifies the shooting control device 100 and the remote control device 800 if there is a change in either the settings of the main camera or the settings of the control content of the sub-camera. This allows the change in the role settings to be reflected in the operation of the shooting control device 100 and the controlled objects of the remote control device 800.

[0076] (Remote control device 800) Next, the remote control device 800 will be described. First, referring to Figure 2, the functional configuration of the remote control device 800 will be explained. The remote control device 800 functions as a remote controller that remotely controls the shooting direction of the main camera.

[0077] The remote control device 800 may have a configuration in which an operation unit, described later, is connected to a small general-purpose computer device, such as a tablet computer, in a communication manner. The remote control device 800 has a configuration in which a CPU 801, RAM 802, ROM 803, network interface (I / F) 805, user input unit 811, display unit 808, and attitude calculation unit 807 are interconnected via an internal bus 810.

[0078] The CPU 801 is a microprocessor capable of executing programmed instructions. For example, the CPU 801 can implement the functions of the remote control device 800, described later, by loading a program stored in the ROM 803 into the RAM 802 and executing it. The CPU 801 can also implement the functions of the remote control device 800 by, for example, executing a shooting control application that runs on the operating system (OS).

[0079] RAM 802 is used to load programs executed by CPU 801, and to temporarily store data processed by CPU 801, data being processed, etc. A portion of RAM 802 may also be used as video memory for the display unit 808.

[0080] ROM803 is a rewritable, non-volatile memory that stores programs executed by the CPU801 (OS and applications), user data, and other similar information.

[0081] The network interface 805 is an interface for connecting the remote control unit 800 to the communication network 700. The remote control unit 800 (CPU 801) can communicate with external devices on the communication network 700, such as cameras 300-500, role control device 600, and shooting control device 100, via the network interface 805. The remote control unit 800 may also communicate with any external devices, including the devices shown in Figure 2, via other communication interfaces (USB, Bluetooth®) not shown.

[0082] The CPU 801 receives information (such as an IP address) from the role control device 600 via the network interface 805 to identify the camera with the role "main" assigned to it (the main camera). Subsequently, the CPU 801 transmits the attitude and operation values ​​of the operation unit 812 to the shooting control device 100, thereby remotely controlling the shooting direction and field of view of the main camera via the shooting control device 100. In addition to the shooting direction and field of view, the remote control device 800 may also remotely control other operations of the main camera (for example, starting and stopping shooting, turning the power on and off, changing various settings, etc.) via the shooting control device 100.

[0083] The user input unit 811 is an input device such as a mouse, keyboard, or touch panel. The CPU 801 receives user instructions through the user input unit 811.

[0084] The display unit 808 is a display device such as a liquid crystal display (LCD). The display unit 808 displays the video being captured by the main camera (live view video). The display unit 808 also displays GUI screens provided by the OS or shooting control applications.

[0085] When the remote control device 800 receives video for live view display from all cameras other than the overhead camera 300 (in this case, cameras 400 and 500), the CPU 801 selects the video from the current main camera and displays it on the display unit 808.

[0086] The attitude sensor 806 detects the movement of the operating unit 812 and outputs it to the attitude calculation unit 807. The attitude sensor 806 may be, for example, a SLAM (Simultaneous Localization and Mapping) module or a combination of a gyroscope and an accelerometer. The attitude sensor 806 outputs motion signals to the attitude calculation unit 807 that indicate translational motion in each axis direction of a three-dimensional Cartesian coordinate system including the direction of gravity, and rotational motion around each axis.

[0087] The attitude calculation unit 807 calculates the pan angle and tilt angle corresponding to the attitude of the control unit 812 from the motion signal output by the attitude sensor 806. The attitude calculation unit 807 can calculate the pan angle and tilt angle using general self-position estimation techniques. The attitude calculation unit 807 stores the calculated pan angle and tilt angle as attitude information (ORIENTATION) in the RAM 802. Note that the attitude information is not limited to the pan angle and tilt angle, but may be any value that can identify the attitude of the control unit 812.

[0088] The attitude calculation unit 807 periodically calculates the pan angle and tilt angle. A shorter calculation period is preferable from the viewpoint of suppressing the time lag between the movement of the operation unit 812 and the attitude change of the main camera. In practice, the calculation period can be determined by considering the speed of typical pan and tilt operations, the resolution of the attitude sensor 806, and the drive resolution of the main camera's pan and tilt.

[0089] When the posture information is stored in the RAM 802, the CPU 801 transmits it to the imaging control device 100 via the network I / F 805.

[0090] If the operation unit 812 has an input device for zoom control, the CPU 801 periodically detects the amount and direction of operation of the input device. The detection period may be the same as the calculation period of the attitude calculation unit 807. The CPU 801 then calculates a zoom value (ZOOM) according to the detected amount and direction of operation and transmits it to the shooting control device 100 via the network I / F 805. The CPU 801 may also transmit the attitude information (ORIENTATION) and the zoom value (ZOOM) together to the shooting control device 100.

[0091] Figure 17 is a schematic diagram showing an example of the external appearance of the operating unit 812 of the remote control device 800. The operating unit 812 may have an operating unit in the form of a tripod 811 equipped with a pan / tilt head 809, which is an example of an operating member capable of physical panning and tilting. The operating member can change the physical angle in the panning and tilting directions and can maintain the changed angle (attitude). The operating unit 812 may also have an input member (such as a ring-shaped member or a slider switch) for controlling the field of view of the main camera. Alternatively, the direction and amount of movement of the entire operating unit may be detected and used to control the field of view.

[0092] The control unit 812 is fitted with a display unit 808 for displaying the live view image from the main camera. The display unit 808 is mounted such that, for example, when the pan and tilt angles of the tripod head 809 are at their initial values ​​(0°), the screen is positioned directly in front (the pan angle of the screen becomes 0°).

[0093] A user remotely controlling the main camera can, for example, use the pan bar 810 provided on the tripod head 809 to perform pan and tilt operations so that the live view image displayed on the display unit 808 becomes the desired image. In this way, the remote control device 800 makes it possible to remotely control the pan and tilt of the main camera with the same feeling as operating the main camera's tripod head.

[0094] Here, as a typical example, we have described the hardware that constitutes the control unit 812 of the remote control device 800, using a combination of tripod 811 and pan / tilt head 809. However, the configuration and form of the control unit are arbitrary, as long as it is a control unit that the user can physically manipulate the orientation of, and that the orientation of the control unit can be substantially synchronized with the orientation of the main camera.

[0095] Furthermore, the orientation of the control unit 812 and the orientation of the main camera do not necessarily have to be identical, and may have a constant offset within a threshold. In this embodiment, the main camera remotely controlled by the remote control device 800 may be dynamically changed. Therefore, for each of the multiple cameras in the imaging system, excluding the overhead camera 300, calibration is performed so that the offset between the orientation of the control unit 812 and the orientation of the camera is the same in the initial state.

[0096] <Explanation of the operation of each device> Next, the operation of each device in the multi-camera imaging system will be explained. Here, the shooting control device 100 will automatically control the shooting operation of the sub-camera 400 based on the image from the overhead camera 300, the information obtained from the main camera 500, and the role set for the sub-camera 400.

[0097] Figure 3 is a diagram illustrating the series of processes performed by the shooting control device 100 when controlling the operation of the sub-camera 400, focusing on the main operations and signal flows. The functional blocks shown within the shooting control device 100 schematically represent the main operations and correspond to the main functions provided by the shooting control application. Each functional block in Figure 3 is realized by a combination of the CPU 101 that executes the shooting control application and one or more of the functional blocks of the shooting control device 100 shown in Figure 2.

[0098] Figure 5 is a flowchart showing the operation of the CPU 101 as the role determination unit 120. Figures 6(a) to 6(d) are flowcharts relating to the operation of the shooting control device 100, the overhead camera 300, the main camera 500, and the sub-camera 400, respectively.

[0099] In the following description, it is assumed that the 3D coordinate values ​​of the viewpoint position and the shooting direction (optical axis direction) of the overhead camera 300 are known to the shooting control device 100. Furthermore, it is assumed that known positional information, such as the 3D coordinate values ​​of the viewpoint positions of the sub-camera 400 and the main camera 500, and the coordinate values ​​of markers placed within the shooting range, are pre-stored in the ROM 103 as default positional information REF_POSI. Note that the coordinate system of the position is predetermined according to the type of position.

[0100] (Operation of the role determination unit 120) First, the operation of the CPU 101 as the role determination unit 120 in Figure 3 will be explained with reference to the flowchart shown in Figure 5. The operation described below is achieved by the CPU 101 executing the shooting control application.

[0101] There are no particular restrictions on the timing of initiating the operations shown in the flowchart of Figure 5, but they should be performed at least before starting control of the sub-camera. They should also be performed when the role control device 600 receives notification via the network interface 105 that the role settings for cameras 400 and 500 have been changed.

[0102] In S101, the CPU 101, acting as the role determination unit 120, obtains the roles (role setting information) set for cameras 400 and 500 from the role control device 600. The CPU 101 can obtain the aforementioned role setting information from the role control device 600 by, for example, sending a role acquisition command to the role control device 600 via the network interface 105. The CPU 101 stores the acquired role setting information in the RAM 102. Here, it is assumed that camera 400 is set as a sub-camera and camera 500 is set as the main camera, and these will be referred to as sub-camera 400 and main camera 500 below.

[0103] In step S103, the CPU 101 acquires the camera (camera 400) set as a sub-camera and its corresponding control content based on the role setting information stored in RAM 102. Then, the CPU 101, acting as a role determination unit 120, transmits the acquired control content (CAMERA_ROLE) for sub-camera 400 to the tracking subject determination unit 123. In practice, the CPU 101 stores the control content in a specific area of ​​RAM 102 and refers to it when functioning as the tracking subject determination unit 123.

[0104] In S104, the CPU 101, acting as the role determination unit 120, transmits the acquired control information (CAMERA_ROLE) to the first zoom value calculation unit 125. In practice, the CPU 101 stores the control information in a specific area of ​​the RAM 102 and refers to it when functioning as the first zoom value calculation unit 125.

[0105] (Operation of the shooting control device 100) Next, the operation of the shooting control device 100 in controlling shooting by the sub-camera 400 will be explained with reference to Figures 3 and 6(a). The operation described below corresponds to the operation of the CPU 101 as the recognition unit 121, the focus subject determination unit 122, the tracking subject determination unit 123, the first pan / tilt value calculation unit 124, and the first zoom value calculation unit 125 in Figure 3. Note that the operation described below is realized by the CPU 101 executing the shooting control application.

[0106] In S201, the CPU 101 sends a shooting command to the overhead camera 300 via the network I / F 105 using a predetermined protocol. In response to this command, the overhead camera 300 begins supplying a video signal (video data) IMG to the video input unit 107. After the CPU 101 begins storing the video signal received by the video input unit 107 into the RAM 102, it executes S202.

[0107] In S202, the CPU 101 acquires information ANGLE indicating the shooting direction from the main camera 500. Specifically, the CPU 101 sends a shooting direction acquisition command to the main camera 500 via the network I / F 105 using a predetermined protocol. In response to the shooting direction acquisition command, the CPU 501 of the main camera 500 sends information ANGLE indicating the current shooting direction of the main camera 500 to the shooting control device 100. Information ANGLE may be, for example, the pan and tilt angles of the drive unit 509. The CPU 101 stores the acquired information ANGLE in the RAM 102.

[0108] In S203, the recognition unit 121 performs the following process. (1) Apply subject area detection processing to the input frame image and store the detection result. (2) For each detected subject area, the position information (image coordinates) is transformed. (3) Apply identification processing to each detected subject area to identify identification information (add information for identification processing in the case of a new subject). (4) Store the identification information ID[n] and location information POSITION[n] associated with each detected subject area.

[0109] The recognition unit 121 is mainly implemented by the CPU 101 and the inference unit 104. The CPU 101 reads one frame of video received from the overhead camera 300 from the RAM 102 and inputs it to the inference unit 104.

[0110] The operation of the recognition unit 121 will be explained step by step below. (1) First, the inference unit 104 inputs the frame image to the machine learning model and detects the subject area. The inference unit 104 stores the position and size of each detected subject area, and the detection confidence level, which are output by the machine learning model as detection results, in the RAM 102. The position and size of the subject area may be any information that can identify the position and size of the rectangular area inscribed in the subject area. Here, the coordinates of the center of the bottom edge of the rectangular area, as well as its width and height, are used as the position and size of the subject area.

[0111] Furthermore, the inference unit 104 stores the detection result for the first frame image in RAM 102 in association with the subject identification information ID[n]. Here, n is the subject number and is an integer that takes values ​​from 1 to the total number of detected subject regions. In addition, the inference unit 104 stores the subject regions detected from the first frame image in RAM 102 in association with the subject identification information ID[n] as a template for identifying individual subjects. If template matching is not used for subject identification, it is not necessary to store the template.

[0112] Figure 8(a) shows an example of the results of subject detection processing by the inference unit 104 on the image from the overhead camera 300 shown in Figure 7(a). Here, the areas of human subjects A to C that are within the shooting range 20 are detected, and the coordinates of the center of the bottom edge of the rectangular area inscribed with the subject area (foot coordinates) are output as the position.

[0113] Furthermore, for coordinate transformations described later, if markers are placed at known positions within the shooting range 20, as shown in Figure 7(b), the CPU 101 detects the marker images included in the frame image (Figure 7(a)) and stores their positions in the RAM 102. The detection of marker images may also be configured to be performed by the inference unit 104. Marker image detection can be performed by any known method, such as pattern matching using a marker template. Marker images may also be detected using a pre-stored machine learning model for marker detection.

[0114] (2) Next, the coordinate transformation performed by the inference unit 104 will be explained. Figure 7(a) schematically shows the image from the overhead camera 300, and Figure 7(b) schematically shows the shooting range 20 as viewed from directly above its center. The inference unit 104 transforms the position of the subject area in the coordinate system of the overhead camera to the coordinate system (plane coordinate system) when the shooting range 20 is viewed from directly above its center.

[0115] The reason for transforming the coordinates to a planar coordinate system here is that it is convenient for calculating the pan angle (angle of movement in the horizontal plane) required to photograph a specific subject with the sub-camera 400. It is assumed here that the sub-camera 400 is positioned so that the drive unit 409 pans within a horizontal plane parallel to the floor of the shooting range 20.

[0116] Coordinate transformation can be performed in various ways, but here, markers are placed at multiple known positions on the floor within the shooting range 20, and the coordinates are transformed from the overhead camera coordinate system to the planar coordinate system based on the marker positions in the image obtained from the overhead camera 300. Alternatively, the coordinate transformation may be performed without using markers, such as by using the viewpoint position and shooting direction of the overhead camera 300.

[0117] The coordinate transformation can be performed using the homography transformation matrix H, according to Equation 1 below.

number

[0118] The homography transformation matrix can be calculated by substituting the coordinates of the four markers detected from the video and the coordinates (known) of the four markers placed in the shooting range 20 into Equation 1 and solving the system of equations. If the positional relationship between the shooting range 20 and the overhead camera 300 is fixed, the homography transformation matrix H can be calculated in advance during test shooting and saved, for example, in ROM 103.

[0119] The CPU 101 sequentially reads the position of the subject area from the RAM 102 and transforms the coordinates to values ​​in a planar coordinate system. Figure 8(b) schematically shows the state after the foot coordinates (x, y) of each subject area detected in the overhead camera 300 video shown in Figure 8(a) have been transformed to coordinate values ​​(X, Y) in a planar coordinate system using Equation 1 and the homography transformation matrix H stored in ROM 103. The CPU 101 stores the transformed foot coordinates as POSITION[n] in the RAM 102.

[0120] (3) Next, the operation of the inference unit 104 in identifying the subject identification information ID[n] will be described. Here, the subject will be identified using template matching. Subject identification will be performed on the processing results of the second and subsequent subject detections. For the first processing result, a new identification information ID[n] will be assigned to the subject region.

[0121] The inference unit 104 identifies the identification information ID[n] of the detected subject area by template matching using templates stored in RAM 102. This identifies the subject within the shooting range. For example, the inference unit 104 calculates an evaluation value representing the correlation of individual templates for each detected subject area. Then, the inference unit 104 identifies the identification information ID[n] corresponding to the template with the highest correlation that has a certain level of correlation as the identification information ID[n] of the subject area. The evaluation value can be a known value, such as the sum of the absolute differences of pixel values.

[0122] Furthermore, the inference unit 104 assigns a new identification information ID[n] to any subject region that does not have a certain level of correlation with all templates, and adds the image of the subject region to the template.

[0123] Furthermore, the inference unit 104 may update existing templates using subject regions detected in the most recent frame image, or delete templates for which no subject regions with a certain level of correlation have been found for a certain period of time. In addition, the inference unit 104 may store templates corresponding to frequently appearing identification information ID[n] in the ROM 103.

[0124] Furthermore, subjects may be identified by methods other than template matching. For example, at least one of the previously detected position and size may be identified as having the same identification information ID [n] as the nearest subject region. Alternatively, the position in the current frame image may be predicted using a Kalman filter or the like based on the position changes in multiple past detection results associated with the same identification information, and the same identification information ID may be identified as the subject region closest to the predicted position. These methods may also be combined. By not using template matching, the accuracy of identifying different subjects that look similar can be improved.

[0125] (4) The inference unit 104 associates the identified identification information ID[n] with the corresponding position (plane coordinate system) POSITION[n] of the subject area and stores it in the RAM 102.

[0126] Note that, of the processes (1) to (4), the CPU 101 may execute the processes other than subject detection instead of the inference unit 104.

[0127] Here, the image from the overhead camera 300 was used to determine the identification information ID[n] and position[n] of the subject within the shooting range 20. However, the image from the sub-camera 400 may also be used. If there are multiple sub-cameras 400, the CPU 101 performs the operations shown in the flowchart in Figure 6(a) for each sub-camera 400. The position of the subject area is output as a value in the coordinate system of each sub-camera 400. Thus, although the overhead camera 300 is not essential, it is considered that using the overhead camera 300 improves the accuracy of subject detection.

[0128] Returning to the explanation of Figure 6(a), in S204, the CPU 101, which acts as the subject of interest determination unit 122 in Figure 3, determines the subject of interest to be tracked by the main camera 500. The CPU 101 can determine the subject of interest of the main camera 500 from among the subjects detected in S203, based on the shooting direction of the main camera 500 acquired in S202. The CPU 101 stores the identification information ID[n] corresponding to the subject area determined to be the subject of interest of the main camera 500 in the RAM 102 as the identification information of the subject of interest MAIN_SUBJECT.

[0129] For example, the CPU 101 can determine the subject closest to the shooting direction of the main camera 500 in a planar coordinate system as the subject of interest for the main camera 500. If there are multiple subjects whose distance from the shooting direction of the main camera 500 is below a threshold, the user may be allowed to select the subject of interest from among them.

[0130] When prompting the user to select a subject of interest, the CPU 101 displays the frame image to which subject detection processing has been applied in S202 on the display unit 108 or an external display device, along with an indicator showing the shooting direction and an indicator showing the subject area that is a candidate for the subject of interest. The subject area indicator may be a rectangular frame indicating the outer edge of the subject area, as shown in Figure 8(a), but other indicators may also be used. The CPU 101 may also display messages on the display unit 108 prompting the user to select a subject of interest in the image.

[0131] The user can select a subject area corresponding to a desired subject of interest by operating the user input unit 106 (input device). There are no particular restrictions on the selection method, but it may be an operation to specify the desired subject area by using a mouse or keyboard.

[0132] When the CPU 101 detects a user operation specifying a subject area, it stores the identification information ID[n] corresponding to the specified subject area in the RAM 102 as the identification information MAIN_SUBJECT of the subject of interest.

[0133] Next, in S205, the CPU 101, acting as the tracking subject determination unit 123 in Figure 3, obtains the control content CAMERA_ROLE corresponding to the role set for the sub-camera 400. Specifically, the CPU 101 reads the control content CAMERA_ROLE obtained in the role determination process explained using Figure 5 and stored in the RAM 102. If there are multiple sub-cameras 400, the CPU 101 executes the processes in S205 to S207 for each sub-camera.

[0134] In S206, the CPU 101, acting as the tracking subject determination unit 123, determines the subject to be tracked and photographed by the sub-camera 400 according to the control content CAMERA_ROLE. The CPU 101 determines the subject to be tracked by the sub-camera 400 according to the definition of the tracking subject included in the control content CAMERA_ROLE (Figure 4).

[0135] If the subject being tracked by sub-camera 400 is to be the same as the subject of interest of main camera 500, CPU 101 sets the identification information of the subject of interest determined in S203, MAIN_SUBJECT, as the identification information of the subject being tracked by sub-camera 400, SUBJECT_ID.

[0136] If the subject to be tracked by the sub-camera 400 is to be a subject located to the left of the subject of interest of the main camera 500, the CPU 101 detects the subject region located at the leftmost edge of the subject region other than the subject of interest among the subject regions detected in S203. Then, the CPU 101 sets the identification information ID[n] corresponding to the detected subject region as the identification information SUBJECT_ID of the subject to be tracked by the sub-camera 400.

[0137] The CPU 101 writes the identification information SUBJECT_ID of the determined tracked subject to RAM 102. If the tracked subject may differ depending on the sub-camera, the CPU 101 stores the identification information SUBJECT_ID of the tracked subject in association with the identification information of the sub-camera. If the tracked subject changes, the CPU 101 retains the information of the previous tracked subject in RAM 102 without erasing it.

[0138] Here, we will explain the operation when the control setting (CAMERA_ROLE) for the sub-camera 400 is set to "main follow," using Figure 9. When the control setting "main follow" is set for the sub-camera 400, the shooting control device 100 controls it to track the subject of interest of the main camera 500.

[0139] Therefore, as shown in Figure 9(a), if the main camera 500 determines that the subject of interest is subject B, the CPU 101 decides that subject B will be the subject to be tracked by the sub-camera 400. Subsequently, as shown in Figure 9(b), if the main camera 500 determines that the subject of interest has changed to subject A, the CPU 101 changes the subject to be tracked by the sub-camera 400 to subject A. Similarly, as shown in Figure 9(c), if the main camera 500 determines that the subject of interest has changed to subject C, the CPU 101 changes the subject to be tracked by the sub-camera 400 to subject C.

[0140] The operation of the sub-camera 400 when the control setting is "Assist Follow" will be explained using Figure 10. When the sub-camera 400 is set to the role "Assist Follow", the shooting control device 100 controls it to track a subject located to the left of the main camera 500, which is a subject other than the subject of interest of the main camera 500.

[0141] Therefore, as shown in Figure 10(a), if the main camera 500 determines that the subject of interest is subject B, the CPU 101 decides that subject A, the leftmost of subjects A and C, will be the subject tracked by the sub-camera 400. Subsequently, as shown in Figure 10(b), if the main camera 500 determines that the subject of interest has changed to subject A, the CPU 101 changes the subject tracked by the sub-camera 400 to subject B, the leftmost of subjects B and C. Also, as shown in Figure 10(c), if the main camera 500 determines that the subject of interest has changed to subject C, the CPU 101 changes the subject tracked by the sub-camera 400 to subject A, the leftmost of subjects A and B.

[0142] The role control device 600 dynamically changes the role assigned to the sub-camera 400, thereby changing the subject tracked by the sub-camera 400 and enabling flexible automatic shooting.

[0143] Returning to Figure 6(a), in S207, the CPU 101, acting as the first pan / tilt value calculation unit 124, calculates the amount of change in pan and tilt angles necessary for the sub-camera 400 to track and photograph the subject determined in S206. The CPU 101, acting as the first zoom value calculation unit 125, also calculates the zoom value of the sub-camera 400 in accordance with the change in the field of view of the main camera 500. The following describes the case where there is one sub-camera 400, but if there are multiple sub-cameras 400, the calculation of the pan and tilt angle changes and the zoom value are performed for each sub-camera.

[0144] First, the operation of the CPU 101 as the first pan-tilt value calculation unit 124 will be explained. Here, it is assumed that the following information is pre-stored in the ROM 103 as default position information REF_POSI for each sub-camera 400. • 3D coordinates of the installation location (values ​​in a planar coordinate system) • Shooting direction corresponding to the initial values ​​of the pan and tilt angles of the drive unit • Controllable range of pan and tilt angles

[0145] The CPU 101 reads the position information POSITION_OH, which corresponds to the identification information SUBJECT_ID of the subject being tracked by the sub-camera 400, from the RAM 102. Then, the CPU 101 first determines the pan angle based on the position information POSITION_OH and the installation position of the sub-camera 400.

[0146] Figure 11 shows an example of the positional relationship between the sub-camera 400 and the tracked subject in a planar coordinate system. Here, we will determine the pan angle θ so that the optical axis of the sub-camera 400 is directed towards the subject position. The CPU 101 calculates the pan angle θ using the following equation 2.

number

[0147] In Equation 2, px and py are the horizontal and vertical coordinates of the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracked subject. Also, subx and suby are the horizontal and vertical coordinates of the installation position of the sub-camera 400. Here, it is assumed that the current pan angle is the initial value of 0° and the optical axis direction is the vertical direction (Y-axis direction). If the current optical axis direction is not the vertical direction, the angle difference between the current optical axis direction and the vertical direction may be reflected in the angle obtained by Equation 2. Also, the direction of pan is counterclockwise if subx > px, and clockwise if subx < px.

[0148] Next, a method for determining the tilt angle will be described using FIG. 12. FIG. 12 shows a state of the sub-camera and the tracked subject as viewed from the side. It is assumed that the current optical axis of the sub-camera 400 is in the horizontal direction and the height is h1, and the height of the face of the tracked subject facing the optical axis is h2. Let ρ be the angle difference in the height direction (tilt angle) between the current optical axis direction and the target optical axis direction. The CPU 101 calculates the tilt angle ρ using the following Equations 3 and 4.

Equation

[0149] The coordinate values used in Equation 4 are the same as those used in Equation 2. It is assumed that h1 and h2 are input in advance to the shooting control application and stored in the RAM 102. In this case, the identification number associated with h2 for each subject is made equal to the identification number assigned in the subject detection process. Alternatively, h2 may be a value measured in real time using a sensor not shown in the figure.

[0150] Here, it is assumed that the current tilt angle is the initial value of 0° and the optical axis direction is the horizontal direction (constant height). If the current optical axis direction is not the horizontal direction, the angle difference between the current optical axis direction and the horizontal direction may be reflected in the angle obtained by Equation 4. Also, the direction of tilt is downward if h1 > h2, and upward if h1 < h2.

[0151] The CPU 101 periodically communicates with the sub-camera 400 via the communication network 700 to obtain the current optical axis direction (pan angle and tilt angle of the drive unit) and stores it in the RAM 102. The communication period can be, for example, less than or equal to the reciprocal of the frame rate. Alternatively, the CPU 101 may store the sum of the pan angle and tilt angle controlled for the sub-camera 400 from its initial state in the RAM 102 and use this as the current optical axis direction.

[0152] The CPU 101 calculates the amount of change in the pan angle and tilt angle of the sub-camera 400 in this manner and stores it in the RAM 102. If there are multiple sub-cameras 400, the CPU 101 calculates the amount of change in the pan angle and tilt angle for each sub-camera.

[0153] The changes in the pan and tilt angles may be the angular velocity required for the sub-camera 400 to rotate towards the tracked subject. For example, the CPU 101 obtains the current pan and tilt angles from the sub-camera 400 via the communication network 700. The CPU 101 then calculates the angular velocity of the pan, which is proportional to the difference between the pan angle θ read from the RAM 102 and the current pan angle. The CPU 101 also calculates the angular velocity of the tilt, which is proportional to the difference between the tilt angle ρ read from the RAM 102 and the current tilt angle. The CPU 101 stores these calculated angular velocities in the RAM 102.

[0154] Next, the operation of the CPU 101 as the first zoom value calculation unit 125 will be explained. The CPU 101 as the first zoom value calculation unit 125 periodically acquires information MAIN_ZOOM indicating the field of view of the main camera 500 and stores it in RAM 102. When the information MAIN_ZOOM changes, the CPU 101 calculates the zoom value Z_VALUE for the sub-camera 400 according to the control content CAMERA_ROLE corresponding to the role set for the sub-camera 400.

[0155] The CPU 101 can determine the zoom operation of the main camera 500 and its phase, for example, by detecting changes in the field of view of the image from the main camera 500. For example, changes in the field of view may be detected from changes in the size and spacing of the subject area over time.

[0156] Figure 13 shows an example of mapping the zoom values ​​of the main camera and the sub-camera. Here, the main camera 500 and the sub-camera 400 are assumed to optically change their field of view (the imaging optical system has a zoom function). However, a similar function may be achieved with digital zoom using the image processing units 406 and 506.

[0157] The zoom value is a parameter whose value corresponds to the field of view. In this embodiment, the smaller (narrower) the field of view, the smaller the zoom value, and the zoom value on the telephoto side is smaller than the zoom value on the wide-angle side. The sub-camera 400 and the main camera 500 can control the imaging optical system to the field of view corresponding to the zoom value by sending a command that specifies the zoom value. In other words, the zoom value is information about the field of view and represents the zoom state. The zoom value may also be, for example, the focal length (mm) of the imaging optical system corresponding to a 35mm full-frame image sensor, in which case the zoom value on the telephoto side will be larger than the zoom value on the wide-angle side.

[0158] In Figure 13, the zoom range of the main camera 500, MAIN_ZOOM, is main_min to main_max. The zoom range of the sub-camera 400 is sub_min to sub_max. main_min and sub_min correspond to the telephoto ends of the main camera 500 and sub-camera 400, respectively, while main_max and sub_max correspond to the wide-angle ends of the main camera 500 and sub-camera 400, respectively. Figure 13 shows an example where the zoom range of the main camera 500 is wider than the zoom range of the sub-camera 400 at both the telephoto and wide-angle ends.

[0159] When controlling the zoom value SUB_ZOOM of sub-camera 400 to be in phase with the zoom value MAIN_ZOOM of main camera 500, CPU 101 calculates the SUB_ZOOM corresponding to the current MAIN_ZOOM using the following equation 5.

number

[0160] On the other hand, when controlling the zoom value SUB_ZOOM of sub-camera 400 in the opposite phase to the zoom value MAIN_ZOOM of main camera 500, the SUB_ZOOM corresponding to the current MAIN_ZOOM is calculated using the following equation 6. Specifically, the CPU 101 calculates the SUB_ZOOM corresponding to the current MAIN_ZOOM by substituting the SUB_ZOOM calculated in equation 5 into the right-hand side of the following equation 6. SUB_ZOOM=sub_max-(SUB_ZOOM-sub_min) (Formula 6)

[0161] When the main camera 500 performs digital zoom and controls the field of view by cropping, the CPU 101 can determine the zoom value SUB_ZOOM of the sub-camera 400 according to the size of the area to be cropped by the main camera 500. Specifically, the CPU 101 sets the zoom value SUB_ZOOM to a smaller size (higher magnification) when the area to be cropped by the main camera 500 is small, and to a larger size (lower magnification) when the area to be cropped is large.

[0162] Returning to Figure 6(a), in S207, the CPU 101 reads the pan and tilt angle changes and the zoom value calculated in S206 from the RAM 102. The CPU 101 then generates a control command PT_VALUE that instructs the sub-camera 400 to change the pan and tilt angles by the amounts of these changes. The CPU 101 also generates a control command Z_VALUE that instructs the sub-camera 400 to change the field of view by the zoom value. The format of the control commands is predetermined. The CPU 101 stores the generated control commands PT_VALUE and Z_VALUE in the RAM 102. Note that if the tracked subject is stationary or the field of view of the main camera 500 does not change, and there is no need to generate control commands, S207 may be skipped.

[0163] The CPU 101 then reads the control commands PT_VALUE and Z_VALUE from RAM 102 and sends them to the communication network 700 via the network interface 105. The sub-camera 400 receives the control commands PT_VALUE and Z_VALUE via the network interface 405.

[0164] In S208, the CPU 101 determines the PTZ value of the main camera 500 based on the orientation information and zoom value received from the remote control device 800. The CPU 101 then sends a control command corresponding to the determined PTZ value to the main camera 500.

[0165] Specifically, the CPU 101, acting as the second pan-tilt value calculation unit 126, receives attitude information (ORIENTATION) from the remote control device 800 and calculates the amount of change in pan angle and tilt angle based on the difference with the previously received attitude information. The second pan-tilt value calculation unit 126 then generates a control command PT_VALUE that instructs the change in pan angle and tilt angle corresponding to the amount of change. The second pan-tilt value calculation unit 126 sets the destination of the generated control command PT_VALUE to the main camera (camera 500 in this case) and transmits it to the communication network 700 via the network I / F 805.

[0166] Furthermore, when the CPU 101, acting as the second zoom value calculation unit 127, receives a zoom value (ZOOM) from the remote control device 800, it calculates the amount of change in the zoom value based on the difference with the previously received zoom value. The second zoom value calculation unit 127 then generates a control command Z_VALUE that instructs the zoom value to be changed based on the amount of change. The second zoom value calculation unit 127 then sets the destination of the generated control command Z_VALUE to the main camera (camera 500 in this case) and transmits it to the communication network 700 via the network I / F 805.

[0167] Camera 500 (CPU 501) receives the control command PT_VALUE via the network interface 505. Based on PT_VALUE, CPU 501 drives the drive unit 509 via the drive interface 508 to change the orientation (shooting direction) of camera 500. CPU 501 may also notify the shooting control device 100 of the changed pan angle and tilt angle.

[0168] Furthermore, the camera 500 (CPU 501) receives the control command Z_VALUE via the network I / F 505. Based on Z_VALUE, the CPU 501 drives the drive unit 509 via the drive I / F 508 to change the camera 500's field of view. The CPU 501 may also notify the shooting control device 100 of the zoom value corresponding to the changed field of view.

[0169] The CPU 101 executes the processing from S201 on the next frame image of the video from the overhead camera 300. Note that the processing shown in Figure 6(a) does not necessarily have to be executed every frame.

[0170] (Operation of overhead camera 300) Next, the operation of the overhead camera 300 will be explained with reference to Figure 6(b). The operation described below is achieved by the CPU 301 executing a program.

[0171] When the overhead camera 300 is powered on, the CPU 301 initializes each functional block, and then the camera enters a shooting standby state. In the shooting standby state, the CPU 301 may start video recording processing for live view display and output the display image data generated by the image processing unit 306 to the shooting control device 100 via the network I / F 305.

[0172] In the shooting standby state, the CPU 301 waits for the reception of control commands via the network I / F 305. When the CPU 301 receives a control command, it executes an action corresponding to the control command. This section describes the operation when a shooting command is received as a control command from the shooting control device 100.

[0173] In S301, CPU301 receives shooting commands from the shooting control device 100 via the network I / F305.

[0174] The shooting command may also specify shooting parameters such as frame rate and resolution. Furthermore, it may include settings related to the processing to be applied by the image processing unit 306.

[0175] In S302, the CPU 301 responds to the reception of a shooting command and starts video recording processing to be supplied to the shooting control device 100. This video recording processing captures video with higher image quality than the video recording processing for live view display. For example, at least one of the video resolution and the shooting frame rate is higher than that of the video for live view display. The image processing unit 306 applies processing to the image based on the settings for the video to be supplied to the shooting control device 100. The image processing unit 306 sequentially stores the generated video data in the RAM 302.

[0176] In step S303, the CPU 101 reads the video data from the RAM 302 and transmits it to the shooting control device 100 via the network interface 305. From this point onward, the processing from shooting to the supply of video data continues until a control command to stop shooting is received.

[0177] (Operation of Main Camera 500) Next, the operation of the main camera 500 will be explained with reference to Figure 6(c). The operation described below is achieved by the CPU 501 executing a program.

[0178] When the main camera 500 is powered on, the CPU 501 initializes each functional block, and then starts video recording processing to supply to the shooting control device 100. The image processing unit 506 applies processing to the analog image signal obtained from the image sensor 507 based on the settings for video to be supplied to the shooting control device 100. The image processing unit 506 sequentially stores the generated video data in the RAM 502. The CPU 501 reads the video data from the RAM 502 and outputs it to an external device such as a switcher via the network I / F 505. The CPU 501 also supplies video for live view display based on the video data to the shooting control device 100 and the remote control device 800. The video for live view display may be the same as the video data, but may have its resolution reduced by the image processing unit 506.

[0179] The CPU 501 supplies video data to the shooting control device 100 while waiting for control commands to be received via the network interface 305. When the CPU 501 receives a control command, it executes an action corresponding to the control command. This section describes the operation when a shooting direction acquisition command is received. Note that when the pan / tilt control command PT_VALUE or the zoom control command Z_VALUE is received, the CPU 501 drives the drive unit 509 according to the command.

[0180] In S501, CPU501 receives a shooting direction acquisition command via network I / F505. CPU501 stores the received shooting direction acquisition command in RAM502.

[0181] In S502, the CPU 501, in response to receiving a command to acquire the shooting direction, acquires the current pan angle and tilt angle from the drive unit 509 via the drive I / F 508 and stores them in the RAM 502.

[0182] In S503, the CPU 501 reads the current pan angle and tilt angle from RAM 502 and transmits the shooting direction information ANGLE to the shooting control device 100 via the network I / F 305.

[0183] (Operation of sub-camera 400) Next, the operation of the sub-camera 400 will be explained with reference to Figure 6(d). The operation described below is achieved by the CPU 401 executing a program.

[0184] When the sub-camera 400 is powered on, the CPU 401 initializes each functional block, and then starts video recording processing to supply to the shooting control device 100. The image processing unit 406 applies processing to the analog image signal obtained from the image sensor 407 based on the settings for video to be supplied to the shooting control device 100. The image processing unit 406 sequentially stores the generated video data in the RAM 402. The CPU 401 reads the video data from the RAM 402 and supplies it to the shooting control device 100 via the network I / F 405.

[0185] The CPU 401 supplies video data to the shooting control device 100 while waiting for control commands to be received via the network interface 305. When the CPU 401 receives a control command, it executes an action corresponding to the control command. This section describes the operation when the CPU 401 receives the pan / tilt control command PT_VALUE and the zoom control command Z_VALUE from the shooting control device 100.

[0186] In S401, CPU401 receives at least one of the pan / tilt control command PT_VALUE and the zoom control command Z_VALUE from the imaging control device 100 via the network I / F405. CPU401 stores the received control commands in RAM402.

[0187] In S402, CPU401 reads the operating direction and corresponding manipulated amount from the control command stored in RAM402 and stores them in RAM402. Here, in the case of the pan / tilt control command PT_VALUE, the operating direction is the direction of pan and / or tilt, and the manipulated amount is the target angle. Also, in the case of the zoom control command Z_VALUE, the manipulated amount is the zoom value, and since the operating direction can be determined from the zoom value, reading and storing the operating direction is unnecessary.

[0188] In S403, the CPU 401 generates drive parameters for the drive unit 409 based on the operating direction and operating amount read in S403. The CPU 401 may, for example, obtain drive parameters corresponding to a combination of operating direction and operating amount using a table previously stored in ROM 403. If the operating amount is given as a target value (target angle or zoom value), the CPU 410 obtains the drive parameters from the difference between that value and the current value.

[0189] In S404, CPU401 controls drive unit 409 via drive I / F408 based on the drive parameters acquired in S404. This causes drive unit 409 to change the shooting direction of sub-camera 400 to the operating direction and angle specified by the pan / tilt control command PT_VALUE. Additionally, drive unit 409 changes the field of view of the shooting optical system to the zoom value specified by the zoom control command Z_VALUE.

[0190] Next, using the flowchart shown in Figure 14, we will explain in more detail how the shooting control device 100 controls the shooting direction (pan and tilt) and field of view (zoom value) of the sub-camera 400 according to the role set for the sub-camera. The operations shown in the flowchart of Figure 14 are performed as part of the operations S205 to S207 in Figure 6(a).

[0191] S601 corresponds to S205, and CPU101 reads the control content CAMERA_ROLE stored in RAM102 in S103 of Figure 5.

[0192] S602-S607 are implemented, for example, in S206. In S602, the CPU 101 determines whether the tracking subject specification for sub-camera 400, included in the control content CAMERA_ROLE, indicates the tracking subject (subject of interest) of main camera 500. For example, if the tracking subject specification for sub-camera 400 has a value indicating "same as main", the CPU 101 determines that the tracking subject specification for sub-camera 400 indicates the tracking subject of main camera 500 and executes S603. On the other hand, if the tracking subject specification for sub-camera 400 has a value indicating "different from main (left side)", the CPU 101 determines that the tracking subject specification for sub-camera 400 does not indicate the tracking subject of main camera 500 and executes S604.

[0193] In S603, CPU101 decides to control the shooting direction of sub-camera 400 so that it tracks the subject being tracked (the subject of interest) by main camera 500. In S604, CPU101 decides to control the shooting direction of sub-camera 400 to track a subject located to the left of the main camera 500's subject of interest.

[0194] In S605, the CPU 101 determines whether the zoom control specification for the sub-camera 400, included in the control content CAMERA_ROLE, indicates control in the same phase as the main camera 500. For example, if the zoom control specification for the sub-camera 400 has a value indicating "same phase as the main," the CPU 101 determines that the zoom control specification for the sub-camera 400 indicates control in the same phase as the main camera 500 and executes S606. On the other hand, if the zoom control specification for the sub-camera 400 has a value indicating "opposite phase to the main," the CPU 101 determines that the zoom control specification for the sub-camera 400 does not indicate control in the same phase as the main camera 500 and executes S607.

[0195] In S606, CPU101 decides to control the zoom value (angle of view) of sub-camera 400 in phase with the change in the zoom value of main camera 500. In S607, CPU101 decides to control the zoom value (angle of view) of sub-camera 400 in the opposite phase to the change in the zoom value of main camera 500.

[0196] Figure 15 illustrates an example of sub-camera control when the control setting for sub-camera 400 is "main follow". Figure 15 schematically shows how the shooting control device 100 controls the shooting direction and field of view of sub-camera 400 when the subject of focus and field of view of the main camera 500 change over time during shooting. In the figure, time progresses from left to right. Note that in Figure 15, the zoom state is shown in three stages: "telephoto end", "intermediate", and "wide-angle end". This is because, as shown in Figure 13, the zoom range of the sub-camera and the main camera may differ. "Telephoto end" corresponds to the state when zoomed up to the telephoto end, "wide-angle end" corresponds to the state when zoomed down to the wide-angle end, and "intermediate" corresponds to a zoom state between the telephoto end and the wide-angle end, but the actual zoom values ​​may differ between the sub-camera and the main camera. For example, when the zoom state is "telephoto end", the zoom value of the main camera is main_min, and the zoom value of the sub-camera is sub_min.

[0197] Initially, the main camera 500's focus subject (tracking subject) is subject B, and the zoom level is "intermediate." Therefore, the CPU 101 determines that the sub-camera 400's tracking subject is subject B and controls the shooting direction so that the sub-camera 400 tracks subject B. The CPU 101 also controls the zoom level of the sub-camera 400 to "intermediate."

[0198] Subsequently, the main camera 500's focus subject changes from subject B to subject A, and its zoom state is changed to "telephoto end." In response, the CPU 101 changes the subject being tracked by the sub-camera 400 from subject B to subject A, and controls the shooting direction so that the sub-camera 400 tracks subject A. The CPU 101 also controls the zoom state of the sub-camera 400 to "telephoto end."

[0199] Subsequently, the main camera 500's focus subject changes from subject A to subject C, and its zoom state changes to "wide-angle end." In response, the CPU 101 changes the subject being tracked by the sub-camera 400 from subject A to subject C and controls the shooting direction so that the sub-camera 400 tracks subject C. The CPU 101 also controls the zoom state of the sub-camera 400 to "wide-angle end."

[0200] Thus, when the control setting for the sub-camera 400 is "main follow," the CPU 101 automatically changes the tracked subject and zoom value of the sub-camera 400 to follow the changes in the subject of interest and field of view (zoom value) of the main camera 500. In the example shown in Figure 15, the sub-camera 400 is controlled so that the degree of change in its zoom state is equivalent to that of the main camera 500, but the actual zoom values ​​may differ as long as the direction of change in the zoom values ​​is in phase. For example, the zoom state of the sub-camera 400 does not have to be at the telephoto end when the zoom state of the main camera 500 is at the telephoto end. Whether or not to match the zoom value of the sub-camera 400 to the zoom value of the main camera 500 may be configurable using role setting information.

[0201] Next, an example of sub-camera control when the control setting for sub-camera 400 is "assist counter" will be explained using Figure 16, which is similar to Figure 15.

[0202] Initially, the main camera 500's focus subject (tracking subject) is subject B, and the zoom state is at the "telephoto end". Therefore, the CPU 101 determines that the sub-camera 400's tracking subject is subject A, the leftmost of subjects A and C other than subject B, and controls the shooting direction so that the sub-camera 400 tracks subject A. The CPU 101 also controls the zoom state of the sub-camera 400 to the "wide-angle end", which is in the opposite phase to that of the main camera 500.

[0203] Subsequently, the main camera 500's focus subject changes from subject B to subject A, and its zoom level changes to "intermediate." In response, the CPU 101 changes the subject being tracked by the sub-camera 400 to subject B, which is on the left of subjects B and C other than subject A, and controls the shooting direction so that the sub-camera 400 tracks subject B. Also, because the main camera 500's zoom level has changed from "telephoto end" to "intermediate," the CPU 101 controls the sub-camera 400's zoom level from "wide-angle end" to "intermediate" (in opposite phase).

[0204] Subsequently, the main camera 500's focus subject changes from subject A to subject C, and its zoom state changes to "wide-angle end." In response, the CPU 101 changes the subject being tracked by the sub-camera 400 to subject A, which is the leftmost of subjects A and B (excluding subject C), and controls the shooting direction so that the sub-camera 400 tracks subject A. Also, because the main camera 500's zoom state has changed from "intermediate" to "wide-angle end," the CPU 101 controls the sub-camera 400's zoom state from "intermediate" to "telephoto end" (in opposite phase).

[0205] Thus, when the control setting for the sub-camera 400 is "assist counter," the CPU 101 automatically changes the subject being tracked by the sub-camera 400 to a different subject from the main camera 500's focus subject, in response to changes in the main camera 500's focus subject. In addition, the CPU 101 automatically changes the zoom value of the sub-camera 400 in the opposite direction to the change in the main camera 500's field of view (zoom value).

[0206] (Main camera change) Next, we will explain the behavior when the main camera settings are changed. Here, we will explain assuming that the main camera has been changed from camera 500 to camera 400. As mentioned above, the main camera settings are performed by the role control unit 600.

[0207] Here, as an example, we assume that the main camera has been changed from camera 500 to camera 400 in the state shown in Figure 18(a). Furthermore, we assume that the control settings of camera 400 before the change will be inherited as the control settings of camera 500, which will become the sub-camera after the change.

[0208] In the state shown in Figure 18(a), the main camera 500 is under the control of the remote control device 800 and is capturing images with subject B as the primary subject. The sub-camera 400 is set to the control content "Assist Follow," and the shooting control device 100 is controlling the shooting direction and field of view so that it captures images with subject A as the primary subject.

[0209] In this state, the main camera is changed to camera 400, and at the same time, camera 500 switches to the sub-camera. Furthermore, the control setting for camera 500 becomes "assist follow". Figure 18(b) shows the state immediately after the main camera has switched.

[0210] As described above, when the role setting for a camera is changed, the role control device 600 notifies the shooting control device 100 and the remote control device 800 of the changed setting. As a result, the shooting control device 100 starts controlling the sub-camera's operation according to the changed setting. Also, if the main camera is changed, the shooting control device 100 continues to apply the operation control that was applied to the changed main camera when it was a sub-camera until the difference between the attitude of the changed main camera and the attitude of the operation unit 812 of the remote control device 800 falls below a threshold. While operation control is being continued, the shooting control device 100 disables remote control of the changed main camera by the remote control device 800.

[0211] This is because, at the time the main camera was changed, the orientation of the control unit 812 was the same as that of the main camera 500 before the change, as shown in Figure 18(a), and was significantly different from the orientation of the main camera 400 after the change, as shown in Figure 18(b).

[0212] Figure 19(a) shows the orientation of the main camera 500 and the control unit 812 in the state shown in Figure 18(a), and Figure 19(b) shows the orientation of the camera 400 and the control unit 812 when the main camera is changed to camera 400 in the state shown in Figure 18(a). As shown in Figure 19(b), when the main camera is changed to camera 400, the orientation of the control unit 812 and the orientation of camera 400 are significantly different. In this case, it is difficult to intuitively remotely control the shooting direction of camera 400 using the control unit 812.

[0213] The operation of the shooting control device 100 when the camera's role settings are changed will be further explained using the flowchart shown in Figure 20. The operation described here can be performed, for example, when the role control device 600 notifies the user of a setting change, but it may also be performed at other times.

[0214] In S701, the CPU 101 stores the role setting information notified by the role control device 600 in the RAM 102 and compares it with the role setting information before the notification. Then, the CPU 101 determines whether the main camera has been changed. If the CPU 101 determines that the main camera has been changed, it executes S702; otherwise, it executes S703.

[0215] In S702, the CPU 101 determines whether the camera to be controlled is the main camera. If the CPU 101 determines that the camera to be controlled is the main camera, it executes S704; otherwise, it executes S703. If the main camera is changed, the CPU 101 sequentially controls each camera other than the overhead camera 300 and performs the processing from S702 onwards.

[0216] In S703, the CPU 101 controls the operation of each sub-camera according to the set control settings, as explained using Figure 6(a). Thereafter, the CPU 101 continues to control the operation of the sub-cameras until it is notified of a change in settings from the role control device 600.

[0217] In S704, the CPU 101 acquires attitude information A of the operating unit 812 from the remote control device 800.

[0218] In S705, CPU101 acquires attitude information B from the modified main camera (camera 400 in Figure 18(b)).

[0219] In S706, CPU 101 determines whether the absolute value of the difference between attitude information A and attitude information B is less than a threshold. Since the attitudes of each camera and the attitude of the control unit 812 are calibrated to be equal during initial setup, attitude information A and attitude information B can be compared directly.

[0220] If the posture information is a combination of pan angle and tilt angle, the CPU 101 can determine whether both the difference in pan angle and the difference in tilt angle are below a threshold. A smaller threshold is preferable, but it does not have to be zero. The threshold may be common to both pan angle and tilt angle, or it may be different. If the thresholds are different, the threshold for pan angle can be made larger than the threshold for tilt angle. This is because, in general shooting, the tilt angle is rarely changed significantly, but the pan angle is not uncommonly changed significantly, so there is a high possibility of a large difference when changing the main camera. Therefore, by making the threshold for pan angle larger than the threshold for tilt angle, it is possible to specify a reduction in the time until the main camera operation by the control unit 812 becomes effective.

[0221] Furthermore, the threshold may be reduced as the field of view of the main camera decreases. This is because, even with the same angle difference, the smaller the field of view, the greater the shift in the shooting range. Additionally, the threshold may be increased as the movement of the subject of interest in the main camera increases. This is because, when the movement of the subject of interest is large, it is particularly desirable to shorten the time until the operation of the main camera using the control unit 812 becomes effective.

[0222] The CPU 101 executes S708 if it determines that the absolute value of the difference between attitude information A and attitude information B is less than a threshold, and executes S707 otherwise.

[0223] In S707, the CPU 101 continues the operation control that it had been performing on camera 400, which has now become the main camera, before it became the main camera. However, since it is the main camera, it does not perform field of view control and continues subject tracking operation. The CPU 101 also disables the remote control operation of the main camera by the remote control device 800. Specifically, the CPU 101 stops the operation of the second pan / tilt value calculation unit 126 and the second zoom value calculation unit 127. Alternatively, the CPU 101 prevents the control commands generated by the second pan / tilt value calculation unit 126 and the second zoom value calculation unit 127 from being sent to the main camera. Alternatively, the CPU 101 may instruct the remote control device 800 to stop sending attitude information and zoom values. Then, the CPU 101 executes S705.

[0224] For example, if the orientation of the main camera is controlled to synchronize with the orientation of the control unit 812, the shooting direction of the main camera may suddenly change when the main camera is switched. However, by disabling the control unit 812 when the difference between the orientation of the new main camera and the control unit 812 exceeds a threshold, it is possible to prevent the shooting direction of the new main camera from suddenly changing. Also, when controlling the orientation of the main camera according to the amount of operation of the control unit 812, it is possible to suppress the deterioration of operability caused by a large discrepancy between the orientation of the control unit 812 and the orientation of the main camera.

[0225] In addition, in S707, CPU101 may, instead of continuing the operation control it was performing before becoming the main camera, stop the operation control and maintain the shooting direction and field of view as they were immediately before becoming the main camera. Alternatively, CPU101 may continue only the field of view control from the operation control it was performing before becoming the main camera, and maintain the shooting direction as they were immediately before becoming the main camera.

[0226] In S708, CPU 101 enables remote control of the main camera by the remote control device 800. Subsequently, CPU 101 performs the operations described using Figure 6(a).

[0227] Figure 21 is a timing chart of the operation described using Figure 20. Here, it is assumed that the main camera was changed from camera 500 to camera 400 at time t1. During the period prior to time t1, camera 500 is set as the main camera, and remote control by the remote control device 800 is enabled. Camera 400 is set as a sub-camera, and the shooting control device 100 automatically controls camera 400 according to the set control content.

[0228] When the main camera is changed from camera 500 to camera 400 at time t1, camera 500 becomes a sub-camera. The shooting control device 100 then automatically controls camera 500 according to the set control settings. Meanwhile, camera 400 becomes the main camera, but the shooting control device 100 continues to automatically track the subject that was being tracked at time t1 until the difference in attitude between the camera 400 and the control unit 812 falls below a threshold. Remote control by the remote control device 800 is disabled.

[0229] When the difference in attitude between the camera 400 and the control unit 812 falls below a threshold at time t2, remote control of the camera 400 by the remote control device 800 becomes active. The period from time t1 to t2 is the period during which the remote control device 800 is inactive.

[0230] (modified version) In this embodiment, the generation and transmission of control commands to the main camera was described as being performed by the shooting control device 100. However, the generation and transmission of control commands to the main camera may be performed by the remote control device 800 (CPU 801). In this case, the operation of the shooting control device 100 does not need to be changed except to control only the operation of the sub-camera.

[0231] If the main camera is changed, the CPU 801 obtains the attitude of the main camera after the change, and if the difference between this attitude and the attitude of the control unit 812 exceeds a threshold, the CPU 801 disables the control unit 812. The CPU 801 can disable the control unit 812, for example, by not sending control commands to the main camera even if the control unit 812 is operated.

[0232] Furthermore, when the main camera settings are changed, the display unit 808 of the remote control device 800 will display the live view image of the main camera 400 after the change. However, if the orientation of the control unit 812 and the live view image do not match, it may confuse the operator. For this reason, the CPU 801 may, when the main camera is changed, acquire the orientation of the main camera after the change and refrain from displaying the live view image on the display unit 808 for the period during which the difference between the orientation of the control unit 812 and the new orientation exceeds a threshold.

[0233] The above explanation shows an example in which, when the difference between direction information A and direction information B is large, the system automatically tracks the subject before the change without using the pan-tilt control values ​​of the remote control device 800. In this case, the CPU 101 may be configured not to use the acquired control values ​​of the remote control device 800, or alternatively, the CPU 101 may be controlled so that it does not transmit the pan-tilt angle from the remote control device 800.

[0234] According to this embodiment, the orientation of a camera located remotely is controlled to correspond to the orientation of a physical control unit operated by the user. Therefore, the user can remotely control the shooting direction of the camera with the same feeling as if they were directly operating the camera.

[0235] Furthermore, if the remotely controlled camera is changed, remote control is disabled until the difference between the orientation of the control unit and the orientation of the newly remotely controlled camera falls below a threshold. Therefore, even if the orientation of the control unit and the orientation of the remotely controlled camera differ significantly, it is possible to prevent the camera's shooting direction from being controlled in an unintended direction.

[0236] ●<Second Embodiment> Next, a second embodiment of the present invention will be described. This embodiment relates to the display on the remote control device 800 during the invalid period of the remote control device 800 in the first embodiment. Since this embodiment can be implemented with the imaging system described in the first embodiment, the same matters as in the first embodiment will be omitted.

[0237] First, we will explain an example of displaying a notification to the user of the remote control device 800 that the remote control device 800 is disabled during the period when the remote control device 800 is disabled, using the flowchart shown in Figure 23(a).

[0238] S803 can be performed when the CPU 101 first executes S707 in Figure 20. The CPU 101 notifies the remote control device 800 via the network interface 105 that the operation is invalid. At this time, the CPU 101 may also notify the attitude difference calculated in S706.

[0239] In S804, the CPU 801 of the remote control device 800 receives a notification from the shooting control device 100. The CPU 801 then displays information on the display unit 808 indicating that the operation of the operation unit 812 is invalid.

[0240] Figure 22(a) shows an example where a circular mark indicating that the operation of the control unit 812 is disabled is displayed on the display unit 808. The shape of the mark may be any other shape, such as a square or a star. Also, if a difference in posture is notified, the size and number of marks may be changed according to the magnitude of the posture difference. For example, the posture difference may be divided into multiple stages, and marks of the size and number corresponding to each stage may be displayed. In addition, a message such as "The posture of the main camera and the posture of the control unit are misaligned, so operation is temporarily disabled" may be displayed instead of the mark, or in addition to the mark.

[0241] Subsequently, when the CPU 101 executes S708 in Figure 20, it notifies the remote control device 800 that the operation is valid. In response to this notification, the CPU 801 terminates the display that was started in S804. At this time, the CPU 801 may display a message such as "Operation has been enabled" on the display unit 808.

[0242] In this way, by informing the user that the operation is invalid via the remote control device 800, the user can be prompted to adjust the orientation of the control unit to match the orientation of the main camera.

[0243] Next, an example of displaying information to inform the user of the magnitude and direction of the difference in posture between the camera being operated and the operating unit 812 during the period when the remote control device 800 is inactive will be explained using the flowchart shown in Figure 23(b).

[0244] S903 can be performed when the CPU 101 first executes S707 in Figure 20. The CPU 101 notifies the remote control device 800 via the network interface 105 that the operation is invalid. At this time, the CPU 101 also notifies the attitude information A and attitude information B (or the attitude difference calculated in S706) acquired in S704 and S705.

[0245] In S904, the CPU 801 of the remote control device 800 receives a notification from the shooting control device 100. The CPU 801 then displays information on the display unit 808 indicating the direction and magnitude of the difference in attitude between the main camera and the operation unit 812.

[0246] For example, if the attitude information includes pan angle and tilt angle, the CPU 801 generates a vector indicating the shooting direction of the main camera from the attitude information of the main camera, and a vector indicating the shooting direction corresponding to the attitude of the control unit 812 from the attitude information of the control unit 812. Then, the CPU 801 converts each of these vectors into a two-dimensional vector projected onto a vertical plane, for example. Furthermore, the CPU 801 displays the information based on the two-dimensional vector on the display unit 808.

[0247] Figure 23(b) shows an example where an arrow indicating the orientation (shooting direction) of the main camera and an arrow indicating the shooting direction corresponding to the orientation of the control unit 812 are displayed. The user can understand the direction and magnitude of the difference in orientation by the difference in the direction of the arrows. Therefore, they can understand which way to operate the control unit 812 to match the orientation of the main camera.

[0248] Other display methods are also possible. The CPU 801 calculates the difference in pan angle and tilt angle based on the attitude information of the main camera and the control unit 812. The CPU 801 then converts the direction and magnitude of the difference in pan angle into a horizontal vector, and the direction and magnitude of the difference in tilt angle into a vertical vector. The CPU 801 then displays arrows based on these vectors on the display unit 808. By showing the direction and magnitude of the difference between tilt angle and pan angle with separate arrows, the user can correct the attitude deviation in each direction.

[0249] Note that marks other than arrows may be used. For example, different colors may be used to indicate direction, and the intensity of the color may indicate size. Alternatively, direction may be indicated with positive or negative signs, and size with numerical values. In addition, a message may be displayed instructing the user to perform the necessary operation to eliminate the difference, such as "Pan to the right by ○ degrees."

[0250] During the invalidation period, CPU 101 and CPU 801 repeatedly execute S903 and S904. This updates the display content of the display unit 808, allowing the user to quickly correct the difference in posture between the main camera and the control unit 812 by operating the control unit 812 while looking at the display unit 808 to reduce the difference in posture.

[0251] The two display examples described here may be combined. Furthermore, although the above operations were described assuming they are performed by CPU 101 and CPU 801, all operations may be performed by CPU 801 alone. In this case, when a new main camera is notified from the role control device 600, CPU 801 should acquire attitude information from the main camera and attitude calculation unit 807, and then execute the processes shown in the flowcharts of Figures 23(a) and (b).

[0252] According to this embodiment, even if the main camera is changed and the control unit is disabled due to a difference in orientation between the main camera and the control unit, the user can quickly correct the misalignment between the orientation of the main camera and the control unit and enable remote control of the main camera by the control unit.

[0253] (Other embodiments) In the above-described embodiment, the image capture control device 100 and the remote control device 800 were treated as separate devices, but they may be the same device. Therefore, the remote control device 800 can also function as the image capture control device 100. Conversely, the image capture control device 100 can also function as the remote control device 800.

[0254] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0255] This embodiment includes the following imaging control device, imaging control method, and program. (Item 1) A control means for controlling the operation of the second imaging device, among a plurality of imaging devices including a first imaging device and a second imaging device, based on information about the first imaging device and set control content. When the first imaging device is changed, the system includes a determination means for determining whether the difference between the orientation of the operating member for remotely controlling the shooting direction of the first imaging device and the orientation of the first imaging device after the change is less than or equal to a threshold. If the control means determines that the difference is not less than or equal to the threshold, it disables the remote control by the operating member until it determines that the difference is less than or equal to the threshold. A photographic control device characterized by the following: (Item 2) While the remote control by the operation member is invalid, the control means controls the operation of the changed first imaging device based on the control content when the changed first imaging device was the second imaging device. The imaging control device according to item 1, characterized in that. (Item 3) While the remote control by the operation member is valid, the first imaging device is controlled so that its attitude becomes the same as the attitude of the operation member. The imaging control device according to item 1 or 2, characterized in that. (Item 4) The operation member includes a pan / tilt head capable of panning and tilting. The imaging control device according to any one of items 1 to 3, characterized in that. (Item 5) The operation member has display means, The display means is provided so that the screen is positioned in the front when the pan angle is the initial value. The imaging control device according to item 4, characterized in that. <00009�8>(Item 6) The operation member has display means, A live view image of the first imaging device is displayed on the display means. The imaging control device according to any one of items 1 to 3, characterized in that. (Item 7) While the remote control by the operation member is invalid, the live view image is not displayed. The imaging control device according to item 4, characterized in that. (Item 8) }While the remote control by the operation member is invalid, information indicating that the remote control by the operation member is invalid is displayed on the display means. The imaging control device according to item 6 or 7, characterized in that. (Item 9) While the remote control by the operation member is invalid, information indicating the direction and magnitude of the difference between the attitude of the first imaging device and the operation member is displayed on the display means. The imaging control device according to any one of items 6 to 8, characterized in that. (Item 10) The shooting control device according to any one of items 1 to 9, characterized in that the control means controls at least one of the tracked subject and the angle of view of the second imaging device. (Item 11) A shooting control method performed by a shooting control device, Among a plurality of imaging devices, including a first imaging device and a second imaging device, the operation of the second imaging device is controlled based on information about the first imaging device and the set control content. When the first imaging device is changed, it is determined whether the difference between the orientation of the operating member for remotely controlling the shooting direction of the first imaging device and the orientation of the first imaging device after the change is less than or equal to a threshold. If it is determined that the difference is not less than or equal to the threshold, the remote control by the operating member is disabled until it is determined that the difference is less than or equal to the threshold. A method for controlling photography, characterized by having the following features. (Item 12) A program for causing a computer to function as one of the means of a photographic control device described in any one of items 1 through 10.

[0256] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0257] 100...Shooting control device, 300...Overhead camera, 400...Sub-camera, 500...Main camera, 800...Remote control device, 101, 801...CPU, 106...User input unit, 808...Display unit, 812...Operation unit

Claims

1. A control means for controlling the operation of the second imaging device, among a plurality of imaging devices including a first imaging device and a second imaging device, based on information about the first imaging device and set control content. When the first imaging device is changed, the system includes a determination means for determining whether the difference between the orientation of the operating member for remotely controlling the shooting direction of the first imaging device and the orientation of the first imaging device after the change is less than or equal to a threshold. If the control means determines that the difference is not below the threshold, it disables the remote control by the operating member until it determines that the difference is below the threshold. A photographic control device characterized by the following features.

2. The imaging control device according to claim 1, characterized in that the control means controls the operation of the modified first imaging device based on the control content when the modified first imaging device was the second imaging device, while the remote control by the operating member is ineffective.

3. The imaging control device according to claim 1, characterized in that while the remote control by the operating member is effective, the first imaging device is controlled so that its posture is the same as the posture of the operating member.

4. The shooting control device according to claim 1, characterized in that the operating member comprises a pan head capable of panning and tilting.

5. The operating member has a display means, The display means is positioned so that the screen is facing forward when the pan angle is at its initial value. The imaging control device according to claim 4.

6. The operating member has a display means, The display means displays the live view image of the first imaging device. The photographic control device according to claim 1.

7. The shooting control device according to claim 6, characterized in that the live view image is not displayed while the remote control by the operating member is invalid.

8. While the remote control by the operating member is disabled, the display means displays information indicating that the remote control by the operating member is disabled. The imaging control device described in Item 6.

9. While the remote control by the operating member is ineffective, the display means displays information indicating the direction and magnitude of the difference between the orientation of the first imaging device and the operating member. The imaging control device described in Item 6.

10. The shooting control device according to claim 1, characterized in that the control means controls at least one of the tracked subject and the field of view of the second imaging device.

11. A shooting control method performed by a shooting control device, Among a plurality of imaging devices, including a first imaging device and a second imaging device, the operation of the second imaging device is controlled based on information regarding the first imaging device and the set control content. When the first imaging device is changed, it is determined whether the difference between the orientation of the operating member for remotely controlling the shooting direction of the first imaging device and the orientation of the first imaging device after the change is less than or equal to a threshold. If it is determined that the difference is not less than or equal to the threshold, the remote control by the operating member is disabled until it is determined that the difference is less than or equal to the threshold. A method for controlling photography, characterized by having the following features.

12. A program for causing a computer to function as one of the means of the imaging control device described in any one of claims 1 to 10.

Images