Camera control device and camera control method, and multi-camera system

Abstract

To provide an imaging control device that can solve problems that may arise when changing the settings of multiple imaging devices at once. [Solution] This is a shooting control device that is communicatively connected to multiple cameras and capable of controlling at least one of the shooting direction and field of view for each of the multiple cameras. When changing the settings of multiple cameras at once, the shooting control device determines which camera will have its settings changed first, instructs the first camera to change its settings, and determines whether the instructed setting change has been completed in the first camera. If the shooting control device determines that the instructed setting change has been completed in the first camera, it instructs the other cameras to change their settings.

Description

[Technical Field]

[0001] The present invention relates to a camera control device and a camera control method, as well as a multi-camera system, and more particularly to a camera control technology for camera shooting using multiple imaging devices. [Background technology]

[0002] A shooting system using multiple imaging devices (hereinafter referred to as a multi-camera system) is known (Patent Document 1). Conventional multi-camera systems control the switching of the video output from the system among the video input from multiple cameras. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2014-197831 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] For example, in a multi-camera system using cameras whose field of view and shooting direction can be remotely controlled, there is a need to change the shooting direction and field of view of all cameras at a specific time. However, changing the shooting direction and field of view takes time.

[0005] Therefore, if settings are changed for all cameras at once, there may be a period during which all cameras are undergoing the settings change. If the footage captured during this period is unsuitable in terms of content or quality, the video feed will be interrupted during the change. This can be a problem, for example, when shooting live video. Similar problems can occur when changing other settings.

[0006] In one embodiment, the present invention provides an imaging control device and an imaging control method that can solve problems that may arise when changing the settings of multiple imaging devices at once. [Means for solving the problem]

[0007] In one aspect, the present invention provides a camera control device that is communicatively connected to a plurality of cameras and capable of controlling at least one of the shooting direction and field of view for each of the plurality of cameras, and comprises a control means, the control means having, when changing the settings of a plurality of cameras at once, determine which camera among the plurality of cameras will have its settings changed first, instruct the first camera to change its settings, determine whether the instructed setting change has been completed in the first camera, and, depending on the determination that the instructed setting change has been completed in the first camera, instruct the other cameras among the plurality of cameras to change their settings. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an imaging control device and an imaging control method that can solve problems that may arise when changing the settings of multiple imaging devices at once. [Brief explanation of the drawing]

[0009] [Figure 1] Schematic diagram of the multi-camera system according to the first embodiment [Figure 2] Block diagram showing an example configuration of each device in the multi-camera system shown in Figure 1. [Figure 3] This figure shows an example of a settings screen provided by the controller 400 according to the first embodiment. [Figure 4] Flowchart relating to the operation of the controller 400 according to the first embodiment [Figure 5] Timing chart of an example of batch change processing of shooting settings according to the first embodiment [Figure 6] Timing chart of another example of batch change processing of shooting settings according to the first embodiment [Figure 7] Flowchart relating to the operation of the controller 400 and camera 100 according to the first embodiment [Figure 8] Schematic diagram of the multi-camera system according to the second embodiment [Figure 9] Block diagram showing a configuration example of each device of the multi-camera system in FIG. 8 [Figure 10] Flowchart regarding the operation of each device of the multi-camera system according to the second embodiment [Figure 11] Schematic diagram of control according to the role of the camera in the second embodiment [Figure 12] Schematic diagram of control according to the role of the camera in the second embodiment [Figure 13] Diagram showing an example of the relationship between the roles that can be set for cameras 100 and 200 in the second embodiment and the operation control [Figure 14] Schematic diagram of another control according to the role of the camera in the second embodiment [Figure 15] Flowchart regarding the operation of the controller 1400 in the second embodiment [Figure 16] Diagram for explaining the calculation of the pan value in the second embodiment [Figure 17] Diagram for explaining the calculation of the tilt value in the second embodiment [Figure 18] Diagram showing an example of the mapping of the zoom values between the main camera and the sub-camera in the second embodiment

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] ●<First Embodiment> Figure 1 is a schematic diagram of the overall configuration of a multi-camera system 10 according to the first embodiment. The multi-camera system 10 includes a plurality of cameras 100 and 200, and a controller 400 which acts as a shooting control device that controls the operation of cameras 100 and 200. Each of the cameras 100 and 200 and the controller 400 are configured to communicate with each other via a network 300. The network 300 may be part of the multi-camera system 10 or it may be an external network. In addition, the cameras 100 and 200 and the controller 400 may be directly connected. In this case, the network 300 is not necessary.

[0012] Figure 1 shows a scenario where subject 20 is photographed using cameras 100 and 200, but the number of subjects and cameras is merely an example. Cameras 100 and 200 are PZT cameras, and their shooting direction (pan and tilt angles) and field of view (zoom) can be remotely controlled from controller 400.

[0013] Figure 2 is a block diagram showing an example configuration of cameras 100, 200, and controller 400. For the sake of simplicity, in this embodiment, cameras 100 and 200 are assumed to have the same configuration. Figure 2 shows only the components necessary to explain the following operations.

[0014] The CPU 101 controls the operation of the components of the camera 100 by executing a computer program loaded from the ROM 103 into the RAM 102, thereby realizing the operation of the camera 100 described later.

[0015] RAM 102 is a high-speed memory device such as DRAM. RAM 102 has areas for storing computer programs and data loaded from ROM 103, and areas for storing images output from image processing unit 106. In addition, RAM 102 has areas for storing various information received from controller 400 via network interface (I / F) 105, and a work area used by CPU 101 when executing various processes. In this way, RAM 102 can appropriately provide areas for storing various types of data.

[0016] ROM103 is a rewritable, non-volatile storage device such as a semiconductor memory card or SSD. ROM103 stores settings data for camera 100, computer programs and data related to camera 100's operation, and other similar information.

[0017] The network interface 105 is an interface for connecting camera 100 to network 300. The network interface 105 can operate in accordance with one or more known wired and wireless communication standards. Camera 100 can communicate with external devices such as camera 200 and controller 400 connected to network 300 via the network interface 105. However, as mentioned above, camera 100 may communicate directly with external devices such as camera 200 and controller 400 without going through network 300.

[0018] The image sensor 107 comprises an imaging optical system and an image sensor. The image sensor may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The image sensor has a pixel array in which multiple pixels are arranged in two dimensions, and peripheral circuits for reading signals from each pixel. Each pixel accumulates charge according to the amount of incident light by photoelectric conversion. By reading signals with a voltage corresponding to the amount of charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface is obtained.

[0019] The image processing unit 106 applies predetermined signal processing and image processing to the analog image signal output by the image sensor 107 to generate signals and image data according to the application, and to acquire and / or generate various types of information.

[0020] The processing applied by the image processing unit 106 may include, for example, preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effects processing. Preprocessing may include A / D conversion, signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is performed when a color filter is provided on the image sensor 107, and it is a process that interpolates the values ​​of color components that are not included in the individual pixel data that make up the image data. Color interpolation is also called demosaicing. Correction processing may include white balance adjustment, gradation correction, correction of image degradation caused by optical aberrations in the imaging optical system (image recovery), correction of the effects of vignetting in the imaging optical system, and color correction. Data processing may include processes such as region extraction (trimming), merging, scaling, encoding and decoding, and header information generation (data file generation). The generation of video signals to be output externally and video data to be recorded in ROM103 are also included in data processing. The evaluation value calculation process may include generating signals and evaluation values ​​used for autofocus detection (AF), and generating evaluation values ​​used for automatic exposure control (AE). AF and AE are executed by CPU 101. Special effects processing may include adding blur effects, changing color tones, and relighting. These are merely examples of processes that the image processing unit 106 can apply, and do not limit the processes that the image processing unit 106 can apply. The image processing unit 106 outputs the acquired or generated information and data to the CPU 101, RAM 102, etc., depending on the application.

[0021] The type and settings of processing applied by the image processing unit 106 can be controlled by sending commands from the controller 400 to the camera 100. Alternatively, the image processing unit 106 may perform the above-described processing according to the commands received from the controller 400.

[0022] The drive I / F 108 is a communication interface between the CPU 101 and the drive unit 109. The drive unit 109 has a drive mechanism for changing the shooting direction and field of view of the camera 100, and a drive source such as a motor. Specifically, the drive unit 109 can independently control the horizontal (lateral) angle (pan angle) and the vertical (vertical) angle (tilt angle) of the optical axis of the camera 100's shooting optical system. The drive unit 109 can also drive a lens group (zoom lens) that changes the field of view of the shooting optical system. Note that the drive unit 109 only needs to be able to control at least one of the pan angle, tilt angle, and field of view (zoom value).

[0023] Therefore, the shooting direction and field of view of the camera 100 can be controlled by sending commands to the drive unit 109 via the drive I / F 108 from the CPU 101 to control pan, tilt, and zoom.

[0024] The video output interface 110 is an interface for outputting video signals generated by the image processing unit 106 to an external source. The video output interface 110 may be an interface compliant with standards such as SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interface) (registered trademark).

[0025] The CPU 101, RAM 102, ROM 103, network I / F 105, image processing unit 106, drive I / F 108, and video output I / F 110 mentioned above are connected to the system bus 111.

[0026] Camera 200 includes a CPU 201, RAM 202, ROM 203, video output I / F 210, image processing unit 206, image sensor 207, and video output I / F 210. The details of the components of camera 200 are the same as those described for the components of the same name in camera 100.

[0027] The controller 400 may be a general-purpose computer device (such as a personal computer) that runs the shooting control application. The controller 400 receives video signals transmitted from cameras 100 and 200 via the network 300 and sends control signals (commands) to cameras 100 and 200.

[0028] For example, the controller 400 can send a command to cameras 100 and 200 specifying one or more of the pan angle, tilt angle, and zoom value based on operations received through the user input interface 406. The controller 400 can also send a command to cameras 100 and 200 specifying the shooting size of the tracked subject based on operations received through the user input interface 406. The shooting size may be, for example, the size of the rectangle (number of pixels vertically, horizontally, or diagonally) that circumsects the area of ​​the tracked subject in the image.

[0029] In this way, the user can instruct the shooting settings of cameras 100 and 200 by operating the user input interface 406. Shooting settings include, for example, pan value, tilt value, zoom value, focus value, image quality mode, and, in the case of tracking photography, the subject to be photographed, the subject's shooting size, the subject's shooting composition, and the tracking speed. One or more of these shooting settings may be automatically specified by the controller 400.

[0030] The CPU 401 controls the operation of the components of the controller 400 by executing the computer program loaded from the ROM 403 into the RAM 402, thereby realizing the operation of the controller 400 described later.

[0031] RAM402 is a high-speed memory device such as DRAM. RAM402 has an area for storing computer programs and data loaded from ROM403, and an area for storing video signals received from cameras 100 and 200 via network I / F404. RAM402 also has a work area used by CPU401 when executing various processes. In this way, RAM402 can appropriately provide areas for storing various types of data.

[0032] ROM403 is a rewritable, non-volatile storage device such as a semiconductor memory card or SSD. ROM403 stores configuration data for the controller 400, computer programs and data related to the controller 400's operation, and the aforementioned shooting control application.

[0033] The network interface 404 is an interface for connecting the controller 400 to the network 300. The network interface 404 can operate in accordance with one or more known wired and wireless communication standards. The controller 400 can communicate with external devices such as cameras 100 and 200 connected to the network 300 through the network interface 404. As mentioned above, the controller 400 may also communicate directly with external devices such as cameras 100 and 200 without going through the network 300.

[0034] The display unit 405 may be a liquid crystal display or an organic EL display. The display unit 405 displays video signals received from cameras 100 and 200, and screens provided by programs (OS, shooting control applications, etc.) running on the controller 400.

[0035] In the following description, the display unit 405 is assumed to be a touch display. Although Figure 2 shows the display unit 405 built into the controller 400, it may also be configured to be connected as an external device.

[0036] The user input interface 406 is an input device for the user to input instructions to the controller 400. The user input interface 406 includes, for example, one or more of the following: a mouse, keyboard, buttons, dial, joystick, touch panel, etc.

[0037] The CPU 401, RAM 402, ROM 403, network interface 404, display unit 405, and user input interface 406 are connected to the system bus 407.

[0038] Figure 3 shows an example of a settings screen provided by the shooting control application running on the controller 400. Note that the settings screen shown in Figure 3 is just an example, and the layout and displayed items may be changed.

[0039] The settings screen has a video area 501 and an operation area 506. The video area 501 is an area that individually displays the video received from the cameras controlled by the controller 400. In the example shown in Figure 3, the video area 501 has four sub-areas 502 to 505, and is configured to display video from up to four cameras. In this embodiment, since the controller 400 controls two cameras, the video received from camera 100 is displayed in sub-area 502, and the video received from camera 200 is displayed in sub-area 503. A label area is provided in the upper left of each sub-area to display the name of the camera that is capturing the video. For convenience, "camera 100" and "camera 200" are displayed here, but in reality, the model name of each camera or a name assigned by the user will be displayed.

[0040] When a user changes the camera's shooting settings, the user first selects the camera whose shooting settings they want to change from the video area 501. The CPU 401 selects the camera according to the small area or label area touched in the video area 501. The CPU 401 may also provide feedback to the user that the selection has been accepted by making the display of the touched small area and label area different from the display of other small areas. In Figure 3, the CPU 401 thickens the border of small area 502 and inverts the label area in response to the detection of a touch operation in small area 502.

[0041] If a touch operation is detected in the small area or label area corresponding to the selected camera, the CPU 401 deselects that camera and returns to the display mode of deselection. The CPU 401 also allows the selection of multiple cameras through operations on the video area 501.

[0042] The operation area 506 is the area for specifying the shooting settings for the selected camera. The operation area 506 has sub-areas corresponding to the items to be specified. Here, as an example, the operation area 506 is assumed to have three sub-areas 507 and 509, but the number of sub-areas is not limited to two. Note that although sub-area 510 does not directly specify shooting settings, it is included in the operation area 506 because it is an area for setting the method of specification.

[0043] The small area 507 is the area for specifying presets. A preset is a set of pre-set values ​​for multiple setting items. Here, a preset is assumed to be a specific combination of pan angle, tilt angle, and zoom value, but the types and number of items that can be preset may be changed. For example, the focus position on the screen or the drive speed of the drive unit 109 may be included as preset items. In the example shown in Figure 3, four presets can be registered, and the small area 507 has preset buttons 508 corresponding to each preset, but the number of registerable presets is not limited to four.

[0044] In this embodiment, settings for each camera can be registered in a single preset. That is, "Preset 1" can independently register settings for camera 100 and settings for camera 200. As a result, by simply specifying "Preset 1," individual settings can be specified for both camera 100 and camera 200 in a single operation.

[0045] When CPU401 detects a touch operation on one of the preset buttons 508, it reads the preset corresponding to the operated preset button from, for example, ROM403. Then, for each selected camera, CPU401 extracts the setting value from the preset and sends a command specifying the item and setting value via network I / F404. Further details will be described later.

[0046] Small area 509 is an area for manually specifying the camera's shooting direction and field of view, and includes user interface (UI) elements for changing the pan and tilt angles, and UI elements for changing the field of view (zoom value). The UI elements for changing the pan and tilt angles include buttons corresponding to the up, down, left, and right directions. The up and down buttons are UI elements for changing the tilt angle, and the right and left buttons are UI elements for changing the pan angle. For example, one press of a button corresponds to an instruction to rotate by a unit angle in the corresponding direction.

[0047] Furthermore, the UI element for changing the zoom value is a slider that can be moved up and down. For example, moving the slider upwards corresponds to zooming in, and moving it downwards corresponds to zooming out. The position of the slider indicates the current zoom value (field of view).

[0048] When CPU401 detects an operation on these UI elements, it generates a command corresponding to the detected operation and sends it to the selected camera via network I / F404.

[0049] The small area 510 is a switch that specifies whether or not to use the "coordinated" method for changing settings for multiple cameras at once (whether or not to coordinate the change operations). Details of the small area 510 will be described later.

[0050] As an example, we will describe an operation in which the user can collectively change the shooting direction and field of view of cameras 100 and 200 from the state in which they are shooting the image shown in Figure 3(a) to the state in which they are shooting the image shown in Figure 3(b) by specifying a preset.

[0051] First, the user registers presets in advance. Specifically, the user registers the shooting direction and field of view for camera 100 to capture the image shown in Figure 3(b) as preset 1 for camera 1. Similarly, the user registers the shooting direction and field of view for camera 200 to capture the image shown in Figure 3(b) as preset 1 for camera 2. Preset registration is performed through a separate settings screen provided by the shooting control application, and the registered presets are stored in ROM 403.

[0052] Subsequently, in the state shown in Figure 3(a), the user selects cameras 100 and 200 by operating the small area 502 corresponding to camera 100 and the small area 503 corresponding to camera 200. Then, with cameras 100 and 200 selected, the user operates the "Preset 1" button in small area 507.

[0053] As a result, the controller 400 controls cameras 100 and 200 to use the shooting direction and field of view registered as preset 1. At this time, the controller 400 performs setting change operations according to the batch change method ("normal" or "cooperative") specified in the small area 510, as will be described later. In this way, the user can change the shooting settings of cameras 100 and 200 all at once. Although this explanation describes the case where the user changes the shooting settings by operating the preset buttons, the controller 400 may also change the settings automatically when predetermined conditions are met. For example, the controller 400 can automatically change the shooting settings of cameras 100 and 200 all at once when a certain amount of time has elapsed from a reference time.

[0054] The operation of cameras 100 and 200, as well as controller 400, to achieve the simultaneous change of the shooting settings described above will be explained. Figure 4 is a flowchart illustrating the operation of the controller 400 when changing settings for multiple cameras simultaneously. The operations described below are performed by the CPU 401 executing the shooting control application.

[0055] In the case of changing settings for multiple cameras at once, for example, when a setting change instruction (operation of preset button 508) is detected on the settings screen shown in Figure 3, multiple cameras are selected. The CPU 401 recognizes a setting change instruction as a batch change instruction if multiple cameras are selected when the instruction for a setting change is detected. Note that this is just one example of a batch change instruction, and setting change instructions may be given to multiple cameras by other methods.

[0056] On the other hand, if only one camera is selected when a setting change instruction is detected, CPU401 sends a command to the selected camera to specify the new settings. If no camera is selected when a setting change instruction is detected, CPU401 ignores the instruction and displays a message such as "Please select a camera."

[0057] Here, as an example, we assume that when cameras 100 and 200 are selected, an operation on one of the preset buttons 508 is detected, and the flowchart shown in Figure 4 is executed. The CPU 401 stores the type of preset button that was operated and the state of area 510 (whether the linkage is on or not) in, for example, RAM 402.

[0058] In S401, CPU401 determines the order in which the settings of the cameras to be changed (cameras 100 and 200) will be modified. The order can be determined arbitrarily; for example, it may be the order selected through the settings screen shown in Figure 3, or it may be the order in which the cameras establish a connection with controller 400. Here, as an example, we assume that CPU401 has decided to change the shooting settings of camera 100, then camera 200.

[0059] Next, in S402, the CPU 401 determines whether the batch change method is "normal" or "cooperative". If the batch change method is "cooperative", it executes S405; otherwise, it executes S403. The CPU 401 determines that the batch change method is "cooperative" if the state of area 510 at the time of detecting the batch change instruction is on, and "normal" if it is off.

[0060] First, let's explain the case where the batch change method is "normal". In S403, the CPU 401 sends commands to change the shooting settings to the cameras whose shooting settings are to be changed, either simultaneously or sequentially starting with the camera in the order of priority, via the network I / F 404. Note that if the batch change method is "normal" and commands are not sent simultaneously, the order of transmission does not necessarily have to follow the order determined in S401.

[0061] As described above, CPU 401 extracts the shooting settings for cameras 100 and 200 whose settings are to be changed from the camera-specific shooting settings included in the preset specified by the user. Then, CPU 401 generates commands to send to camera 100 and commands to send to camera 200 based on the extracted shooting settings.

[0062] Subsequently, the CPU 401 sends the generated commands to each camera via the network interface 404. When the CPU 101 of camera 100 receives a command via the network interface 105, it changes the shooting settings according to the command. For example, if the command specifies target values ​​for the pan angle and tilt angle, the CPU 101 calculates the difference (drive amount) between these target values ​​and the current pan angle and tilt angle. Then, the CPU 101 generates a command to drive the drive unit 109 in the direction based on the sign of the difference and the angle based on the magnitude of the difference, and sends it to the drive unit 109 via the drive interface 108. This changes the shooting direction of camera 100. The CPU 101 may also notify the command source device (in this case, the controller 400) that the operation according to the received command (in this case, the change in shooting settings) has been completed.

[0063] If one or more specific shooting directions are pre-registered in ROM 103, the command sent from controller 400 to the camera only needs to include identification information (such as a number) indicating one of the specific shooting directions. The operation of CPU 201 in camera 200 is the same as that of CPU 101, so the explanation is omitted.

[0064] When the batch change method is set to "normal," it allows multiple cameras to have their settings changed simultaneously, so you can send commands to change the shooting settings to the target cameras at any time you like. Therefore, even if there are many target cameras, batch settings can be changed, and the time required for setting changes is reduced.

[0065] Next, we will explain the case where the batch change method is "cooperative". In S405, CPU401 generates a command to change the shooting settings for the camera that is the first in the sequence determined by S401 among the cameras whose shooting settings are to be changed. Then, CPU401 sends the generated command to the target camera via network I / F404.

[0066] In S406, CPU401 obtains the current shooting settings from the first camera. In S407, CPU401 determines whether the shooting settings have been changed for the camera that received the command in S405. Specifically, CPU401 determines whether the setting values ​​obtained in S406 reflect the specified setting changes.

[0067] For example, if a command specifying the pan angle and shift angle is sent in S405, CPU401 obtains the current pan angle and tilt angle from the camera. Then, CPU401 determines that the change in shooting settings is complete if the obtained current pan angle and tilt angle match the pan angle and tilt angle specified in S405, and determines that it is incomplete if they do not match.

[0068] Even if the current setting does not match the instructed setting, it may be determined that the shooting setting has been changed if certain conditions are met. For example, if the difference between the current setting and the instructed setting is below a threshold, it can be determined that the setting has been changed. This is because a value close to the instructed setting is more likely to produce an image closer to the desired image.

[0069] When changing settings for multiple items, CPU401 in S407 can determine that the shooting settings have been changed if one or more of the items match the specified setting value, or if the difference between all of the items and the specified value is less than or equal to a threshold (e.g., a predetermined percentage).

[0070] Also, regarding the shooting direction (at least one of the pan angle and the tilt angle), if the specified shooting direction is included in the shooting angle, it may be determined that the setting change has been completed. For example, the CPU 401 can determine that the specified shooting direction is included in the shooting angle when all of the following formulas (1) to (4) are satisfied. Pan target >Pan cur -Zoom_h cur ···(Formula 1) Pan target <Pan cur +Zoom_h cur ···(Formula 2) Tilt target >Tilt cur -Zoom_v cur ···(Formula 3) Tilt target <Tilt cur +Zoom_v cur ···(Formula 4)

[0071] In Formulas 1 to 4, Pan target , Tilt target are the pan angle and the tilt angle after the change. Also, Pan cur , Tilt cur are the current pan angle and tilt angle of the camera whose settings are being changed. Zoom_h cur is the magnitude of the pan angle corresponding to half of the current horizontal shooting angle of the camera whose settings are being changed, and Zoom_h cur is the magnitude of the tilt angle corresponding to half of the current vertical shooting angle of the camera whose settings are being changed. Therefore, in the horizontal direction, Pan cur ±Zoom_h cur , and in the vertical direction, Tilt cur ±Zoom_h cur is the current shooting range.

[0072] Alternatively, instead of determining whether the difference between the specified value and the current value is below a threshold, it may be determined whether the amount of change in the set value has decreased. This is because, in configurations that mechanically drive a component to a target position, such as the drive unit 109 of the camera 100, it is common practice to reduce the movement speed from just before the component reaches the target position, and to implement control such that the movement speed becomes 0 at the target position. Therefore, if the movement speed (amount of change in position or angle) decreases, it is considered that the component is approaching the target position.

[0073] Therefore, the CPU 401 calculates the amount of change for each item (pan angle, tilt angle, zoom lens position) from the time series of current settings acquired in the loops of S406 and S407. The CPU 401 can then determine that the change in shooting settings is complete for items where the amount of change has decreased (movement speed has decreased).

[0074] Furthermore, to simplify the determination process, the CPU 401 may determine that the change in shooting settings is complete if the elapsed time since sending the command to change the shooting settings exceeds a predetermined time. The predetermined time is predetermined according to the shooting setting item to be changed, for example. For example, a longer predetermined time (e.g., 4 seconds) can be set for items that involve the operation of the drive unit 109 when changed, such as the pan angle, tilt angle, or zoom value. Conversely, a shorter predetermined time (e.g., 1 second) can be set for items that do not involve the operation of the drive unit 109, such as changing setting values ​​stored in ROM.

[0075] Furthermore, for items whose changes involve the operation of the drive unit 109, the length of the predetermined time may be varied depending on the amount of change. For example, the CPU 401 obtains the current value before executing S405 and calculates the difference (amount of change) between this value and the value specified in the command sent in S405. The CPU 401 may then divide the amount of angle change (absolute value) into six equal parts between 0 and 180° and set six predetermined times so that the predetermined time gradually increases. Alternatively, the CPU 401 may divide the amount of change by the average movement speed of the drive unit to predict the time required to change the setting and use the predicted time as the predetermined time. When changing multiple items, the CPU 401 calculates the predicted time for each item and uses the maximum predicted time as the predetermined time. When the movement speed is specified in the command, the CPU 401 calculates the predicted time using the movement speed specified in the command.

[0076] Furthermore, the CPU 401 may change the order determined in S401 to the order with the shortest estimated time calculated in this way before executing S405. This allows the controller 400 to change the shooting settings of cameras that can have their settings changed quickly, thereby shortening the time it takes to obtain images shot according to the changed settings.

[0077] In S408, CPU401 generates commands to change the shooting settings for each of the remaining (second and subsequent) cameras. Then, CPU401 sends the generated commands to each camera simultaneously, or in the order determined in S401 (or in the order of shortest predicted time as described above), via network I / F404. After that, CPU401 finishes the batch setting process. For cameras other than those that received commands in S405, it is not necessary to determine whether the setting changes have been completed.

[0078] When the batch change method is set to "cooperative," the system waits to send commands to the remaining cameras until it is determined that the shooting settings have been changed on at least one of the multiple cameras whose settings are being changed in a batch. Therefore, there is no period during which all of the multiple cameras whose settings are being changed in a batch are in the process of changing their settings. Once the shooting settings have been changed on at least one camera, even if there is a period during which all of the remaining cameras are in the process of changing their settings, it will not be impossible to acquire footage shot with the changed settings.

[0079] In this example, the default setting for the sub-area 510 where the batch change method is configured is "Normal," and if the "Cooperative" method is not explicitly set, the batch change process will be executed using the "Normal" method. However, conversely, the default setting for the sub-area 510 may also be "Cooperative."

[0080] If the number of cameras whose shooting settings are to be changed in bulk is three or more, the number of cameras to confirm the completion of the shooting change may be two or more. In this case, CPU 401 executes processes S405 to S407 for each camera whose shooting change is to be confirmed, and then executes S408. Increasing the number of cameras to confirm the completion of the shooting change will increase the time required for the bulk change of shooting settings, so the number can be set so that the sum of the estimated times mentioned above does not exceed the threshold. In addition, to secure images from different shooting directions with the changed settings, it may be the case to use two cameras, the first and second.

[0081] Additionally, for cameras whose shooting settings are changed in bulk, the number of cameras whose shooting settings are checked for completion should be equal to (or approximately equal to) the number of cameras whose settings are not checked. For example, if there are six cameras whose shooting settings are changed in bulk, the completion of the shooting settings can be checked for half of them (3 cameras), while the completion of the shooting settings can not be checked for the remaining 3 cameras. This eliminates the imbalance between the number of cameras whose settings are changed and which capture video of the content and quality desired by the user, and the number of cameras whose priority is to reduce the time required for setting changes.

[0082] Figure 5 is a timing chart for changing the shooting settings of cameras 100 and 200 simultaneously using the flowchart shown in Figure 4. Figure 5(a) is the timing chart for the "normal" simultaneous change method, and Figure 5(b) is the timing chart for the "coordinated" simultaneous change method. Common reference numbers are used for times common to both methods.

[0083] Here, we assume that CPU 401 started the batch change operation at time 600. As mentioned above, the batch change operation may be started in response to the detection of operation of the preset button 508, or in response to CPU 401 detecting that predetermined conditions other than user operation have been met, such as the elapsed time since the start of shooting.

[0084] At time 601, a command to change the shooting settings is sent from controller 400 to camera 100. This corresponds to the first execution of S403 or the execution of S405 in Figure 4. Here, if the batch change method is "normal", the commands are sent in the order determined in S401. Upon receiving the command, camera 100 begins changing the shooting settings according to the command.

[0085] At time 602 in Figure 5(a), a command to change the shooting settings is sent from controller 400 to camera 200. This corresponds to the second command transmission at S403 in Figure 4. Upon receiving the command, camera 200 begins changing the shooting settings according to the command. The interval between times 601 and 602 is assumed to be predetermined.

[0086] In Figure 5(a), assume that the change in the shooting settings of camera 200 is complete at time 603. At time 603, the setting change is not yet complete for camera 100. This is because the time required to change the shooting settings varies depending on the item being changed and the amount of change. For example, changing an item that involves the operation of the drive unit 109 takes longer than changing an item that does not involve the operation of the drive unit 109. Also, when changing an item that involves the operation of the drive unit 109, the larger the amount of change, the longer the time required to change the shooting settings.

[0087] At time 604, the change in the shooting settings for camera 100 is assumed to be complete. If the batch change method is "normal", the change in the shooting settings for cameras 100 and 200 is completed at time 604. On the other hand, if the batch change method is "cooperative", since camera 100 is the first camera, no command has been sent to camera 200 at time 604. When CPU 401 executes S407 at or after time 604, it is determined that the change in shooting settings is complete, and S408 is executed, sending a command to camera 200 to change the shooting settings.

[0088] At time 605 in Figure 5(b), the change in the shooting settings of camera 200 is completed.

[0089] As described above, when the batch change method is set to "normal," the shooting settings for cameras 100 and 200 are changed at time 604, allowing all cameras to change their shooting settings in a shorter time compared to when the batch change method is set to "coordinated."

[0090] On the other hand, if the batch change method is set to "normal," the period from time 602 to time 603 is a time when both camera 100 and camera 200 are changing their settings. Therefore, the video obtained from cameras 100 and 200 during the period from time 602 to time 603 may not be of the content or quality desired by the user.

[0091] If the batch change method is set to "coordinated," the period between time 601 and 604 when the shooting settings are changed on camera 100 and the period between time 604 and 605 when the shooting settings are changed on camera 200 will not overlap. Therefore, it is guaranteed that the user will be able to obtain video of the desired content and quality from at least one of the cameras during all periods.

[0092] Furthermore, as explained in the example of the S407 judgment process in Figure 4, if the change in the shooting settings of camera 200 is started without confirming the change in the shooting settings of camera 100, the time chart in Figure 5(b) may look like Figure 6, for example. In Figure 6, common reference numbers are used for times that are common to both Figure 5(b) and Figure 6.

[0093] In the time chart shown in Figure 6, the controller 400 sends a command to camera 200 to change its shooting settings at time 706, which is earlier than time 604, when the change to camera 100's shooting settings is actually completed. As a result, the period from time 706 to time 604 is the time when both camera 100 and camera 200 are changing their settings. However, this period is considerably shorter than when the batch change method is "normal".

[0094] In the flowchart in Figure 4, it is explained that the controller 400 determines whether or not the change in shooting settings based on the transmitted command has been completed. However, the camera that received the command may determine that the change in shooting settings based on the command has been completed and notify the controller 400. In this case, the CPU 401 does not need to execute S406 in Figure 4. Then, in S407, the CPU 401 only needs to determine whether or not it has received notification from the camera that sent the command to change the shooting settings in S405, via the network I / F 404, that the setting change has been completed.

[0095] The specific operation of the controller 400 and camera 100 will be explained using the flowchart shown in Figure 7. In Figure 7(a), steps that perform the same processing as in Figure 4 are given the same reference numbers as in Figure 4, and the explanation is omitted.

[0096] Figure 7(a) is a flowchart showing the operation of controller 400 (CPU 401), and Figure 7(b) is a flowchart showing the operation of camera 100 (CPU 101). This explanation describes the case when camera 100 receives a command, but CPU 201 performs similar processing when camera 200 receives a command.

[0097] The operation of controller 400 when the batch change method is set to "normal" is the same as in Figure 4, so the explanation is omitted. If the batch change method is "cooperative," after sending a command to camera 100 in S405, CPU 401 determines in S801 whether or not it has received a "shooting setting change completion notification" from camera 100 via network I / F 404, as described later. If CPU 401 determines that it has received the change completion notification, it executes S408; otherwise, it repeatedly executes S801. As a result, controller 400 does not send a command to change the shooting settings to camera 200 until the shooting settings change is complete in camera 100.

[0098] In step S811 of Figure 7(b), the CPU 101 of the camera 100 starts changing the shooting settings based on a command received via the network interface 105. For example, if the received command specifies target pan and tilt angles, the CPU 101 generates a control command based on the difference between these target angles and the current pan and tilt angles. The CPU 101 then transmits the generated control command to the drive unit 109 via the drive interface 108. For changes to items that do not involve the operation of the drive unit 109, the CPU 101 rewrites the setting values ​​stored in the ROM 103 to the specified values, for example.

[0099] In S812, the CPU 101 of the camera 100 obtains the current values ​​of the shooting settings items that were changed in S811 from the drive unit 109 and ROM 103. Then, in S813, the CPU 101 determines whether the changes to the shooting settings are complete. For example, the CPU 101 determines whether the current pan angle and tilt angle obtained from the drive unit 109 in S812 have reached the target pan angle and tilt angle specified in the command. If the changes involve multiple items, the CPU 101 determines whether the changes to all items are complete. If the CPU 101 determines that all the changes to the shooting settings instructed by the received command are complete, it executes S814; otherwise, it repeatedly executes S812 and S813.

[0100] In S814, the CPU 101 of the camera 100 sends a "shooting setting change completion notification" to the command source (controller 400) via the network I / F 105. When this change completion notification is detected, the CPU 401 determines in S801 of Figure 7(a) that the shooting setting change for the camera 100 has been completed.

[0101] Thus, the completion of the shooting setting change can be determined by the controller 400 obtaining the current shooting setting value from the camera, or by the camera that received the command determining that the shooting setting change based on the command has been completed and notifying the controller 400.

[0102] According to the first embodiment, in a multi-camera system, the shooting settings of multiple cameras can be changed simultaneously, significantly reducing the user's effort, especially when there are many cameras. Furthermore, since the simultaneous change can be performed in a manner that guarantees at least one camera is shooting video according to the changed settings, it is possible to prevent interruptions in the desired video content and quality even while the simultaneous change is being performed.

[0103] (modified version) Cameras 100 and 200 may be configured, for example, with remotely controllable cameras, including zoom capabilities, mounted on motorized pan / tilt heads that allow for remote control of pan and tilt angles. In this case, the CPU 401 of the controller 400 sends commands to control the shooting direction to the motorized pan / tilt head and commands to control the zoom value to the camera. The motorized pan / tilt head has a configuration corresponding to the CPU, the camera 100's drive I / F 108, drive unit 109, and network I / F 105, and controls the pan and tilt angles of the pan / tilt head according to commands received from the controller 400. The CPU also transmits the current pan and tilt angles in response to commands from the controller 400, and notifies the controller 400 of the completion of operations corresponding to the commands.

[0104] ●<Second Embodiment> Next, a second embodiment of the present invention will be described. The multi-camera system according to the second embodiment includes a camera operated by the user and a camera that automatically tracks and photographs a subject. In this embodiment, the controller 400, which is a shooting control device, can change the shooting settings of multiple cameras that automatically track and photograph according to the state of the camera operated by the user.

[0105] (Overview of the multi-camera system) Figure 8 shows the overall configuration of the multi-camera system 1000 according to this embodiment. Components identical to those in the first embodiment are denoted by the same reference numerals as in Figure 1, and their descriptions are omitted. The multi-camera system 1000 includes a plurality of cameras 100, 200, 1100, and 1200, and a controller 1400.

[0106] Cameras 100, 200, 1100, and 1200 are imaging devices for photographing the subject 20, and controller 1400 is a shooting control device that remotely controls the operation of cameras 100, 200, 1100, and 1200. Each of the cameras 100, 200, 1100, and 1200 and controller 1400 are configured to communicate with each other via network 300. Network 300 may be part of the multi-camera system 10 or it may be an external network. Alternatively, each of the cameras 100, 200, 1100, and 1200 and controller 1400 may be directly connected. In this case, network 300 is not necessary.

[0107] Camera 1100 captures the entire predetermined shooting range. Camera 1100's shooting direction and field of view are set to capture the area where, for example, a subject 20 might be present in a studio. The shooting direction and field of view of camera 1100 are basically fixed. Therefore, the image from camera 1100 captures all subjects 20 that are within the shooting range. To distinguish it from the other cameras 100, 200, and 1200, whose shooting direction and field of view are not basically fixed during shooting, camera 1100 will be referred to as the overhead camera 1100 below for convenience.

[0108] Camera 1200 is a camera equipped with a mechanism that allows for pan, tilt, and zoom (PTZ) control to change the shooting direction and shooting angle of view, for example. Here, the operation of Camera 1200 is controlled by the user. Alternatively, Camera 1200 may be configured to allow control of the shooting direction (pan and tilt) by mounting the camera body on an electric pan / tilt head. Hereafter, Camera 1200 will be referred to as User-Operated Camera 1200.

[0109] The controller 1400 remotely controls cameras 100 and 200, the overhead camera 1100, and the user-operated camera 1200. In this embodiment, the user operates the user-operated camera 1200 using the controller 1400. However, the user-operated camera 1200 may be operated using a separate dedicated controller from the controller 1400. The controller 1400 also detects a subject from the video signal received from the overhead camera 1100 and, based on the detection result, automatically controls the shooting direction and field of view so that cameras 100 and 200 automatically track and photograph a specific subject.

[0110] (Structure description) Figure 9 is a block diagram showing an example configuration of cameras 100, 200, 1100, 1200, and controller 1400. Components identical to those in the first embodiment are given the same reference numerals as in Figure 2, and their descriptions are omitted. Figure 9 shows only the components necessary to explain the following operations.

[0111] The overhead camera 1100 includes a CPU 1101, RAM 1102, ROM 1103, network interface 1105, image processing unit 1106, image sensor 1107, video output interface 1110, and system bus 1111. The functions of each part are the same as those of the corresponding components in cameras 100 and 200. The overhead camera 1100 may also have a drive interface and drive unit, like cameras 100 and 200.

[0112] The user-operated camera 1200 includes a CPU 1201, RAM 1202, ROM 1203, network interface 1205, image processing unit 1206, image sensor 1207, drive interface 1208, drive unit 1209, video output interface 1210, and system bus 1211. The functions of each part are the same as those of the corresponding components in cameras 100 and 200.

[0113] Controller 1400 includes a CPU 1401, RAM 1402, ROM 1403, network interface 1404, display unit 1405, user input interface 1406, and system bus 1407. The configuration of each part is the same as that of the components of the same name in controller 400. Furthermore, controller 1400 includes an inference unit 1408.

[0114] The inference unit 1408 performs object region detection processing on the video from the overhead camera 1100 using a trained machine learning model. The inference unit 1408 can be implemented using hardware circuits capable of high-speed execution of machine learning model calculations, for example. Such hardware circuits include GPUs (Graphics Processing Units) and NPUs (Neural Network Processing Units). Alternatively, the inference unit 1408 may be implemented using reconfigurable logic circuits such as FPGAs (Field-Programmable Gate Arrays). Furthermore, the CPU 1401 may implement the functions of the inference unit 1408 by executing a program.

[0115] The machine learning model may be a convolutional neural network (CNN) trained according to the type of subject to be detected. Here, the inference unit 1408 detects human body regions or human face regions as subject regions from the input image. The inference unit 1408 also outputs the position and size of the rectangular region inscribed in the subject region, as well as the detection confidence level, for each detected subject region. Multiple machine learning models may be used to perform detection processing of different types of subject regions on the same input image. The inference unit 1408 may also perform subject region detection processing using known methods that do not use machine learning models. For example, the inference unit 1408 can detect subject regions using methods that utilize local features such as SIFT or SURF, or methods that utilize pattern matching.

[0116] (Operation description) Next, we will explain the operation of the multi-camera system 1000. In the multi-camera system 1000, the user (the photographer or the user of the controller 1400) controls the shooting direction and field of view of the user-operated camera 1200. On the other hand, the controller 1400 automatically controls the shooting direction and field of view of cameras 100 and 200. Here, the controller 1400 controls the shooting direction and field of view of cameras 100 and 200 so that they track and shoot a specific subject. In addition, the settings for how the shooting direction and field of view are automatically controlled for cameras 100 and 200 (hereinafter referred to as "roles") are pre-set for each camera. The controller 1400 determines the state of the user-operated camera 1200 and automatically controls the shooting direction and field of view of cameras 100 and 200 according to the state of the user-operated camera 1200 and the roles set for each camera 100 and 200.

[0117] The operation of each device is described below. Figures 10(a) to 10(d) are flowcharts showing the operation of the controller 1400, the overhead camera 1100, the user-operated camera 1200, and cameras 100 and 200, respectively.

[0118] In the following description, it is assumed that the 3D coordinate values ​​of the viewpoint position and the shooting direction (optical axis direction) of the overhead camera 1100 are known to the controller 1400. Furthermore, it is assumed that known positional information, such as the 3D coordinate values ​​of the viewpoint positions of cameras 100 and 200, and the user-operated camera 1200, and the coordinate values ​​of markers placed within the shooting range, are pre-stored in ROM 403 as default positional information REF_POSI.

[0119] (Operation of Controller 1400) In S1001, the CPU 1401 of the controller 1400 sends a shooting command to the overhead camera 1100 via the network I / F 1404 using a predetermined protocol. In response to this command, the overhead camera 1100 starts supplying a video signal (video data) IMG via the network I / F 1404. After the CPU 1401 starts storing the received video signal in the RAM 102, it executes S1002.

[0120] In step S1002, the CPU 1401 acquires information ANGLE indicating the shooting direction from the user-operated camera 1200. Specifically, the CPU 1401 sends a shooting direction acquisition command to the user-operated camera 1200 via the network I / F 1404 using a predetermined protocol. In response to the shooting direction acquisition command, the CPU 1201 of the user-operated camera 1200 sends information ANGLE indicating the current shooting direction of the user-operated camera 1200 to the controller 1400. Information ANGLE may be, for example, the pan and tilt angles of the drive unit 1209. The CPU 1401 stores the acquired information ANGLE in the RAM 1402.

[0121] In S1003, CPU 1401 uses inference unit 1408 to perform the following processing. (1) Apply subject area detection processing to the input frame image and store the detection result. (2) For each detected subject area, the position information (image coordinates) is transformed. (3) Apply identification processing to each detected subject area to identify identification information (add information for identification processing in the case of a new subject). (4) Store the identification information ID[n] and location information POSITION[n] associated with each detected subject area.

[0122] The following explains the process of S1003 step by step. (1) First, the CPU 1401 reads one frame of video received from the overhead camera 1100 from the RAM 1402 and inputs it to the inference unit 1408. Next, the inference unit 1408 inputs the frame image to a machine learning model and detects the subject area. The inference unit 1408 stores the position and size of each detected subject area, and the detection confidence level, which are output by the machine learning model as detection results, in the RAM 1402. The position and size of the subject area may be any information that can identify the position and size of the rectangular area inscribed in the subject area. Here, the coordinates of the center of the bottom edge of the rectangular area, as well as its width and height, are used as the position and size of the subject area.

[0123] Furthermore, the inference unit 1408 stores the detection result for the first frame image in RAM 1402, associated with the subject identification information ID[n]. Here, n is the subject number, an integer ranging from 1 to the total number of detected subject regions. In addition, the inference unit 1408 stores the subject regions detected from the first frame image in RAM 1402, associated with the subject identification information ID[n], as a template for identifying individual subjects. If template matching is not used for subject identification, it is not necessary to store the template.

[0124] Figure 11(a) shows an example of the results of subject detection processing by the inference unit 1408 on the image from the overhead camera 1100. Here, the areas of human subjects A to C that exist within the shooting range 2000 are detected, and the coordinates of the center of the bottom edge of the rectangular area inscribed with the subject area (foot coordinates) are output as the position.

[0125] Furthermore, for coordinate transformations described later, if markers are placed at known positions within the shooting range 2000, as shown in Figure 11(b), the CPU 1401 detects the marker images included in the frame image (Figure 11(a)) and stores their positions in the RAM 102. The detection of marker images may also be configured to be performed by the inference unit 1408. Marker image detection can be performed by any known method, such as pattern matching using a marker template. Marker images may also be detected using a pre-stored machine learning model for marker detection.

[0126] (2) Next, the coordinate transformation performed by the inference unit 1408 will be explained. Figure 11(a) schematically shows the image from the overhead camera 1100, and Figure 11(b) schematically shows the shooting range 2000 as viewed from directly above its center. The inference unit 1408 transforms the position of the subject area in the coordinate system of the overhead camera to the coordinate system (plane coordinate system) when the shooting range 2000 is viewed from directly above its center.

[0127] The reason for transforming the coordinates to a planar coordinate system here is that it is convenient for calculating the pan value (the change in movement angle or pan angle in the horizontal plane) required for cameras 100 and 200 to photograph a specific subject. It is assumed here that cameras 100 and 200 are positioned so that the drive units 109 and 209 perform panning movements within a horizontal plane parallel to the floor of the shooting range 2000.

[0128] Coordinate transformation can be performed in various ways, but here, markers are placed at multiple known positions on the floor within the shooting range of 2000, and the coordinates are transformed from the overhead camera coordinate system to the planar coordinate system based on the marker positions in the image obtained by the overhead camera 1100. Alternatively, the coordinate transformation may be performed without using markers, such as by using the viewpoint position and shooting direction of the overhead camera 1100.

[0129] The coordinate transformation can be performed using the homography transformation matrix H, according to equation 5 below.

number

[0130] The homography transformation matrix can be calculated by substituting the coordinates of the four markers detected from the video and the coordinates (known) of the four markers placed in the shooting range 2000 into Equation 5 and solving the system of equations. If the positional relationship between the shooting range 2000 and the overhead camera 1100 is fixed, the homography transformation matrix H can be calculated in advance during test shooting and saved, for example, in ROM 1403.

[0131] The CPU 1401 sequentially reads the position of the subject area from the RAM 1402 and transforms the coordinates to values ​​in a planar coordinate system. Figure 12(b) schematically shows the state after the foot coordinates (x, y) of each subject area detected in the overhead camera 1100 video shown in Figure 12(a) have been transformed to coordinate values ​​(X, Y) in a planar coordinate system using Equation 1 and the homography transformation matrix H stored in ROM 103. The CPU 1401 stores the transformed foot coordinates as POSITION[n] in the RAM 1402.

[0132] (3) Next, the operation of the inference unit 1408 in identifying the subject identification information ID[n] will be described. Here, the subject will be identified using template matching. Subject identification will be performed on the processing results of the second and subsequent subject detections. For the first processing result, a new identification information ID[n] will be assigned to the subject region.

[0133] The inference unit 1408 identifies the identification information ID[n] of the detected subject area by template matching using templates stored in RAM 1402. This identifies the subject within the shooting range. For example, the inference unit 1408 calculates an evaluation value representing the correlation of individual templates for each detected subject area. Then, the inference unit 1408 identifies the identification information ID[n] corresponding to the template with the highest correlation that has a certain level of correlation as the identification information ID[n] of the subject area. The evaluation value can be a known value, such as the sum of the absolute differences of pixel values.

[0134] Furthermore, the inference unit 1408 assigns a new identification information ID[n] to any subject region that does not have a certain level of correlation with all templates, and adds the image of the subject region to the template.

[0135] Furthermore, the inference unit 1408 may update existing templates using subject regions detected in the most recent frame image, or delete templates for which no subject regions with a certain level of correlation exist for a certain period of time. In addition, the inference unit 1408 may store templates corresponding to frequently appearing identification information ID[n] in the ROM 103.

[0136] Furthermore, subjects may be identified by methods other than template matching. For example, at least one of the previously detected position and size may be identified as having the same identification information ID [n] as the nearest subject region. Alternatively, the position in the current frame image may be predicted using a Kalman filter or the like based on the position changes in multiple past detection results associated with the same identification information, and the same identification information ID may be identified as the subject region closest to the predicted position. These methods may also be combined. By not using template matching, the accuracy of identifying different subjects that look similar can be improved.

[0137] (4) The inference unit 1408 associates the identified identification information ID[n] with the corresponding position (plane coordinate system) POSITION[n] of the subject area and stores it in the RAM 1402.

[0138] Note that, of the processes (1) to (4), the CPU 1401 may execute the processes other than subject detection instead of the inference unit 1408.

[0139] Here, the image from the overhead camera 1100 was used to determine the identification information ID[n] and position[n] of a subject within the shooting range 2000. However, images from cameras 100 and 200 may also be used. In this case, the CPU 1401 performs the operations shown in the flowchart of Figure 10(a) for cameras 100 and 200, respectively. The position of the subject area is output as a value in the coordinate system of cameras 100 and 200. Thus, although the overhead camera 1100 is not essential, it is considered that using the overhead camera 1100 results in better subject detection accuracy.

[0140] Returning to the explanation of Figure 6(a), in S1004, the CPU 1401 determines the subject of interest to be tracked by the user-operated camera 1200. The CPU 1401 can determine the subject of interest of the user-operated camera 1200 from among the subjects detected in S1003, based on the shooting direction of the user-operated camera 1200 acquired in S1002. The CPU 1401 stores the identification information ID[n] corresponding to the subject area determined to be the subject of interest of the user-operated camera 1200 in the RAM 1402 as the identification information MAIN_SUBJECT of the subject of interest.

[0141] For example, the CPU 1401 can determine the subject closest to the shooting direction of the user-operated camera 1200 in a planar coordinate system as the subject of interest of the user-operated camera 1200. If there are multiple subjects whose distance from the shooting direction of the user-operated camera 1200 is below a threshold, the user may be allowed to select the subject of interest from among them.

[0142] When prompting the user to select a subject of interest, the CPU 1401 displays the frame image to which the subject detection processing in S1003 has been applied on the display unit 1405 or an external display device, along with an indicator showing the shooting direction and an indicator showing the subject area that is a candidate for the subject of interest. The subject area indicator may be a rectangular frame indicating the outer edge of the subject area, as shown in Figure 12(a), but other indicators may also be used. The CPU 1401 may also display messages on the display unit 1405 prompting the user to select a subject of interest in the image.

[0143] The user can select a subject area corresponding to a desired subject of interest by operating the user input interface 1406 (input device). There are no particular restrictions on the selection method, but it may be an operation to specify the desired subject area using a mouse or keyboard.

[0144] When CPU1401 detects a user operation specifying a subject area, it stores the identification information ID[n] corresponding to the specified subject area in RAM1402 as the identification information MAIN_SUBJECT of the subject of interest.

[0145] Next, in S1005, the CPU 1401 obtains the role information (role setting information) corresponding to cameras 100 and 200. The role setting information is information that associates the identification information of cameras 100 and 200 with information indicating the role to which they are set.

[0146] Figure 13 shows examples of the types of roles that can be set for cameras 100 and 200, and the control content associated with those roles. The control content for each role can be stored in ROM 1403 in a table format, for example, as shown in Figure 13.

[0147] Here, the role can be set to one of the following: "Main Follow," "Main Counter," "Assist Follow," or "Assist Counter." Note that different roles can be assigned to cameras 100 and 200.

[0148] For cameras whose role is "main follow," the controller 1400 (CPU 1401) sets the same tracking subject as the user-operated camera 1200, and performs in-phase zoom control when the user-operated camera 1200 is zoomed.

[0149] Here, "in phase" means that the zoom direction (telephoto or wide-angle) is the same, that is, the direction of the angle of view change is the same. On the other hand, "out of phase" means that the zoom direction (telephoto or wide-angle) is in the opposite direction, that is, the direction of the angle of view change is in the opposite direction. Note that even if the zoom direction is in phase, the angle of view does not have to be the same as that of the user-operated camera 1200, and in either case, the degree of zoom change (such as the speed of change or the rate of change) does not have to be the same as that of the user-operated camera 1200.

[0150] For cameras whose role is "main counter," the controller 1400 (CPU 1401) sets the same tracking subject as the user-operated camera 1200, and when the user-operated camera 1200 is zoomed, it performs zoom control in the opposite phase.

[0151] For cameras whose role is "Assist Follow," the controller 1400 (CPU 1401) sets a separate tracking subject from the user-operated camera 1200, and performs in-phase zoom control when the user-operated camera 1200 is zoomed.

[0152] For cameras whose role is "assist counter," the controller 1400 (CPU 1401) sets a different tracking subject from the user-operated camera 1200. In addition, when the user-operated camera 1200 is zoomed, the controller 1400 performs a zoom control in the opposite phase.

[0153] Here, for cameras with the role set to "Assist Follow" and "Assist Counter," the subject located to the left of the image, which is separate from the subject of interest of user-operated camera 1200, is set as the camera's tracking subject. Note that the camera's tracking subject may be set according to other conditions. For example, the camera 100 or 200 may set the camera's tracking subject to a subject located to the right, above, or below the image, which is separate from the subject of interest of user-operated camera 1200. Alternatively, the camera may set the camera's tracking subject to the subject located closest to or furthest from the subject of interest of user-operated camera 1200.

[0154] In the multi-camera system 1000 of this embodiment, when the subject of interest of the user-operated camera 1200 changes, the CPU 1401 determines a new subject to be tracked according to the roles set for cameras 100 and 200. Then, the shooting settings of cameras 100 and 200 are changed collectively in order to track the determined subject.

[0155] As described above, the roles assigned to cameras 100 and 200 define how to automatically control the subject being tracked and the size of the tracked subject in the video (frame image). However, the content defined by the roles is not limited to these. For example, it may include composition settings that specify the position of the tracked subject that should be maintained in the video. It may also include settings related to the responsiveness of the shooting direction and zoom value during tracking (the sensitivity of the control of cameras 100 and 200 to changes in the shooting direction and zoom of the user-operated camera 1200). Including these settings allows the user to adjust the tracking shooting settings in more detail. When these settings are included, settings related to composition, and settings related to the drive speed and acceleration of the pan angle, tilt angle, and zoom value during tracking shooting can be added to the table shown in Figure 13.

[0156] In S1007, the CPU 1401 determines the subject that cameras 100 and 200 should track and photograph, based on the subject of interest of the user-operated camera 1200 determined in S1004 and the roles set for cameras 100 and 200.

[0157] For example, for cameras with the roles "Main Follow" and "Main Counter" set, the CPU 1401 determines that the subject of interest of the user-operated camera 1200 will be the subject to be tracked. Therefore, the CPU 1401 sets the identification information MAIN_SUBJECT of the subject of interest determined in S1003 as the identification information SUBJECT_ID of the subject to be tracked for cameras with the roles "Main Follow" and "Main Counter" set.

[0158] Furthermore, for cameras with the roles "Assist Follow" and "Assist Counter" set, the CPU 1401 determines that the subject located to the left of the subject of interest of the user-operated camera 1200 will be the subject to be tracked. In this case, the CPU 1401 detects a subject area located at the left edge of the subject area other than the subject of interest of the user-operated camera 1200 from the subject area detected in S1003. The CPU 1401 then sets the identification information ID[n] corresponding to the detected subject area as the identification information SUBJECT_ID of the tracked subject for cameras with the roles "Assist Follow" and "Assist Counter" set.

[0159] The CPU 1401 writes the identification information SUBJECT_ID of the determined tracked subject to RAM 1402. If the tracked subjects of cameras 100 and 200 may be different, the CPU 1401 stores the identification information SUBJECT_ID of the tracked subject in association with the identification information of cameras 100 and 200. If the tracked subject changes, the CPU 101 retains the information of the previous tracked subject in RAM 1402 without erasing it.

[0160] Here, we will explain the operation when the role set for camera 100 is "main follow," using Figure 14. When the role "main follow" is set for camera 100, the controller 1400 controls it to track the subject of interest of the user-operated camera 1200.

[0161] Therefore, as shown in Figure 14(a), if the subject of interest of the user-operated camera 1200 is determined to be subject B, the CPU 1401 decides that subject B will be the subject tracked by camera 100. Subsequently, as shown in Figure 14(b), if the subject of interest of the user-operated camera 1200 is determined to have changed to subject A, the CPU 1401 changes the subject tracked by camera 100 to subject A. Similarly, as shown in Figure 14(c), if the subject of interest of the user-operated camera 1200 is determined to have changed to subject C, the CPU 1401 changes the subject tracked by camera 100 to subject C.

[0162] Next, in S1008, the CPU 1401 calculates the amount of change in pan and tilt angles required for cameras 100 and 200 to track and photograph the subject determined in S1007. The CPU 1401 also calculates the zoom values ​​for cameras 100 and 200 in accordance with the change in the field of view of the user-operated camera 1200. The calculation method for camera 100 is explained below, but the same calculation method is applied to camera 200.

[0163] Here, it is assumed that the following information is pre-stored in ROM 1403 as default position information REF_POSI for each camera 100 and 200. • 3D coordinates of the installation location (values ​​in a planar coordinate system) • Shooting direction corresponding to the initial values ​​of the pan and tilt angles of the drive unit • Controllable range of pan and tilt angles

[0164] The CPU 1401 reads the position information POSITION_OH, which corresponds to the identification information SUBJECT_ID of the subject being tracked by the camera 100, from the RAM 1402. Then, the CPU 1401 first determines the pan angle based on the position information POSITION_OH and the installation position of the camera 100.

[0165] Figure 16 shows an example of the positional relationship between camera 100 and the tracked subject in a planar coordinate system. Here, we will determine the pan angle θ so that the optical axis of camera 100 is directed towards the subject position. CPU 1401 calculates the pan angle θ using the following equation 6.

number

[0166] In Equation 6, px and py are the horizontal and vertical coordinates of the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracked subject. Also, subx and suby are the horizontal and vertical coordinates of the installation position of camera 100. Here, we assume that the current pan angle is initially 0° and the optical axis direction is vertical (Y axis direction). If the current optical axis direction is not vertical, the angle obtained in Equation 2 should be adjusted to reflect the angle difference between the current optical axis direction and the vertical direction. Also, the pan direction is counterclockwise if subx > px, subx <pxであれ It is clockwise.

[0167] Next, we will explain how to determine the tilt angle using Figure 17. Figure 17 shows the camera 100 and the subject being tracked viewed from the side. Assume that the current optical axis of camera 100 is horizontal and its height is h1, and the height of the subject's face to which the optical axis is pointed is h2. Let ρ be the angle difference in the height direction between the current optical axis direction and the target optical axis direction (tilt angle). The CPU 1401 calculates the tilt angle ρ using the following equations 7 and 8.

number

[0168] The coordinate values used in Equation 8 are the same as those used in Equation 7. Assume that h1 and h2 are input to the shooting control application in advance and stored in the RAM 1402. In this case, make the identification number associated with h2 for each subject equal to the identification number assigned in the subject detection process. Alternatively, h2 may be a value measured in real time using a sensor (not shown).

[0169] Here, assume that the current tilt angle is the initial value of 0°, and the optical axis direction is the horizontal direction (constant height). If the current optical axis direction is not the horizontal direction, the angle difference between the current optical axis direction and the horizontal direction may be reflected in the angle obtained by Equation 8. Also, the tilt direction is downward if h1 > h2, and upward if h1 < h2.

[0170] The CPU 1401 communicates with the camera 100 periodically, acquires the current optical axis direction (the pan angle and tilt angle of the drive unit), and stores it in the RAM 1402. Note that the communication cycle can be, for example, less than or equal to the reciprocal of the frame rate. Alternatively, the CPU 1401 may hold the total value of the pan angle and tilt angle controlled from the initial state for the camera 100 in the RAM 1402 and use it as the current optical axis direction.

[0171] The CPU 1401 calculates the change amounts of the pan angle and tilt angle of the camera 100 in this way and stores them in the RAM 1402.

[0172] The change amounts of the pan angle and tilt angle may be used as the angular velocities for the camera 100 to rotate in the direction of the tracking subject. For example, the CPU 1401 acquires the current pan angle and tilt angle from the camera 100. Then, the CPU 1401 obtains the pan angular velocity proportional to the difference between the pan angle θ read from the RAM 1402 and the current pan angle. Also, the CPU 1401 obtains the tilt angular velocity proportional to the difference between the tilt angle ρ read from the RAM 1402 and the current tilt angle. The CPU 1401 stores the angular velocities calculated in this way in the RAM 1402.

[0173] Alternatively, the change in pan angle and tilt angle may be calculated using the image from camera 100 instead of the image from the overhead camera 1100. In this case, the CPU 1401 may calculate the change in pan angle from the horizontal difference between the current optical axis direction and the direction of the tracked subject in the coordinate system of camera 100, and the change in tilt angle from the vertical difference. Furthermore, the imaging system may change the shooting direction for tracking the tracked subject by only changing either the pan direction or the tilt direction, and in such an imaging system, only the change in either the pan angle or the tilt angle may be calculated.

[0174] Next, we will explain how the CPU 1401 calculates the zoom value. The CPU 1401 periodically acquires information MAIN_ZOOM, which indicates the field of view of the user-operated camera 1200, and stores it in RAM 1402. When the information MAIN_ZOOM changes, the CPU 1401 calculates the zoom value Z_VALUE for the camera 100 according to the control content CAMERA_ROLE corresponding to the role set for the camera 100.

[0175] The CPU 1401 can determine the zoom operation of the user-operated camera 1200 and its phase, for example, by detecting changes in the field of view of the video from the user-operated camera 1200. For example, changes in the field of view may be detected from changes in the size or spacing of the subject area over time.

[0176] Figure 18 shows an example of mapping the zoom values ​​of the user-operated camera 1200 and camera 100. Here, it is assumed that the user-operated camera 1200 and camera 100 optically change the field of view (the imaging optical system has a zoom function). However, a similar function may be achieved with digital zoom using the image processing units 106 and 1206.

[0177] The zoom value is a parameter whose value corresponds to the angle of view. In this embodiment, the smaller (narrower) the angle of view, the smaller the zoom value, and the zoom value on the telephoto side is smaller than the zoom value on the wide-angle side. Camera 100 and user-operated camera 1200 can control the imaging optical system to the angle of view corresponding to the zoom value by sending a command that specifies the zoom value. In other words, the zoom value is information about the angle of view and represents the zoom state. The zoom value may also be, for example, the focal length (mm) of the imaging optical system corresponding to a 35mm full-frame image sensor, in which case the zoom value on the telephoto side will be larger than the zoom value on the wide-angle side.

[0178] In Figure 18, the zoom range of the user-operated camera 1200 is main_min to main_max. The zoom range of camera 100 is sub_min to sub_max. main_min and sub_min are the zoom values ​​corresponding to the telephoto ends of the user-operated camera 1200 and camera 100, respectively, while main_max and sub_max are the zoom values ​​corresponding to the wide-angle ends of the user-operated camera 1200 and camera 100, respectively. Figure 18 shows an example where the zoom range of the user-operated camera 1200 is wider than the zoom range of camera 100 at both the telephoto and wide-angle ends.

[0179] For example, when controlling the zoom value SUB_ZOOM of camera 100 to be in phase with the zoom value MAIN_ZOOM of user-operated camera 1200, the CPU 1401 calculates the SUB_ZOOM corresponding to the current MAIN_ZOOM using the following equation 9.

number

[0180] Returning to Figure 10(a), in S1009, the CPU 1401 reads the pan and tilt angle change amounts and the zoom value calculated in S1008 from the RAM 1402. The CPU 1401 then generates a control command PT_VALUE that instructs the camera 100 to change the pan and tilt angles corresponding to these change amounts. The CPU 1401 also generates a control command Z_VALUE that instructs the camera 100 to change the field of view corresponding to the zoom value. The format of the control commands is predetermined. The CPU 1401 stores the generated control commands PT_VALUE and Z_VALUE in the RAM 1402. Note that if there is no need to generate control commands, such as when the tracked subject is stationary or when the field of view of the user-operated camera 1200 does not change, S1009 may be skipped.

[0181] This section describes how to specify the amount of change from the current pan and tilt angles in commands that instruct the camera to change the pan and tilt angles. However, it is also possible to generate commands that specify target values ​​for the pan and tilt angles instead of the amount of change, allowing the camera to calculate the change.

[0182] In this embodiment, when changing the shooting settings for both cameras 100 and 200, the timing of sending the command in S1009 is adjusted so that the change period for camera 100 and the change period for camera 200 do not overlap. This timing adjustment operation will be explained with reference to Figure 15.

[0183] Figure 15 is a flowchart illustrating the details of S1009. The CPU 1401 performs the process shown in the flowchart of Figure 15 in parallel with the process shown in Figure 10(a).

[0184] In S1501, the CPU 1401 first determines the order in which the shooting settings of cameras 100 and 200 are changed. The order can be determined arbitrarily; for example, it may be the order set in advance by the user, or it may be the order in which the cameras established a connection with the controller 1400.

[0185] Alternatively, a camera assigned the role of tracking and shooting a subject other than the subject of focus of the user-operated camera 1200 may be placed before a camera assigned the role of tracking and shooting the same subject as the subject of focus of the user-operated camera 1200. This allows for prioritizing the shooting of subjects that have not yet been captured by the user-operated camera 1200.

[0186] Alternatively, the inference unit 1408 of the controller 1400 may be used to determine the order in which to change the settings of cameras that have not yet captured the subject to be tracked (hereinafter referred to as "cameras with lost subject"), prioritizing their settings. This allows cameras that are highly likely to be unable to perform the expected tracking shots to be returned to a state where the expected image can be obtained as soon as possible.

[0187] Here, we assume that CPU 1401 has decided, through some means, to change the shooting settings (roles) of camera 100 and then camera 200 in that order.

[0188] In S1502, the CPU 1401 reads the control commands PT_VALUE and Z_VALUE from RAM 1402 to change the shooting settings of camera 100 based on the determined role change order, and sends them to camera 100 via network I / F 1404. At this point, the CPU 1401 does not send any commands to change the shooting settings of camera 200.

[0189] In S1503, the CPU 1401 obtains the current shooting settings from the first camera 100.

[0190] In S1504, CPU1401 determines whether the changes to the shooting settings for camera 100 have been completed. If CPU1401 determines that the changes to the shooting settings have been completed, it executes S1505; otherwise, it executes S1503 again.

[0191] For example, the CPU 1401 extracts the pan angle, tilt angle, and zoom value from the current shooting settings of the camera 100 acquired in S1503. The CPU 1401 then compares these values ​​with the value specified in the control command sent in S1502 (here referred to as the target value), and if they match, it can determine that the change is complete. Here, as in the first embodiment, other determination methods can be employed, such as determining that the change is complete even if they do not match, as long as the difference is below a threshold.

[0192] Alternatively, the CPU 1401 may use the inference unit 1408 to determine whether a new subject to be tracked has been captured in the video from the camera 100, and if so, determine that the change in shooting settings has been completed.

[0193] In S1505, the CPU 1401 reads the control commands PT_VALUE and Z_VALUE from RAM 1402 for the second and subsequent cameras in the determined role change order (in this case, camera 200). Then, the CPU 1401 sends the control commands to camera 200 via network I / F 1404. If there are three or more cameras with assigned roles, in S1505, the CPU 1401 can send control commands to the cameras other than the first one in various ways, as described in the first embodiment. For example, the CPU 1401 may send control commands to all cameras simultaneously. Alternatively, the CPU 1401 may, for example, determine that the shooting settings change based on the control command has been completed for the second camera, similar to the first camera, and then send control commands to the remaining cameras at any time.

[0194] Thus, in this embodiment as well, the shooting settings can be changed all at once in a manner that guarantees that at least one camera among multiple cameras is capturing video according to the changed settings. Therefore, as in the first embodiment, it is possible to prevent interruptions in video of the desired content and quality even while the all-at-on-one change is being performed.

[0195] (Operation of overhead camera 1100) Next, the operation of the overhead camera 1100 will be explained with reference to Figure 10(b). The operation described below is achieved by the CPU 1101 executing a program.

[0196] When the overhead camera 1100 is powered on, the CPU 1101 initializes each functional block, and then the camera enters a shooting standby state. In the shooting standby state, the CPU 1101 may start video recording processing for live view display and output the display image data generated by the image processing unit 1106 to the controller 1400 via the network interface 1105.

[0197] In the shooting standby state, the CPU 1101 waits for control commands to be received via the network I / F 1105. When the CPU 1101 receives a control command, it executes an action corresponding to the control command. This section describes the operation when a shooting command is received as a control command from the controller 1400.

[0198] In S1101, the CPU 1101 receives a shooting command from the controller 1400 via the network interface 1105. The shooting command may specify shooting parameters such as frame rate and resolution. It may also include settings related to the processing to be applied by the image processing unit 1106.

[0199] In S1102, the CPU 1101 responds to the reception of a shooting command and starts video recording processing to be supplied to the controller 1400. This video recording processing captures higher quality video than the video recording processing for live view display. For example, at least one of the video resolution and the shooting frame rate is higher than that of the video for live view display. The image processing unit 1106 applies processing to the image based on the settings for the video to be supplied to the controller 1400. The image processing unit 1106 sequentially stores the generated video data in the RAM 1102.

[0200] In S1103, the CPU 1101 reads the video data from RAM 1102 and sends it to the controller 1400 via the network interface 1105. From then on, processing from shooting to supplying video data continues until a control command to stop shooting is received.

[0201] (Operation of user-operated camera 1200) Next, the operation of the user-operated camera 1200 will be explained with reference to Figure 10(c). The operations described below are realized by the CPU 1201 executing a program.

[0202] When the user-operated camera 1200 is powered on, the CPU 1201 initializes each functional block, and then starts video recording processing to supply to the controller 1400. The image processing unit 1206 applies processing to the analog image signal obtained from the image sensor 1207 based on the settings for video to be supplied to the controller 1400. The image processing unit 1206 sequentially stores the generated video data in the RAM 1202. The CPU 1201 reads the video data from the RAM 1202 and supplies it to the controller 1400 via the network interface 1205.

[0203] The CPU 1201 supplies video data to the controller 1400 while waiting for control commands to be received via the network interface 1205. When the CPU 1201 receives a control command, it executes an action corresponding to the command. This section describes the operation when a shooting direction acquisition command is received. Note that when the pan / tilt control command PT_VALUE or the zoom control command Z_VALUE is received, the CPU 1201 drives the drive unit 1209 according to the command.

[0204] In S1201, CPU1201 receives a shooting direction acquisition command via network I / F1205. CPU1201 stores the received shooting direction acquisition command in RAM1202.

[0205] In S1202, the CPU 1201, in response to receiving a command to acquire the shooting direction, acquires the current pan angle and tilt angle from the drive unit 1209 via the drive I / F 1208 and stores them in the RAM 1202.

[0206] In S1203, CPU1201 reads the current pan angle and tilt angle from RAM1202 and transmits the shooting direction information ANGLE to controller1400 via network I / F1205.

[0207] (Operation of cameras 100 and 200) The operation of cameras 100 and 200 will be explained with reference to Figure 10(d). While this explanation focuses on the operation of camera 100, the same operation is performed for camera 200.

[0208] The operations described below are realized by the CPU 101 executing a program. When the camera 100 is powered on, the CPU 101 initializes each functional block and then starts video recording processing to be supplied to the controller 1400. The image processing unit 106 applies processing to the analog image signal obtained from the image sensor 107 based on the settings for video to be supplied to the controller 1400. The image processing unit 106 sequentially stores the generated video data in the RAM 102. The CPU 101 reads the video data from the RAM 102 and supplies it to the controller 1400 via the network I / F 105.

[0209] The CPU 101 supplies video data to the controller 1400 while waiting for control commands to be received via the network interface 105. When the CPU 101 receives a control command, it executes an action corresponding to the control command. This section describes the operation when the controller 1400 receives the pan / tilt control command PT_VALUE and the zoom control command Z_VALUE.

[0210] In S1301, the CPU 101 receives at least one of the pan / tilt control command PT_VALUE and the zoom control command Z_VALUE from the controller 1400 via the network interface 105. The CPU 101 stores the received control commands in the RAM 102.

[0211] In S1302, the CPU 101 reads the control command stored in RAM 102 and the corresponding control variable, and stores them in RAM 102. Here, in the case of the pan / tilt control command PT_VALUE, the control direction is the direction of pan and / or tilt, and the control variable is the change amount or target angle. In the case of the zoom control command Z_VALUE, the control variable is the zoom value, and since the control direction can be determined from the zoom value, reading and storing the control direction is unnecessary.

[0212] In S1303, the CPU 101 generates drive parameters for the drive unit 109 based on the operating direction and operating amount read in S1302. The CPU 101 may, for example, obtain drive parameters corresponding to a combination of operating direction and operating amount using a table previously stored in the ROM 103. If the operating amount is given as a target value (target angle or zoom value), the CPU 101 obtains the drive parameters from the difference between that value and the current value.

[0213] In S1304, the CPU 101 controls the drive unit 109 via the drive interface 108 based on the drive parameters acquired in S1303. This causes the drive unit 109 to change the shooting direction of the camera 100 to the operating direction and angle specified by the pan / tilt control command PT_VALUE. The drive unit 109 also changes the field of view of the shooting optical system to the zoom value specified by the zoom control command Z_VALUE.

[0214] According to the second embodiment, the same effects as in the first embodiment can be achieved when changing the shooting settings for multiple cameras performing automatic tracking shooting in a multi-camera system including a camera that performs automatic tracking shooting all at once.

[0215] In this embodiment, a configuration was described in which the controller 1400 collectively changes the shooting direction and / or field of view of cameras 100 and 200 in response to a change in the state of the user-operated camera 1200. However, collectively changing settings or other shooting settings may be changed according to other conditions.

[0216] For example, the CPU 1401 may, upon detecting a user instruction via the user input interface 1406 to change the roles of cameras 100 and 200 collectively, execute a batch change of the shooting settings for cameras 100 and 200. Specifically, for example, in response to a menu operation in the shooting control application, the CPU 1401 displays a setting screen for setting or changing the role of each camera on the display unit 1405 or an external display device. The cameras displayed on the setting screen are those automatically controlled by the controller 1400.

[0217] The user can operate the user input interface 1406 (input device) to select a role to assign to each camera (in this case, cameras 100 and 200) and instruct the system to apply the selected roles all at once. By changing the role, the system can collectively change settings associated with that role, such as the subject to track, composition, and zoom control method.

[0218] There are no particular restrictions on how roles are selected in the settings screen. The configuration may involve using a mouse or keyboard to select the desired role from a dropdown list, or by operating buttons corresponding to the desired role.

[0219] When the CPU 1401 detects an operation that instructs the execution of a setting, such as the operation of an execute button included in the settings screen, it executes the process described using Figure 10(a) and sets the roles selected for each camera on the settings screen all at once. This makes it possible to achieve the same effect as in the first embodiment even when changing the roles of cameras 100 and 200 all at once.

[0220] Furthermore, in this embodiment, the controller 1400 determined the tracking subjects for cameras 100 and 200 according to the subject of interest of the user-operated camera 1200 and the roles set for cameras 100 and 200. However, the tracking subjects for cameras 100 and 200 may be determined by other methods.

[0221] For example, the CPU 1401 may execute a batch change of the tracking subjects for cameras 100 and 200 in response to detecting a user instruction to batch change the tracking subjects for cameras 100 and 200 via the user input I / F 1406. Specifically, for example, in response to a menu operation in the shooting control application, the CPU 1401 displays a setting screen for setting or changing the tracking subject for each camera on the display unit 1405 or an external display device. The cameras displayed on the setting screen are cameras that are automatically controlled by the controller 1400.

[0222] The user can operate the user input interface 1406 (input device) to select a subject to track for each camera (in this case, cameras 100 and 200) and instruct the system to set the selected roles collectively. There are no particular restrictions on how the subject to track is selected, but it can be the same as when the user is allowed to select a subject of interest.

[0223] In other words, the CPU 1401 displays an image on the display unit 1405 or an external display device with a rectangular frame superimposed on it, for example, as shown in Figure 12(a), indicating the outer edge of the subject area. The user can operate the user input I / F 1406 (input device) to select a subject area corresponding to the desired tracking subject for each camera. There are no particular restrictions on the selection method, but it may be an operation to specify the desired subject area using a mouse or keyboard.

[0224] When the CPU 1401 detects an operation that instructs the execution of a setting, such as the operation of an execution button included in the settings screen, it executes the process described using Figure 10(a) and sets the tracking subjects selected for each camera on the settings screen all at once. However, in S1006, instead of determining the tracking subjects based on the subject of interest of the user-operated camera 1200 or the roles set for cameras 100 and 200, it determines the tracking subjects set by the user for each of cameras 100 and 200. This makes it possible to achieve the same effect as in the first embodiment even when the tracking subjects of cameras 100 and 200 are changed all at once according to user settings.

[0225] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0226] The disclosure of this embodiment includes the following shooting control device, multi-camera system, shooting control method, and program. (Item 1) A shooting control device that is communicatively connected to multiple cameras and capable of controlling at least one of the shooting direction and field of view for each of the multiple cameras, Having control means, When the control means changes the settings of the multiple cameras all at once, Of the aforementioned multiple cameras, determine which camera will have its settings changed first. The camera described above is instructed to change its settings. In the first camera, it is determined whether or not the specified setting change has been completed. A shooting control device characterized in that, when it is determined that the setting change specified in the first camera has been completed, it instructs the other cameras among the plurality of cameras to change their settings. (Item 2) The shooting control device according to item 1, characterized in that the control means does not change the settings of the other cameras until it is determined that the instructed setting change has been completed for at least one of the plurality of cameras other than the first camera. (Item 3) The shooting control device according to item 1, characterized in that the control means determines that the change of the instructed setting has been completed when the current setting value obtained from the first camera matches the instructed changed setting value. (Item 4) The shooting control device according to item 1, characterized in that the control means determines that the change of the instructed setting has been completed if the difference between the current setting value obtained from the first camera and the instructed changed setting value is less than or equal to a threshold. (Item 5) The aforementioned instructions include instructions for the changed shooting direction, The shooting control device according to item 1, characterized in that the control means determines that the change of the instructed setting has been completed when the current angle of view of the first camera includes the changed shooting direction. (Item 6) The aforementioned change in settings involves mechanical action, The shooting control device according to item 1, characterized in that the control means determines that the instructed setting change has been completed when the amount of change in the setting value obtained based on the time series of the current setting value acquired from the first camera decreases. (Item 7) The shooting control device according to item 1, characterized in that the control means determines that the instructed setting change has been completed when a predetermined time has elapsed since instructing the first camera to change the setting. (Item 8) The imaging control device according to item 7, characterized in that the predetermined time when the change of the instructed setting involves mechanical drive is longer than the predetermined time when the change of the instructed setting does not involve mechanical drive. (Item 9) The shooting control device according to item 1, characterized in that the control means determines that the instructed setting change has been completed after the predicted time required for the setting change has elapsed since instructing the first camera to change the setting. (Item 10) The shooting control device according to item 9, characterized in that the control means determines the first camera based on the predicted time. (Item 11) The shooting control device according to any one of items 1 to 10, characterized in that the control means changes the settings of the plurality of cameras all at once in response to a user operation specifying one of the combinations of setting values ​​for each of the plurality of cameras stored in the shooting control device. (Item 12) The aforementioned shooting control device automatically controls at least one of the shooting direction and field of view for each of the plurality of cameras so as to fit a specific subject into the field of view. The shooting control device according to item 1, characterized in that the control means collectively changes at least one of the following for each of the plurality of cameras as a setting for the plurality of cameras: the specific subject, the shooting size for shooting the specific subject, and the composition for shooting the specific subject. (Item 13) A shooting control device according to any one of items 1 to 12, characterized in that it automatically controls at least one of the shooting direction and field of view of the plurality of cameras according to the state of a camera other than the plurality of cameras. (Item 14) The shooting control device according to item 13, characterized in that it automatically controls at least one of the shooting direction and field of view of the plurality of cameras according to the role set for each of the plurality of cameras and the state of the other camera. (Item 15) The shooting control device according to item 13 or 14, characterized in that the state of the other camera includes at least one of the shooting direction and the subject of interest. (Item 16) The shooting control device according to item 15, characterized in that the state of the other camera includes the shooting direction, and the shooting direction of the plurality of cameras is changed in accordance with the change in the shooting direction of the other camera. (Item 17) The shooting control device according to item 15, characterized in that the state of the other camera includes a subject of interest, and the shooting direction of the plurality of cameras is changed so that the plurality of cameras track and photograph the subject of interest or a specific subject different from the subject of interest. (Item 18) The shooting control device according to item 17, characterized in that the control means determines, among the plurality of cameras, the camera that is not able to track the specific subject as the first camera. (Item 19) The shooting control device according to item 15, characterized in that when the state of the other camera includes a subject of interest, and the subject of interest is changed, the control means changes the settings of the multiple cameras all at once. (Item 20) The shooting control device according to item 14, characterized in that the control means changes the role set for each of the plurality of cameras collectively as the settings for the plurality of cameras. (Item 21) The shooting control device according to item 12 or 17, characterized in that the control means determines that the change of the instructed setting has been completed when the current angle of view of the first camera includes the specific subject after the change. (Item 22) A camera control device as described in any one of items 1 to 21, Multiple cameras connected to the aforementioned shooting control device in a communicative manner, A multi-camera system characterized by having the following features. (Item 23) A shooting control method performed by a shooting control device that is communicatively connected to multiple cameras and capable of controlling at least one of the shooting direction and field of view for each of the multiple cameras, When changing the settings of the aforementioned multiple cameras all at once, The first camera among the aforementioned multiple cameras to have its settings changed is determined, To instruct the camera above 1 to change its settings, In the first camera, it is determined whether or not the change in the specified settings has been completed. A shooting control method characterized by: determining that the setting change specified in the first camera has been completed, and instructing the other cameras among the plurality of cameras to change their settings. (Item 24) A program for causing a computer to function as a control means of the imaging control device described in any one of items 1 to 21.

[0227] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0228] 100, 200…Camera, 300…Network, 400…Controller, 101, 201, 401…CPU, 102, 202, 402…RAM, 103, 203, 403…ROM, 105, 205, 404…Network I / F

Claims

1. A shooting control device that is communicatively connected to multiple cameras and capable of controlling at least one of the shooting direction and field of view for each of the multiple cameras, Having control means, When the control means changes the settings of the multiple cameras all at once, Of the aforementioned multiple cameras, determine which camera will have its settings changed first. The camera described above is instructed to change its settings. In the first camera, it is determined whether or not the specified setting change has been completed. A shooting control device characterized in that, when it is determined that the setting change specified in the first camera has been completed, it instructs the other cameras among the plurality of cameras to change their settings.

2. The shooting control device according to claim 1, characterized in that the control means does not change the settings of the other cameras until it is determined that the instructed setting change has been completed for at least one of the plurality of cameras other than the first camera.

3. The shooting control device according to claim 1, characterized in that the control means determines that the change of the instructed setting has been completed when the current setting value obtained from the first camera matches the instructed changed setting value.

4. The shooting control device according to claim 1, characterized in that the control means determines that the change of the instructed setting has been completed if the difference between the current setting value obtained from the first camera and the instructed changed setting value is less than or equal to a threshold.

5. The aforementioned instructions include instructions for the changed shooting direction, The shooting control device according to claim 1, characterized in that the control means determines that the change of the instructed setting has been completed when the current angle of view of the first camera includes the changed shooting direction.

6. The aforementioned change in settings involves mechanical action, The shooting control device according to claim 1, characterized in that the control means determines that the instructed setting change has been completed when the amount of change in the setting value obtained based on the time series of the current setting value acquired from the first camera decreases.

7. The shooting control device according to claim 1, characterized in that the control means determines that the instructed setting change has been completed when a predetermined time has elapsed since instructing the first camera to change the setting.

8. The shooting control device according to claim 7, characterized in that the predetermined time when the change of the instructed setting involves mechanical drive is longer than the predetermined time when the change of the instructed setting does not involve mechanical drive.

9. The shooting control device according to claim 1, characterized in that the control means determines that the instructed setting change has been completed after the predicted time required for the setting change has elapsed since instructing the first camera to change the setting.

10. The shooting control device according to claim 9, characterized in that the control means determines the first camera based on the predicted time.

11. The shooting control device according to claim 1, characterized in that the control means changes the settings of the plurality of cameras all at once in response to a user operation specifying one of the combinations of setting values ​​for each of the plurality of cameras stored in the shooting control device.

12. The aforementioned shooting control device automatically controls at least one of the shooting direction and field of view for each of the plurality of cameras so as to fit a specific subject into the field of view. The shooting control device according to claim 1, characterized in that the control means collectively changes at least one of the following for each of the plurality of cameras as a setting for the plurality of cameras: the specific subject, the shooting size for shooting the specific subject, and the composition for shooting the specific subject.

13. The shooting control device according to claim 1, characterized in that it automatically controls at least one of the shooting direction and field of view of the plurality of cameras according to the state of a camera other than the plurality of cameras.

14. The shooting control device according to claim 13, characterized in that it automatically controls at least one of the shooting direction and field of view of the plurality of cameras according to the role set for each of the plurality of cameras and the state of the other camera.

15. The shooting control device according to claim 13, characterized in that the state of the other camera includes at least one of the shooting direction and the subject of interest.

16. The shooting control device according to claim 15, characterized in that the state of the other camera includes the shooting direction, and the shooting direction of the plurality of cameras is changed in accordance with the change in the shooting direction of the other camera.

17. The shooting control device according to claim 15, characterized in that the state of the other camera includes a subject of interest, and the shooting direction of the plurality of cameras is changed so that the plurality of cameras track and photograph the subject of interest or a specific subject different from the subject of interest.

18. The shooting control device according to claim 17, characterized in that the control means determines, among the plurality of cameras, the camera that is not able to track the specific subject as the first camera.

19. The shooting control device according to claim 15, characterized in that, if the state of the other camera includes a subject of interest and the subject of interest is changed, the control means changes the settings of the multiple cameras all at once.

20. The shooting control device according to claim 14, characterized in that the control means changes the role set for each of the plurality of cameras collectively as the settings for the plurality of cameras.

21. The shooting control device according to claim 12, characterized in that the control means determines that the change of the instructed setting has been completed when the current field of view of the first camera includes the specific subject after the change.

22. A photographic control device according to any one of claims 1 to 21, Multiple cameras connected to the aforementioned shooting control device in a communicative manner, A multi-camera system characterized by having the following features.

23. A shooting control method performed by a shooting control device that is communicatively connected to multiple cameras and capable of controlling at least one of the shooting direction and field of view for each of the multiple cameras, When changing the settings of the aforementioned multiple cameras all at once, Among the aforementioned multiple cameras, determine the first camera whose settings will be changed first, To instruct the first camera described above to change its settings, In the first camera, it is determined whether or not the change in the instructed settings has been completed. A shooting control method characterized by: determining that the setting change specified in the first camera has been completed, and instructing the other cameras among the plurality of cameras to change their settings.

24. A program for causing a computer to function as the control means of the imaging control device according to any one of claims 1 to 21.

Images