Photographing control device and photographing control method, and imaging system

Abstract

To provide a photographing control device that automatically controls the operations of a plurality of cameras, and that can prevent a camera operation not intended by a user.SOLUTION: A photographic control device determines a specific subject within a photographic range as a subject to be tracked. The photographic control device controls the photographing directions of cameras to track and photograph the subject to be tracked. When images photographed by the cameras are selected by an external selection device as specific images, the photographing control device does not change the subject to be tracked that the cameras should track.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging control device, an imaging control method, and an imaging system, and more particularly to a technique for controlling a plurality of imaging devices. [Background technology]

[0002] For live streaming and recording, a multi-camera imaging system is used that uses a video switcher to dynamically select or switch the video (main line video) to be used for broadcasting or recording from the video captured in parallel by multiple cameras (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-45529 Summary of the Invention [Problem to be solved by the invention]

[0004] In such an imaging system, labor can be saved by introducing a photography control device that centrally controls the operation of the cameras. For example, if the photography control device automatically controls the operation of other cameras based on information obtained from one specific camera, a variety of photography becomes possible even with a small number of people.

[0005] On the other hand, it is necessary to devise a way to prevent automatic control that does not conform to the intention of the user (operator of the video switcher) who dynamically selects the main line video. Therefore, in one aspect, the present invention provides a photography control device and a photography control method that can suppress camera operations that are not intended by the user in a photography control device that automatically controls the operations of multiple cameras. [Means for solving the problem]

[0006] In one aspect, the present invention provides a shooting control device having a determination means for determining a specific subject within a shooting range as a tracking subject, and a control means for controlling the shooting direction of a camera so as to track and photograph the tracking subject, wherein the determination means does not change the tracking subject to be tracked by the camera when an image captured by the camera has been selected as the specific image by an external selection device. Effect of the Invention

[0007] According to the present invention, it is possible to provide a photography control device and a photography control method that can suppress camera operations unintended by a user in a photography control device that automatically controls the operations of a plurality of cameras. [Brief description of the drawings]

[0008] [Figure 1] Schematic diagram of an imaging system according to an embodiment. [Diagram 2] FIG. 1 is a block diagram showing an example of the functional configuration of each device in an imaging system according to an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating the main operations and signal flows of the imaging control device according to the embodiment. [Figure 4] 1 is a flowchart showing the operation of each device in an imaging system according to an embodiment of the present invention; [Diagram 5] FIG. 1 is a diagram for explaining coordinate conversion in an embodiment. [Figure 6] FIG. 1 is a diagram relating to subject detection and coordinate transformation in an embodiment. [Figure 7] FIG. 1 is a diagram for explaining pan value calculation in an embodiment; [Figure 8] FIG. 1 is a diagram for explaining tilt value calculation in an embodiment; [Figure 9] Flowchart regarding the operation of the imaging control device in the first embodiment [Figure 10] 5A to 5C are diagrams illustrating an example of how a circumscribing rectangular frame is displayed in the first embodiment. [Figure 11] Flowchart regarding the operation of the imaging control device in the second embodiment [Figure 12]Flowchart regarding the operation of the imaging control device in the second embodiment [Figure 13] Flowchart regarding the operation of the imaging control device in the third embodiment [Figure 14] FIG. 13 is a diagram showing examples of roles that can be set to a sub-camera in an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] First Embodiment (Overview of the multi-camera imaging system) 1 is a schematic diagram showing an example of the configuration of a multi-camera imaging system 10 (hereinafter simply referred to as an imaging system) according to this embodiment. The imaging system 10 has multiple cameras 300, 400a to 400c, an imaging control device 100, and a video switcher (hereinafter simply referred to as a switcher) 1000. The multiple cameras 300, 400a to 400c, the imaging control device 100, and the switcher 1000 are communicatively connected via a communication network 600.

[0011] Here, the switcher 1000 has an operator. The shooting control device 100 may also have an operator (user), but this is not essential. The operator of the switcher 1000 may be the user of the shooting control device 100. Since the shooting of the sub-cameras 400a to 400c is controlled by the shooting control device 100, no photographer is required. The overhead camera 300 also does not require a photographer.

[0012] The communication network 600 complies with known wired or wireless communication standards such as the IEEE802.3 series, IEEE802.11 series, etc. Each of the cameras 300, 400a to 400c, the shooting control device 100, and the switcher 1000 has a communication interface that complies with the standard of the communication network 600.

[0013] 1 shows all signals communicated via the communication network 600, video signals and control signals may be communicated in different ways. For example, each of the multiple cameras 300, 400a to 400c may directly supply a video signal to the shooting control device 100 or the switcher 1000 via a cable.

[0014] Of the multiple cameras 300, 400a to 400c, camera 300 captures the entirety of a predetermined shooting range 20. The shooting range 20 is set as a range in a studio, for example, where a subject to be shot may exist. Therefore, the video of camera 300 captures all of the subjects within the shooting range.

[0015] The purpose of the camera 300 is to capture an image for detecting a subject of the shooting target existing within the shooting range. Therefore, the shooting direction and angle of view of the camera 300 are determined according to the position of the camera 300 and the shooting range 20, and are basically fixed during shooting. In addition, it is preferable that the camera 300 captures the entire shooting range 20 so that it is not hidden by objects outside the shooting range 20. Therefore, the camera 300 is installed in a position where it overlooks the entire shooting range 20. In order to distinguish the camera 300 from the other cameras 400a to 400c whose shooting direction and angle of view during shooting are not basically fixed, the camera 300 will be referred to as an overhead camera below. However, the installation position of the camera 300 is not limited to a position where it overlooks the shooting range. The operation of the overhead camera 300 can be controlled by the shooting control device 100.

[0016] The cameras 400a to 400c are, for example, PTZ cameras, and the operations including the shooting direction (pan and tilt angles) and the angle of view (zoom) can be controlled by the shooting control device 100. The cameras 400a to 400c may be configured so that the shooting direction (pan and tilt angles) can be controlled by attaching the camera body to a camera platform. The cameras 400a to 400c may be configured so that a zoomable interchangeable lens is attached to the camera body. In the following, the cameras 400a to 400c are referred to as sub-cameras. In the following, the sub-cameras 400a to 400c are collectively referred to as sub-cameras 400. The number of sub-cameras 400 is not limited to three, and may be two or less, or four or more.

[0017] The switcher 1000 is a selection device that receives video signals from each of the sub-cameras 400a to 400c, selects a video from one sub-camera designated by the user, and outputs the video as a main line video.

[0018] The photography control device 100 detects a subject from a video signal received from the overhead camera 300. The photography control device 100 determines a subject to be tracked by each of the sub-cameras 400a to 400c based on the detection result. Then, the photography control device 100 determines a target photography direction (and a target angle of view, if necessary) for each of the sub-cameras 400a to 400c so that each of the sub-cameras 400a to 400c photographs the determined subject. The photography control device 100 transmits a control command including the determined target photography direction to the sub-cameras 400a to 400c. The photography control device 100 continuously executes subject detection, determination of the target photography direction, and transmission of the control command, thereby controlling each of the sub-cameras 400a to 400c to track and photograph a specific subject.

[0019] (Example of functional configuration of each device) Fig. 2 is a block diagram showing an example of the functional configuration of each device constituting the multi-camera imaging system shown in Fig. 1. The configuration expressed as a functional block in the drawing can be realized by an integrated circuit such as an ASIC or an FPGA, by a discrete circuit, or by a combination of a memory and a processor that executes a program stored in the memory. Also, one functional block may be realized by multiple integrated circuit packages, or multiple functional blocks may be realized by one integrated circuit package. Also, the same functional block may be implemented in different configurations depending on the operating environment, required capabilities, etc.

[0020] (Photography control device 100) First, a functional configuration example of the imaging control device 100 will be described. The imaging control device 100 may be a general-purpose computer device such as a personal computer or a workstation. The imaging control device 100 has a configuration in which a CPU 101, a RAM 102, a ROM 103, an inference unit 104, a network interface (I / F) 105, a user input I / F 106, a video input unit 107, and a display unit 108 are mutually connected via an internal bus 110.

[0021] The CPU 101 is a microprocessor capable of executing programmed instructions. The CPU 101, for example, loads a program stored in a ROM 103 into a RAM 102 and executes the program to realize the functions of the imaging control device 100, which will be described later. The CPU 101, for example, can realize the functions of the imaging control device 100 by executing an imaging control application that runs on an operating system (OS).

[0022] The RAM 102 is used to load programs executed by the CPU 101 and to temporarily store data to be processed by the CPU 101, data currently being processed, etc. Furthermore, a part of the RAM 102 may be used as a video memory for the display unit 108.

[0023] The ROM 103 is a rewritable non-volatile memory, and stores the programs (OS and applications) executed by the CPU 101, user data, and the like.

[0024] The inference unit 104 executes a detection process of a subject area using a machine learning model for the image of the overhead camera 300. The inference unit 104 can be implemented using a hardware circuit capable of executing the calculation of the machine learning model at high speed, such as a GPU (Graphics Processing Unit) or an NPU (Neural network Processing Unit). Alternatively, the inference unit 104 may be implemented using a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array). The CPU 101 may execute a program to realize the function of the inference unit 104.

[0025] The machine learning model may be a convolutional neural network (CNN) trained according to the type of subject to be detected. Here, the inference unit 104 detects a human body region or a human face region as a subject region from an input image. The inference unit 104 outputs the position and size of a rectangular region inscribed in the subject region and the detection reliability for each detected subject region. Note that, a plurality of types of machine learning models may be used to execute detection processing of different types of subject regions for the same input image. Note that the inference unit 104 may execute detection processing of the subject region by a known method that does not use a machine learning model. The inference unit 104 can detect the subject region by using, for example, a method using local features such as SIFT or SURF, or a method using pattern matching.

[0026] The network I / F 105 is an interface for connecting the photography control device 100 to a communication network 600. The photography control device 100 (CPU 101) can communicate with external devices on the communication network 600, such as the overhead camera 300, the sub-cameras 400a to 400c, and the switcher 1000, via the network I / F 105. The photography control device 100 may communicate with external devices via other communication interfaces (USB, Bluetooth (registered trademark), etc.) not shown.

[0027] To communicate with each device (overhead camera 300, sub cameras 400a to 400c, switcher 1000) on communication network 600, CPU 101 acquires the network address of each device at any timing and stores it in RAM 102. CPU 101 also acquires information about each device (device type, model name, etc.) at any timing (for example, at the time of the first communication) and stores it in RAM 102. In this way, CPU 101 is assumed to know at least the identification information and device type of overhead camera 300, sub cameras 400a to 400c, and switcher 1000. Note that the user may be allowed to give any name to each device.

[0028] The user input I / F 106 is an interface for connecting input devices (not shown) such as a mouse, a keyboard, a touch panel, etc. The imaging control device 100 receives instructions from a user through the user input I / F 106.

[0029] The video input unit 107 is an input interface for a video signal. The video input unit 107 may be a receiver conforming to the SDI (Serial Digital Interface) standard or the HDMI (High-Definition Multimedia Interface) (registered trademark) standard. The video input unit 107 receives a video signal from the overhead camera 300.

[0030] The display unit 108 is a display device such as a liquid crystal display (LCD), etc. The display unit 108 displays a GUI screen provided by the OS, a shooting control application, etc.

[0031] (Switcher 1000) Next, an example of the functional configuration of the switcher 1000 will be described. The CPU 1001 is a microprocessor capable of executing programmed instructions. For example, the CPU 1001 loads a program stored in a ROM 1008 into a RAM 1002 and executes the program, thereby controlling the operation of each functional block and realizing the functions of the switcher 1000, which will be described later.

[0032] The RAM 1002 is used to load programs to be executed by the CPU 1001, and to temporarily store data to be processed by the CPU 1001, data being processed, etc. The RAM 1002 may also be used as a buffer for the video signal received from the sub-camera 400.

[0033] The ROM 1008 is a rewritable non-volatile memory, and stores the programs executed by the CPU 1001, the settings of the switcher 1000, user data, and the like.

[0034] The video input units 1003a to 1003c are input interfaces for video signals. The video input units 1003a to 1003c may be receivers conforming to, for example, the SDI standard or the HDMI standard. The video input unit 1003a receives a video signal from the sub-camera 400a, the video input unit 1003b receives a video signal from the sub-camera 400b, and the video input unit 1003c receives a video signal from the sub-camera 400c. The number of the video input units 1003 may be equal to or greater than the number of the sub-cameras 400.

[0035] 2 shows a configuration in which the video input unit 1003 and the sub-camera 400 are connected in a one-to-one relationship. However, video signals from one or more of the sub-cameras 400a to 400c may be supplied to the switcher 1000 through the communication network 600. In this case, the switcher 1000 receives the video signals through the network I / F 1007. Note that there may be a mixture of sub-cameras 400 that supply video signals to the video input unit 1003 and sub-cameras 400 that supply video signals through the communication network 600.

[0036] Video switch control unit 1004 selects a video signal selected through user input I / F 1006 as a specific video from among multiple video signals input to video input units 1003a to 1003c. Here, video switch control unit 1004 can select a main line video and a preview video as the specific video. Furthermore, video switch control unit 1004 outputs the video signal selected as the main line video to video output unit 1005. The preview video is a video that is scheduled to be the next main line video.

[0037] The video output unit 1005 is an interface that outputs the video signal selected by the video switch control unit 1004 to an external device (for example, a live distribution device or a program recording device). The video output unit 1005 may be, for example, a transmitter that complies with the SDI standard or the HDMI standard.

[0038] The monitor output unit 1009 generates an image of a screen displaying the video signals from the sub-cameras 400a to 400c received through the video input unit 1003 or the network I / F 1007 in an arranged manner, and outputs the image to an external display device (not shown). The monitor output unit 1009 also outputs an image of the setting screen of the switcher 1000 to the external display device.

[0039] The user input I / F 1006 is an interface for connecting an input device (not shown) such as a button, a dial, a joystick, a touch panel, etc. The switcher 1000 receives user instructions regarding selection of a main line video and a preview video and settings of the switcher 1000 through the user input I / F 1006. The preview video is a video that is scheduled to be the next main line video.

[0040] CPU 1001 classifies the state of each of sub cameras 400a-400c into one of streaming, previewing, and standby based on a user instruction through user input I / F 106. CPU 1001 then associates the identification information of each of sub cameras 400a-400c with the determined state and stores them in RAM 1002. Hereinafter, information indicating the state of each sub camera is referred to as camera state information STREAMING. CPU 1001 updates the camera state information STREAMING, for example, every time the selection of the video signal is changed.

[0041] Specifically, CPU 1001 determines the status of the sub camera capturing the video (main line video) being output from video output unit 1005 to be "distributing". CPU 1001 also determines the status of the sub camera capturing the preview video to be "previewing". CPU 1001 further determines the status of the sub camera capturing the remaining video to be "standby". The identification information of sub camera 400 may be any information capable of identifying each individual sub camera, such as a unique name given by the user, a serial number, or a network address.

[0042] Based on the camera state information STREAMING, the CPU 1001 transmits a control signal (tally information) for performing a tally display to the sub camera during streaming and preview via the network I / F 1007.

[0043] The network I / F 1007 is an interface for connecting the switcher 1000 to the communication network 600. The switcher 1000 (CPU 1001) can communicate with external devices on the communication network 600, such as the overhead camera 300, the sub-cameras 400a to 400c, and the shooting control device 100, via the network I / F 1007. The switcher 1000 may communicate with external devices via other communication interfaces (USB, Bluetooth, etc.) not shown.

[0044] (Overhead camera 300) Next, an example of the functional configuration of the overhead camera 300 will be described. The CPU 301 is a microprocessor capable of executing programmed instructions. For example, the CPU 301 loads a program stored in the ROM 303 into the RAM 302 and executes the program to control the operation of each functional block and realize the function of the overhead camera 300, which will be described later.

[0045] The RAM 302 is used to load programs executed by the CPU 301 and to temporarily store data to be processed by the CPU 301, data being processed, etc. The RAM 302 may also be used as a buffer for video signals obtained by shooting.

[0046] The ROM 308 is a rewritable non-volatile memory. The ROM 308 stores the programs executed by the CPU 301, the setting values ​​of the overhead camera 300, user data, etc. The ROM 308 can also be used as a recording destination for video signals. The ROM 308 may include an internal memory and a removable memory card.

[0047] The image sensor 307 has a photographing optical system and an image sensor. The image sensor may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The image sensor has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from each pixel. Each pixel accumulates electric charge according to the amount of incident light by photoelectric conversion. A pixel signal group (analog image signal) representing the subject image formed on the imaging surface is obtained by reading out from each pixel a signal having a voltage according to the amount of electric charge accumulated during the exposure period.

[0048] The image processing unit 306 applies predetermined signal processing and image processing to the analog image signal output by the image sensor 307 to generate signals and image data according to the application, and to obtain and / or generate various types of information.

[0049] The processes applied by the image processing unit 306 can include, for example, pre-processing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effect processing. Pre-processing may include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on the image sensor 307, and is processing for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging optical system (image restoration), correction of the effect of peripheral light falloff of the imaging optical system, color correction, and the like. The data processing may include processes such as region extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. The generation of the video signal to be output to the switcher 000 and the video data to be recorded in the ROM 308 are also included in the data processing. The evaluation value calculation process may include processes such as generation of a signal and evaluation value used in automatic focus detection (AF), generation of an evaluation value used in automatic exposure control (AE), etc. AF and AE are executed by the CPU 301. The special effects processing may include adding a blur effect, changing color tones, relighting, and the like. Note that these are merely examples of processes that the image processing unit 306 can apply, and do not limit the processes that the image processing unit 306 can apply. The image processing unit 306 outputs the acquired or generated information and data to the CPU 301, the RAM 302, the video output unit 311, etc. depending on the purpose.

[0050] The type and settings of the processing applied by the image processing unit 306 can be controlled by sending commands from the photography control device 100 to the overhead camera 300.

[0051] The network I / F 305 is an interface for connecting the overhead camera 300 to the communication network 600. The overhead camera 300 (CPU 301) can communicate with external devices on the communication network 600, such as the shooting control device 100, the switcher 1000, and the sub-cameras 400a to 400c, via the network I / F 305. The overhead camera 300 may communicate with external devices via other communication interfaces (USB, Bluetooth, etc.) not shown.

[0052] The video output unit 311 is an interface for outputting a video signal captured using the image sensor 307 to an external device (here, the shooting control device 100). The video output unit 311 may be, for example, a transmitter conforming to the SDI standard or the HDMI standard.

[0053] When the monitor output is enabled and an external display device (not shown) is connected to the overhead camera 300, the monitor output unit 312 generates an image for a screen that displays the video signal obtained by imaging and outputs the image to the external display device.

[0054] (Sub-camera 400) Next, an example of the functional configuration of the sub-cameras 400a to 400c will be described. Note that, here, it is assumed that the sub-cameras 400a to 400c have the same functional configuration, and an example of the functional configuration of the sub-camera 400a will be described as a representative.

[0055] The functional blocks having the same names in the sub camera 400a and the overhead camera 300 have the same functions, and the description thereof will be omitted.

[0056] As described above, the sub-camera 400 is a PTZ camera, and the shooting direction and angle of view can be controlled externally. Therefore, the sub-camera 400 has a driving unit 409 capable of panning, tilting, and zooming, and a driving I / F 408. The driving I / F 408 is a communication interface between the driving unit 409 and the CPU 401.

[0057] The driving unit 409 has a pan / tilt mechanism that supports the sub-camera 400 so that it can pan and tilt, a zoom mechanism that changes the angle of view of the photographing optical system, and a motor that drives these mechanisms. The zoom mechanism may use the image enlargement / reduction by the image processing unit 406. The driving unit 409 drives the motor according to instructions received from the CPU 401 via the driving I / F 408, and adjusts the optical axis direction and the angle of view of the photographing optical system.

[0058] The status display unit 412 is a so-called tally lamp. Here, it is assumed that the status display unit 412 has red and green LEDs. Either none of the two LEDs of the status display unit 412 is lit, or only one is lit, never both. The CPU 401 controls the display of the status display unit 412 based on the tally information received from the shooting control device 100. Here, the CPU 401 controls the status display unit 412 so that the red LED is lit when the received tally information indicates "distributing", and the green LED is lit when the received tally information indicates "previewing". Also, when the received tally information indicates "standby", the CPU 401 controls the status display unit 412 so that the red and green LEDs are not lit.

[0059] The number of LEDs in the status display unit 412 may be one or three or more. Any color may be used as long as there are no two or more of the same color. However, the relationship between the tally information and the display state of the status display unit 412 must be consistent across all of the sub-cameras 400.

[0060] <Explanation of the operation of each device> Next, the operation of each device in the multi-camera imaging system when the imaging control device 100 automatically controls the imaging operations of the sub-cameras 400a to 400c based on the video of the overhead camera 300 will be described with reference to FIGS.

[0061] 3 is a diagram showing a series of processes performed when the photography control device 100 controls the operations of the sub-cameras 400a to 400c, focusing on main operations and signal flows. The functional blocks shown in the photography control device 100 show the main operations in a schematic manner, and correspond to the main functions provided by the photography control application. Each functional block in FIG. 3 is realized by a combination of the CPU 101 that executes the photography control application and one or more of the functional blocks of the photography control device 100 shown in FIG. 2.

[0062] 4(a) to 4(d) are respectively flowcharts relating to the operations of the shooting control device 100, the overhead camera 300, the sub camera 400, and the switcher 1000. The operations of the sub cameras 400a to 400c are basically the same, so they will be explained together.

[0063] In the following description, it is assumed that the three-dimensional coordinate values ​​of the viewpoint position and the shooting direction (optical axis direction) of the overhead camera 300 are known to the shooting control device 100. It is also assumed that known position information, such as the three-dimensional coordinate values ​​of the viewpoint positions of the sub-cameras 400a to 400c and the coordinate values ​​of the markers arranged in the shooting range 20, is stored in advance in the ROM 103 as default position information REF_POSI. It is also assumed that the coordinate system of the position is predetermined depending on the type of position.

[0064] (Operation of the imaging control device 100) First, the operation of the photography control device 100 will be described with reference to Fig. 3 and Fig. 4(a). The operation described below is realized by the CPU 101 executing a photography control application.

[0065] In S101, CPU 101 transmits a shooting instruction command in a predetermined protocol to overhead camera 300 through network I / F 105. In response to this command, overhead camera 300 starts supplying a video signal (moving image data) IMG to video input unit 107. CPU 101 starts storing the video signal received by video input unit 107 in RAM 102, and then executes S102.

[0066] In S102, the recognition unit 121 executes the following process. (1) Apply subject area detection processing to the input frame image and store the detection results. (2) For each detected subject area, the position information (image coordinates) is converted. (3) Apply classification processing to each detected subject region to identify the classification information (if a new subject is detected, add classification processing information). (4) For each detected subject area, the identification information ID[n] and the position information POSITION[n] are associated and stored.

[0067] The recognition unit 121 is mainly realized by the CPU 101 and the inference unit 104. The CPU 101 reads one frame of the image received from the overhead camera 300 from the RAM 102 and inputs it to the inference unit 104.

[0068] The operation of the recognition unit 121 will be explained step by step below. (1) First, the inference unit 104 inputs a frame image into a machine learning model to detect a subject region. The inference unit 104 stores in the RAM 102 the position and size of each detected subject region, which are output by the machine learning model as a detection result, and the detection reliability. The position and size of the subject region may be any information capable of identifying the position and size of a rectangular region inscribed by the subject region. Here, the center coordinate of the bottom side, and the width and height of the rectangular region are used as the position and size of the subject region.

[0069] Furthermore, inference unit 104 stores the detection result for the first frame image in RAM 102 in association with the subject identification information ID[n]. Here, n is the subject number and is an integer ranging from 1 to the total number of detected subject regions. Furthermore, inference unit 104 stores the subject regions detected from the first frame image in RAM 102 in association with the subject identification information ID[n] as templates for identifying individual subjects. If template matching is not used to identify subjects, it is not necessary to store templates.

[0070] Fig. 6(a) shows an example of the result of subject detection processing by inference unit 104 for the image captured by overhead camera 300 shown in Fig. 5(a). Here, the areas of human subjects A to C present within shooting range 20 are detected, and the coordinates (foot coordinates) of the center of the bottom side of a rectangular area inscribed with the subject areas are output as the position.

[0071] For coordinate conversion, which will be described later, when a marker is placed at a known position within the shooting range 20 as shown in FIG. 5(b), the CPU 101 detects the image of the marker included in the frame image (FIG. 5(a)) and stores the position in the RAM 102. The detection of the marker image may also be performed by the inference unit 104. The detection of the marker image may be performed by any known method, such as pattern matching using a marker template. The marker image may also be detected using a pre-stored machine learning model for marker detection.

[0072] (2) Next, we will explain the coordinate conversion executed by the inference unit 104. Fig. 5(a) shows a schematic diagram of the image captured by the overhead camera 300, and Fig. 5(b) shows a schematic diagram of the image capture range 20 as viewed from directly above its center. The inference unit 104 converts the position of the subject area in the coordinate system of the overhead camera into values ​​in a coordinate system (planar coordinate system) when the image capture range 20 is viewed from directly above its center.

[0073] The reason for converting the coordinates into values ​​in the plane coordinate system is that it is convenient to calculate a pan value (a movement angle in a horizontal plane) for making the sub-cameras 400a to 400c capture a specific subject. Note that this is based on the premise that the sub-cameras 400 are installed so that the driving unit 409 performs a panning operation in a horizontal plane parallel to the floor of the shooting range 20.

[0074] Coordinate conversion can be performed in various ways, but here, markers are placed at multiple known positions on the floor of the shooting range 20, and coordinate conversion is performed from the overhead camera coordinate system to the planar coordinate system based on the marker positions in the image captured by the overhead camera 300. Note that coordinate conversion may also be performed without using markers, for example, by using the viewpoint position and shooting direction of the overhead camera 300.

[0075] The coordinate transformation can be performed using a homography transformation matrix H according to Equation 1 below.

number

[0076] The homography transformation matrix can be calculated by substituting the coordinates of the four markers detected from the video and the coordinates (known) of the four markers arranged in the shooting range 20 into Equation 1 and solving the simultaneous equations. When the positional relationship between the shooting range 20 and the overhead camera 300 is fixed, the homography transformation matrix H can be calculated in advance during test shooting and stored in, for example, ROM 103.

[0077] CPU 101 sequentially reads out the positions of the subject regions from RAM 102 and converts the coordinates into values ​​in a planar coordinate system. Fig. 6(b) shows a schematic diagram of the state in which the foot coordinates (x, y) of each subject region detected in the video of overhead camera 300 shown in Fig. 6(a) are converted into coordinate values ​​(X, Y) in a planar coordinate system using Equation 1 and homography transformation matrix H stored in ROM 103. CPU 101 stores the converted foot coordinates in RAM 102 as POSITION[n].

[0078] (3) Next, the operation of the inference unit 104 to identify the identification information ID[n] of the subject will be described. Here, the subject is identified using template matching. The subject is identified for the processing results of the second and subsequent subject detections. For the first processing result, a new identification information ID[n] can be assigned to the subject region.

[0079] The inference unit 104 identifies the identification information ID[n] of the detected object region by template matching using the template stored in the RAM 102. This identifies the object within the shooting range. For example, the inference unit 104 calculates an evaluation value representing the correlation of each template for each detected object region. Then, the inference unit 104 identifies the identification information ID[n] corresponding to the template having a certain degree of correlation or higher and having the highest correlation as the identification information ID[n] of the object region. For the evaluation value, a known value such as the sum of absolute differences of pixel values ​​can be used.

[0080] Note that for a subject region that does not have a certain level of correlation with all the templates, the inference unit 104 assigns new identification information ID[n] and adds the image of the subject region to the template.

[0081] In addition, the inference unit 104 may update an existing template using a subject area detected in a most recent frame image, or delete a template in which a subject area having a certain degree of correlation or more has not existed for a certain period of time. Furthermore, the inference unit 104 may store in the ROM 103 templates corresponding to frequently appearing identification information ID[n].

[0082] Note that the subject may be identified by a method other than template matching. For example, the subject may be identified as the same identification information ID[n] as the subject area having the closest immediately preceding detection position and / or size. Alternatively, the position in the current frame image may be predicted by a Kalman filter or the like from the transition of positions in multiple past detection results associated with the same identification information, and the same identification information ID may be identified as the subject area that is closest to the predicted position. These methods may also be combined. By not using template matching, it is possible to improve the accuracy of identifying different subjects that look similar.

[0083] (4) The inference unit 104 associates the specified identification information ID[n] with the position (planar coordinate system) POSITION[n] of the corresponding subject region and stores them in the RAM 102.

[0084] Of the processes (1) to (4), the processes other than the subject detection may be executed by the CPU 101 instead of the inference unit 104.

[0085] Here, the image from the overhead camera 300 was used to determine the identification information ID[n] and the position POSITION[n] of the subject within the shooting range 20. However, the image from the sub camera 400 may also be used. In this case, the CPU 101 executes the operation shown in the flowchart of FIG. 4(a) for each sub camera. The position of the subject area is output as a value in the coordinate system for each sub camera 400. Thus, although the overhead camera 300 is not essential, it is considered that the subject detection accuracy is better when the overhead camera 300 is used.

[0086] Returning to the explanation of Fig. 4(a), in S103, the CPU 101 as the target subject determination unit 122 in Fig. 3 determines a target subject. The CPU 101 can determine, as the target subject, a subject designated by the user or a subject selected based on a predetermined condition.

[0087] When allowing the user to select a subject of interest, CPU 101 causes display unit 108 or an external display device to display the frame image to which subject detection processing has been applied in S102 together with an index indicating the detected subject area. The index may be, for example, a rectangular frame indicating the outer edge of the subject area as shown in Fig. 6(a), but may also be another index. CPU 101 may also cause display unit 108 to display a message prompting the user to select a subject of interest within the image.

[0088] A user can select a subject area corresponding to a desired subject of interest by operating an input device connected to the user input I / F 106. There are no particular limitations on the selection method, and the selection method may be an operation of designating the desired subject area by operating a mouse or a keyboard.

[0089] When CPU 101 detects a user operation to designate any one of the subject regions, it stores the identification information ID[n] corresponding to the designated subject region as the identification information MAIN_SUBJECT of the subject of interest in RAM 102. Alternatively, CPU 101 automatically stores the identification information corresponding to one subject region that meets a predetermined condition as the identification information of the subject of interest in RAM 102. CPU 101 stores, for example, the identification information ID[n] corresponding to the subject region closest to the center of the screen or the largest subject region among the subject regions whose distance from the center of the screen is equal to or less than a threshold in RAM 102 as the identification information MAIN_SUBJECT.

[0090] Alternatively, CPU 101 may determine a subject captured by a predetermined one of sub-cameras 400a to 400c as a subject of interest. In this case, the image from the predetermined one of sub-cameras is supplied to shooting control device 100 via communication network 600 or via image input unit 107.

[0091] Furthermore, the CPU 101 stores in the RAM 102 the position information POSITION_OH corresponding to the identification information MAIN_SUBJECT of the tracking subject.

[0092] After that, CPU 101 executes S104. In S104, CPU 101 functioning as tracking subject determination unit 123 in Fig. 3 determines subjects to be tracked and photographed by each of sub-cameras 400a to 400c.

[0093] Here, it is assumed that all the sub-cameras are caused to track and photograph the target subject, and therefore the CPU 101 sets the identification information SUBJECT_ID of the target subject to the identification information MAIN_SUBJECT of the target subject determined in S103.

[0094] Note that there may be sub-cameras that capture subjects other than the target subject. For example, when three subjects are present in the shooting range 20 as shown in Fig. 6(b), different tracking subjects may be determined for each of the sub-cameras 400a to 400c.

[0095] CPU 101 writes the identification information SUBJECT_ID of the determined tracking subject to RAM 102. If the tracking subject may differ depending on the sub camera, CPU 101 stores the identification information SUBJECT_ID of the tracking subject in association with the identification information of the sub camera. Note that when the tracking subject changes, CPU 101 retains the information of the previous tracking subject in RAM 102 without erasing it.

[0096] Next, in S105, the CPU 101 serving as the pan-tilt calculation unit 124 calculates the amounts of change in the pan angle and tilt angle necessary for the sub-camera to perform tracking shooting of the tracking subject determined in S104.

[0097] Here, it is assumed that the following information is stored in advance in the ROM 103 as the default position information REF_POSI for each of the sub-cameras 400a to 400c. · Three-dimensional coordinates of the installation position (values in a plane coordinate system) · Shooting direction corresponding to the initial values of the pan angle and tilt angle of the drive unit · Controllable range of the pan and tilt angles

[0098] The CPU 101 reads out the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking subject from the RAM 102. Then, the CPU 101 first determines the pan angle from the position information POSITION_OH and the installation position of the sub-camera.

[0099] FIG. 7 is a diagram showing an example of the positional relationship between one sub-camera and a tracking subject in a plane coordinate system. Here, it is assumed that the pan angle θ for directing the optical axis direction of the sub-camera toward the subject position is determined. The CPU 101 calculates the pan angle θ using the following equation 2.

Equation

[0100] In Equation 2, px and py are the horizontal coordinate and vertical coordinate of the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking subject. Also, subx and suby are the horizontal coordinate and vertical coordinate of the installation position of the sub-camera. Here, it is assumed that the current pan angle is the initial value of 0° and the optical axis direction is the vertical direction (Y-axis direction). If the current optical axis direction is not the vertical direction, the angle difference between the current optical axis direction and the vertical direction may be reflected in the angle obtained by Equation 2. Also, the direction of the pan is counterclockwise if subx > px, and clockwise if subx < px.

[0101] Next, with reference to FIG. 8, a method for determining the tilt angle will be described. FIG. 8 shows a state in which the sub-camera and the tracking subject are viewed from the side. Assume that the current optical axis of the sub-camera 400 is in the horizontal direction and has a height of h1, and the height of the face of the tracking subject facing the optical axis is h2. Let ρ be the angle difference (tilt angle) in the height direction between the current optical axis direction and the target optical axis direction. The CPU 101 calculates the tilt angle ρ using the following equations (3) and (4).

Equation

[0102] The coordinate values used in Equation (4) are the same as those used in Equation (2). Assume that h1 and h2 are input in advance to the shooting control application and stored in the RAM 102. In this case, the identification number associated with h2 for each subject is made equal to the identification number assigned in the subject detection process. Alternatively, h2 may be a value measured in real time using a sensor (not shown).

[0103] Here, assume that the current tilt angle is the initial value of 0°, and the optical axis direction is the horizontal direction (constant height). If the current optical axis direction is not the horizontal direction, the angle difference between the current optical axis direction and the horizontal direction may be reflected in the angle obtained by Equation (4). Also, the tilt direction is downward if h1 > h2, and upward if h1 < h2.

[0104] The CPU 101 communicates periodically with each of the sub-cameras 400 through the communication network 600, acquires the current optical axis direction (the pan angle and tilt angle of the drive unit), and stores it in the RAM 102. Note that the communication cycle can be, for example, less than or equal to the reciprocal of the frame rate. Alternatively, the CPU 101 may hold the total values of the pan angle and tilt angle controlled from the initial state for each of the sub-cameras 400 in the RAM 102 and use them as the current optical axis direction.

[0105] In this way, the CPU 101 calculates the amount of change in the pan angle and tilt angle for each of the sub cameras 400 and stores the amount of change in the RAM 102 .

[0106] The change amount of the pan angle and tilt angle may be the angular velocity for the sub camera 400 to rotate in the direction of the tracking subject. For example, the CPU 101 acquires the current pan angle and tilt angle from each of the sub cameras 400 through the communication network 600. Then, the CPU 101 calculates the angular velocity of the pan proportional to the difference between the pan angle θ read from the RAM 102 and the current pan angle. The CPU 101 also calculates the angular velocity of the tilt proportional to the difference between the tilt angle ρ read from the RAM 102 and the current tilt angle. The CPU 101 stores the angular velocity calculated in this manner in the RAM 102.

[0107] The change amount of the pan angle and the tilt angle may be calculated using the image of the sub camera 400 instead of the image of the overhead camera 300. In this case, the CPU 101 may calculate the change amount of the pan angle from the horizontal difference between the current optical axis direction and the direction of the tracked subject in the coordinate system of the sub camera 400, and may calculate the change amount of the tilt angle from the vertical difference. Also, the imaging system may change the shooting direction to track and shoot the tracked subject in only one of the pan direction and the tilt direction, and in such an imaging system, the change amount of only one of the pan angle and the tilt angle may be calculated.

[0108] Next, in S106, CPU 101 reads out the change amounts of the pan and tilt angles calculated in S105 from RAM 102. Then, CPU 101 generates a control command PT_VALUE for each sub camera, which instructs the change of the pan angle and tilt angle corresponding to these change amounts. The format of the control command is determined in advance. CPU 101 stores the generated control command PT_VALUE in RAM 102.

[0109] Next, in S107, the CPU 101 reads out the control command PT_VALUE generated in S from the RAM 102, and transmits it to the communication network 600 via the network I / F 105. The sub-cameras 400a to 400c receive the control command PT_VALUE addressed to them via the network I / F 405.

[0110] The CPU 101 executes the process from S101 on the next frame image of the video captured by the overhead camera 300. Note that the process shown in Fig. 4(a) does not necessarily have to be executed for every frame.

[0111] (Operation of overhead camera 300) Next, the operation of the overhead camera 300 will be described with reference to Fig. 4(b). The operation described below is realized by the CPU 301 executing a program.

[0112] When the overhead camera 300 is powered on, each functional block is initialized by the CPU 301, and then the camera enters a shooting standby state. In the shooting standby state, the CPU 301 starts a video shooting process for live view display. Display image data generated by the image processing unit 306 is output via the monitor output unit 312.

[0113] In the shooting standby state, the CPU 301 waits for reception of a control command through the network I / F 305. When the CPU 301 receives a control command, it executes an operation according to the control command. Here, an operation when a shooting command is received as a control command from the shooting control device 100 will be described.

[0114] In S201 , the CPU 301 receives an imaging command from the imaging control device 100 via the network I / F 305 .

[0115] The shooting command may specify shooting parameters such as frame rate, resolution, etc. Also, the shooting command may include settings related to the processing to be applied by the image processing unit 306.

[0116] In S202, in response to receiving the shooting command, the CPU 301 starts a moving image shooting process for supplying the image to the shooting control device 100. In this moving image shooting process, a moving image with higher image quality is shot than that of the moving image shooting process for live view display. For example, at least one of the resolution of the moving image and the shooting frame rate is higher than that of the moving image for live view display. The image processing unit 306 applies processing to the image based on the settings for the moving image to be supplied to the shooting control device 100. The image processing unit 306 sequentially stores the generated moving image data in the RAM 302.

[0117] In S203, the CPU 101 reads out the moving image data from the RAM 302, and supplies the data from the video output unit 311 to the video input unit 107 of the shooting control device 100 as a video signal.

[0118] Thereafter, the process from shooting to supplying the video signal continues until a control command to stop shooting is received.

[0119] (Operation of sub camera 400) Next, the operation of the sub-camera 400 will be described with reference to Fig. 4(c). The operation described below is realized by the CPU 401 executing a program. Note that the sub-cameras 400a to 400c execute the same operation, so the operation will be described here as the operation of the sub-camera 400.

[0120] When the sub-camera 400 is powered on, each functional block is initialized by the CPU 401, and then moving image capturing processing for supply to the switcher 1000 is started. The image processing unit 406 applies processing to the analog image signal obtained from the image sensor 407 based on the settings for moving images to be supplied to the switcher 1000. The image processing unit 406 sequentially stores the generated moving image data in the RAM 402. The CPU 401 reads out the moving image data from the RAM 402, and supplies it from the video output unit 411 to the video input unit 1003 of the switcher 1000 as a video signal.

[0121] The CPU 401 waits for a control command via the network I / F 305 while supplying a video signal to the switcher 1000. When the CPU 401 receives a control command, it executes an operation according to the control command. Here, an operation will be described when a pan / tilt operation command from the photography control device 100 or a state notification command from the switcher 1000 is received as the control command.

[0122] In S301 , the CPU 401 receives a control command from the imaging control device 100 or the switcher 1000 via the network I / F 405 .

[0123] In S302, the CPU 401 determines whether the received control command is a pan / tilt operation command or a status notification command. If it is determined to be a pan / tilt operation command, the CPU 401 executes S303, and if it is determined to be a status notification command, the CPU 401 executes S306. Descriptions of operations for other control commands will be omitted.

[0124] In S303, the CPU 401 reads out the operation direction (pan and / or tilt direction) and the corresponding operation amount (angle) from the control command, and stores them in the RAM 302.

[0125] In S304, the CPU 401 generates drive parameters for the drive unit 409 based on the operation direction and operation amount read out in S303. For example, the CPU 401 may obtain drive parameters according to a combination of the operation direction and operation amount by using a table previously stored in the ROM 403. Note that when the operation amount is given as a target value (for example, a target angle), the CPU 410 obtains the drive parameters from the difference with the current value.

[0126] In S305, the CPU 401 controls the driving unit 409 through the driving I / F 408 based on the driving parameters acquired in S304. As a result, the driving unit 409 is driven by the specified amount of operation in the operation direction specified by the control command.

[0127] In S306 , the CPU 401 reads out the state information included in the state notification command, and stores it in the RAM 402 .

[0128] In S307, the CPU 401 determines the status information stored in the RAM 402. Here, it is assumed that the status information is one of "Distributing," "Previewing," and "Waiting." Then, the CPU 401 controls the status display unit 412 according to the determined status information.

[0129] As an example, the CPU 401 turns on the red LED and turns off the green LED when the status information is "distributing". Also, the CPU 401 turns on the green LED and turns off the red LED when the status information is "previewing". Furthermore, the CPU 401 turns off both the red and green LEDs when the status information is "standby".

[0130] The status notification command is not limited to being received through the network I / F 405, but may be received through another communication interface.

[0131] (Switcher 1000 operation) Next, the operation of the switcher 1000 will be described with reference to Fig. 4(d). The operation described below is realized by the CPU 1001 executing a program. Here, the operation will be described when the switcher 1000 detects an instruction to change (switch) the main line video or the preview video through the user input I / F 1006, or when it receives a status information request command through the network I / F 1007.

[0132] In step S401 , the CPU 1001 detects a user instruction or a control command via the user input I / F 1006 or the network I / F 1007 .

[0133] In S402, the CPU 1001 determines the content detected in S401. If it is determined that a video switching instruction has been detected through the user input I / F 1006, the CPU 401 executes S403, and if it is determined that a status information request command has been received through the network I / F 1007, the CPU 401 executes S406.

[0134] The video switching instruction is an instruction to change the main line video (video output from video output unit 1005) or the preview video (video to be used as the next main line video) among the videos supplied from sub-cameras 400a to 400c to video input units 1003a to 1003c. Note that the main line video and the preview video do not have to be selected. For example, an instruction to change the main line video or the preview video is input as an operation on the display screen of the input video via a mouse or a keyboard connected to user input I / F 1006.

[0135] In S403, if the instruction is to change the main line video, the CPU 1001 controls the video switch control unit 1004 to switch the video to be supplied to the video output unit 1005 as the main line video.

[0136] Then, in S404, the CPU 1001 notifies at least the sub-cameras whose status has been changed of the changed status information. It is not necessary to notify the status information to the sub-cameras whose status has not been changed. The status information is transmitted to the communication network 600 via the network I / F 1007.

[0137] Instead of notifying the status information, the status of the status display unit 412 of the sub-camera 400 may be directly controlled from the switcher 1000. In this case, it is necessary that each status display unit 412 of the sub-camera 400 is connected to the switcher 1000. As long as the status information can be notified to the sub-camera 400, the means for notifying the sub-camera 400 is not limited.

[0138] In S405, the CPU 1001 stores in the RAM 1002 status information STREAMING that associates identification information with a status for each sub camera.

[0139] In S406 , the CPU 1001 reads the state information STREAMING stored in the RAM 1002 .

[0140] In S407 , the CPU 1001 transmits the status information STREAMING to the imaging control device 100 via the network I / F 1007 .

[0141] When an instruction to check the status information is detected via the user input I / F 1006, the CPU 401 reads out the status information STREAMING stored in the RAM 1002, and outputs a list screen of the status information to the external display device via the monitor output unit 1009.

[0142] (Tracking subject decision operation taking into account the state of the sub camera) Next, the operation of the shooting control device 100 to determine a subject to be tracked, taking into account the state of the sub-camera 400, will be described with reference to the flowchart shown in Fig. 9. The operation shown in Fig. 9 can be performed, for example, instead of S104 in Fig. 4(a).

[0143] In S501, the CPU 101 determines whether or not the target subject has been changed as a result of the processing in S103, and if it is determined that the target subject has been changed, executes S502, and if not, ends the processing of the flowchart shown in FIG.

[0144] The CPU 101 stores at least identification information MAIN_SUBJECT of the most recently determined target subject in the RAM 102. Then, the CPU 101 determines that the target subject has been changed if the target subject determined in S103 is different from the most recently determined target subject.

[0145] Steps S502 to S506 are executed for each sub-camera with which the photography control device 100 can communicate. Information on the communicable sub-cameras 400 can be acquired by the CPU 101 from each of the sub-cameras 400a to 400c via the network I / F 105 when the photography control device 100 is started, for example. Alternatively, the CPU 101 may acquire information on each of the sub-cameras 400a to 400c from the switcher 1000. The CPU 101 stores the acquired information on the sub-cameras in the RAM 102.

[0146] The user may register, in the photography control application, the identification information (e.g., network address, serial number, etc.) of the sub camera to be controlled, instead of acquiring it through the network I / F 105. In either case, the CPU 101 knows the identification information of the sub camera to be controlled, and executes the processes of S502 to S506 while sequentially changing the target sub camera.

[0147] In S502, the CPU 101 transmits a status information request command to the switcher 1000 via the network I / F 105. In response to the status information request command, the CPU 101 receives status information STREAMING from the switcher 1000 and stores it in the RAM 102. Note that although the statuses of the sub-cameras 400a to 400c are acquired collectively from the switcher 1000 here, the statuses may be acquired individually from the sub-cameras 400a to 400c. In this case, the amount of communication between the switcher 1000 and the shooting control device 100 can be reduced.

[0148] In S503, CPU 101 refers to the status information STREAMING stored in RAM 102, and acquires the status of the sub camera 400 that is the target of processing. Then, CPU 101 determines whether the status is "streaming". If CPU 101 determines that the status is "streaming", it executes S504, and if not, it executes S505.

[0149] When the CPU 101 does not determine that the state is "distributing", it may further determine whether the state is "previewing" or not, and may execute S505 when the state is not determined to be "previewing". This makes it possible to prevent the subject of the video that the operator of the switcher 1000 plans to use as the next main line video from being changed unintentionally.

[0150] In S504, the CPU 101 notifies the user of the refusal to change the target subject (or that the target subject will not be changed) together with the identification information of the sub camera to be processed. The notification may be, for example, a message displayed on the display unit .

[0151] Furthermore, CPU 101 may transmit a notification to the outside. For example, CPU 101 can transmit a notification to overhead camera 300 via network I / F 105. When overhead camera 300 receives the notification via network I / F 305, it displays on an external display device connected to monitor output unit 312 that the change of the target subject has been rejected.

[0152] Furthermore, CPU 101 may transmit the subject detection result in recognition unit 121, identification information MAIN_SUBJECT of the target subject, identification information SUBJECT_ID of the tracked subject, and identification information of the sub-camera to be processed together with a notification to overhead camera 300.

[0153] Then, based on the subject detection result, CPU 301 of overhead camera 300 can output an image in which identification information of the sub camera to be processed and an index (e.g., a rectangular frame) indicating the detected subject are superimposed on a live view display image from monitor output unit 312. Furthermore, CPU 301 can visually differentiate the indexes of the target subject and the tracking subject from those of the other subjects among the detected subjects.

[0154] For example, if the target subject and the tracked subject are the same, the indicator is blue, and if they are different, the indicator for the target subject is green and the indicator for the tracked subject is red. Indicators for other subjects are not displayed or are other colors such as white. Note that the colors are merely examples, and visual elements other than color may be different. If a blue indicator is not displayed and green and red indicators are displayed, it can be seen that changing the tracked subject for the sub-camera being processed is rejected.

[0155] For a certain sub-camera, if the focus subject and tracked subject are both subject A, a blue indicator is displayed for subject A (Fig. 10(a)). Here, we will assume that no indicator is displayed for subjects that are neither the focus subject nor the tracked subject. In this state, if the focus subject is changed to subject B, and the sub-camera is not in the "streaming" state (or "previewing"), the new focus subject becomes the tracked subject. Therefore, the indicator for subject A disappears, and a blue indicator is displayed for subject B (Fig. 10(b-1)). However, if the sub-camera is in the "streaming" state (or "previewing"), the tracked subject does not change. Therefore, the indicator for subject A changes to red, and a green indicator is displayed for subject B (Fig. 10(b-2)).

[0156] CPU 101 may transmit the subject detection result, identification information MAIN_SUBJECT of the subject of interest, and identification information SUBJECT_ID of the subject being tracked for each sub-camera to overhead camera 300 regardless of changes in the subject of interest. This allows the above-mentioned display to be continuously performed on an external display device connected to overhead camera 300. Also, the sub-camera to be displayed may be changed in sequence, for example, at a fixed cycle. This allows tracking subjects and subjects of interest for each sub-camera to be grasped when there are multiple sub-cameras. Also, display unit 108 may perform the same display as overhead camera 300.

[0157] It is not always necessary to notify the refusal to change the tracking subject, and the user may be able to set whether or not to execute the notification through the user input I / F 106. When there are multiple sub-cameras 400, the setting may be possible for each camera.

[0158] In S505, the CPU 101 changes the identification information SUBJET_ID of the tracking subject of the sub camera to be processed, which is stored in the RAM 102, to the identification information MAIN_SUBJECT of the target subject after the change. That is, the CPU 101 changes the tracking subject of the sub camera to be processed.

[0159] In S506, the CPU 101 stores in the RAM 102 the identification information MAIN_SUBJECT of the target subject.

[0160] When the CPU 101 has executed the processes of S502 to S506 for each of the sub-cameras 400a to 400c to be controlled, the CPU 101 ends the process shown in the flowchart of Fig. 9. Next, the CPU 101 executes S105 in Fig. 4(a).

[0161] In addition, for a sub camera for which the change of the tracking subject has been rejected, if the state has changed to one in which the tracking subject can be changed the next time S104 in Fig. 4(a) is executed, the tracking subject may be changed to the target subject. Specifically, for a sub camera in which the tracking subject is different from the target subject and the tracking subject can be changed, the CPU 101 can change the tracking subject to the target subject.

[0162] In this way, when the state of the sub camera that is automatically controlled to track and photograph a specific subject is "distributing" (i.e., when selected by the switcher 1000), the shooting control device 100 of this embodiment does not change the tracking subject even if the specific subject is changed. This makes it possible to prevent the subject of the main line video from being changed to a subject not intended by the operator of the switcher 1000. Also, by not changing the tracking subject of a sub camera that is in the "previewing" state even if the specific subject is changed, it is possible to prevent the subject that is scheduled to be the next main line video from being changed to a subject not intended by the operator of the switcher 1000.

[0163] According to the shooting control device 100 of this embodiment, it is possible to easily perform shooting using multiple sub-cameras with a small number of personnel, while preventing unintentional changes to the subject of the main line image.

[0164] (Modification) In addition, in S104 of FIG. 4, the tracking subject determination unit 123 (CPU 101) may determine the tracking subject of the target sub camera 400 based on the role set for the sub camera 400.

[0165] The role that can be set for the sub camera 400 is a predefined way of using information obtained from the main camera for controlling the operation of the sub camera 400 when a main camera is provided. Here, as an example, the information from the main camera is used for controlling the tracking subject and zoom operation of the sub camera 400. The main camera is a camera different from the overhead camera 300 and may be provided separately from the sub camera, or one of the multiple sub cameras may be assigned to the main camera. When the main camera is provided separately from the sub camera, the main camera may be a camera whose shooting is not controlled by the shooting control device 100 (operated by the photographer). When the main camera is provided, for example, the user registers the identification information of the main camera and the role to be set for each sub camera in the shooting control application via the user input I / F 106.

[0166] The subject to be tracked by the main camera may be determined by the photography control device 100, or the image of the main camera may be input to the photography control device 100 in the same manner as the overhead camera 300, and the subject to be tracked by the main camera may be determined from the image. The CPU 101 can determine the subject to be tracked by the main camera, for example, using a known main subject determination method. For example, the CPU 101 can identify the subject area closest to the center of the image and specify the identification information of the subject to be tracked.

[0167] CPU 101 can also determine the zoom operation and its phase of the main camera by, for example, detecting a change in the angle of view of the image of the main camera. For example, the change in the angle of view may be detected from a change over time in the size or interval of the subject area. Alternatively, CPU 101 may detect the zoom operation and its phase of the main camera by periodically acquiring information on the angle of view from the main camera.

[0168] 14 shows an example of the types of roles that can be set for the sub camera 400 and the corresponding control contents. Here, it is assumed that any of "main follow", "main counter", "assist follow", and "assist counter" can be set as the role ROLE.

[0169] For the sub-camera 400 whose ROLE is "main follow", the photography control device 100 (CPU 101) sets the same tracking subject as the main camera, and when the main camera is zoomed, performs zoom control of the sub-camera 400 in the same phase. Here, "in phase" means that the zoom directions (telephoto direction or wide-angle direction) are the same, that is, the direction of the angle of view change is the same. On the other hand, "opposite phase" means that the zoom directions (telephoto direction or wide-angle direction) are opposite, that is, the direction of the angle of view change is opposite. Note that even if the zoom directions are in phase, the angle of view does not have to be the same as the main camera 500, and whether in phase or opposite phase, the degree of zoom change (such as the speed or rate of change) does not have to be the same as the main camera 500.

[0170] For the sub-camera with the role "main counter", the photography control device 100 (CPU 101) sets the same tracking subject as the main camera, and when the main camera is zoomed, the photography control device 100 (CPU 101) controls the sub-camera with this role to zoom down when the main camera is zoomed up. Note that zooming up refers to changing the zoom in the telephoto direction (telephoto end direction), and zooming down refers to changing the zoom in the wide-angle direction (wide end direction). When zoom control is performed by the image processing unit 406, zooming up refers to making the area cut out from the image smaller and the magnification ratio of the cut out area larger than before the area change. On the other hand, zooming down refers to making the area cut out from the image larger and the magnification ratio of the cut out area smaller than before the area change.

[0171] For a sub-camera with the role "assist follow", the photography control device 100 (CPU 101) sets a tracking subject different from that of the main camera, and when the main camera is zoomed, the photography control device 100 (CPU 101) performs zoom control of the sub-camera in the same phase.

[0172] For the sub-camera whose ROLE is "assist counter", the photography control device 100 (CPU 101) sets a tracking subject different from that of the main camera, and when the main camera is zoomed, the photography control device 100 (CPU 101) performs zoom control in the opposite phase on the sub-camera.

[0173] Here, for the sub-camera 400 with the role of "assist follow" and "assist counter", a subject existing on the left side of the subject other than the subject tracked by the main camera in the image is set as the subject tracked by the sub-camera 400. The subject tracked by the sub-camera 400 may be set according to other conditions. For example, a subject existing on the right side, upper side, or lower side of the subject other than the subject tracked by the main camera in the image may be set as the subject tracked by the sub-camera. Alternatively, a subject existing on the front or back of the subjects other than the subject tracked by the main camera may be set as the subject tracked by the sub-camera 400.

[0174] Also, only one of setting the subject to be tracked and zoom control may be performed, or another control item may be added.

[0175] <Second embodiment> Next, a second embodiment of the present invention will be described. In this embodiment, the imaging control device 100 may be similar to the first embodiment except for the operation of determining the subject to be tracked. Therefore, the description of the configuration and operation of the device common to the first embodiment will be omitted.

[0176] The tracking subject determination operation of the photography control device 100 in this embodiment will be described with reference to the flowchart shown in Fig. 11. The operation shown in Fig. 11 can be performed, for example, in place of S104 in Fig. 4(a). Moreover, steps common to the first embodiment are given the same reference numerals as in Fig. 9, and descriptions thereof will be omitted.

[0177] In S501, the CPU 101 determines whether or not the target object has been changed as a result of the process in S103, and if it is determined that the target object has been changed, the process proceeds to S602, and if not, the process proceeds to S601.

[0178] In S601, the CPU 101 executes a reservation process for changing the tracking subject. The details of this process will be described later. When the CPU 101 ends the reservation process for changing the tracking subject, the CPU 101 ends the process of the flowchart shown in FIG.

[0179] In S602, the CPU 101 determines whether or not an object change reservation is stored in the RAM 102. The object change reservation will be described later. If it is determined that an object change reservation is stored in the RAM 102, the CPU 101 executes S603, and if not, the CPU 101 executes S502.

[0180] In S603, the CPU 101 deletes the object change reservation stored in the RAM 102. This is because the target object has changed since the reservation was stored. After that, the CPU 101 executes S502.

[0181] Thereafter, the CPU 101 performs the same processing as in the first embodiment, except that when it is determined in S503 that the state of the sub camera to be processed is "distributing", S606 is executed instead of S504. In this embodiment, too, when the state is "previewing", processing may be performed in the same way as when the state is "distributing".

[0182] In S606, the CPU 101 stores in the RAM 102 the identification information MAIN_SUBJECT of the target subject determined in S103, together with the change reservation of the tracking subject.

[0183] Next, the details of the tracking subject change reservation process performed in S601 in Fig. 11 will be described using the flowchart shown in Fig. 12. In Fig. 12, steps common to the first embodiment are given the same reference numerals as in Fig. 9, and description thereof will be omitted. Note that the tracking subject change reservation process may be performed at any timing other than the timing of executing S104 in Fig. 3. The tracking subject change reservation process is performed for each sub camera.

[0184] In S701, CPU 101 determines whether or not a reservation for changing the tracking subject is stored in RAM 102. The subject change reservation will be described later. If it is determined that a reservation for changing the subject is stored in RAM 102, CPU 101 executes S502, and if not, changes the sub camera to be processed and executes S701. If there is no sub camera on which S701 has not been executed, CPU 101 ends the process shown in the flowchart in FIG.

[0185] In S502, the CPU 101 acquires status information of the sub camera to be processed. The status information may be acquired directly from the sub camera, or status information STREAMING may be acquired from the switcher 1000. When status information STREAMING is acquired from the switcher 1000, the status information of all the sub cameras is included, so S502 does not need to be executed for the remaining sub cameras.

[0186] S503, S505, and S506 are the same as those in the first embodiment. In S505, for the sub-camera for which the change of the tracking subject was rejected when the target subject was changed last time, the state has changed from "distributing" (it is no longer selected by the switcher 1000), so the tracking subject is changed to the latest target subject. After S506, the CPU 101 executes S706.

[0187] In S706, the CPU 101 deletes the subject change reservation stored in the RAM 102. Then, the CPU 101 changes the target sub camera and executes S701. If there is no sub camera on which S701 has not been executed, the CPU 101 ends the process shown in the flowchart in FIG.

[0188] According to this embodiment, in addition to the same effects as those of the first embodiment, for a sub-camera in which changing of the tracking subject has been rejected, if the tracking subject becomes changeable, the tracking subject is automatically changed, thereby realizing the effect of good usability.

[0189] <Third embodiment> Next, a third embodiment of the present invention will be described. This embodiment relates to the angle of view control of the sub camera 400 by the photography control device 100. The photography control device 100 can change the angle of view by transmitting a control command to the sub camera 400. For example, when an instruction to change the angle of view of the sub camera 400 is received from an external device (not shown) on the communication network 600, there is a risk that the angle of view may be changed unintentionally by the operator of the switcher 1000.

[0190] The imaging control device 100 in this embodiment prevents the angle of view of the sub-camera capturing main line video from being changed unintentionally by the operator of the switcher 1000.

[0191] When the CPU 101 of the photography control device 100 receives a zoom command via the network I / F 105, it executes the process shown in the flowchart of FIG.

[0192] In S801, the CPU 101 stores the received zoom command in the RAM 102. Here, it is assumed that the zoom command includes identification information and zoom position information of the target sub camera. Then, the CPU 101 acquires the zoom position information from the zoom command.

[0193] The zoom position can change between the wide-angle end and the telephoto end to change the angle of view. For example, the zoom position information indicates the number of steps of a stepping motor that drives a zoom lens, and the zoom position can change between "0" and "300" steps. Here, the zoom position information may be, for example, the focal length (mm) of a photographing optical system corresponding to a 35 mm full-size image sensor, and in that case, the zoom value, which is the zoom position information on the telephoto side, is larger than the zoom value, which is the zoom position information on the wide-angle side. Note that, instead of the zoom position information that uniquely indicates the angle of view, a zoom command may include an instruction to zoom up or down. The zoom up or down instruction is an instruction to change the angle of view by a certain amount with respect to the current angle of view. The certain amount is determined in advance based on, for example, a zoom drive unit in a drive unit or a magnification control unit of a digital zoom.

[0194] In S803, the CPU 101 acquires the status information STREAMING from the switcher 1000, and stores it in the RAM 102. Note that the status information may be acquired from the sub-camera 400 individually.

[0195] In S804, CPU 101 refers to the status information STREAMING stored in RAM 102, and acquires the status of sub camera 400 specified by the zoom command. Then, CPU 101 determines whether the status is "streaming". If CPU 101 determines that the status is "streaming", it executes S805, and if not, it executes S807.

[0196] In S805, the CPU 101 reads from the RAM 102 a setting value (permission setting value) indicating whether or not a change in the angle of view of the sub camera capturing the main line video (video being output by the switcher 1000) (or a change in the angle of view of the main line video) is permitted. It is assumed that the permission setting value is set in advance.

[0197] When a zoom command is received, the permitted setting value may be set by transmitting setting screen data to the switcher 1000 and allowing an operator of the switcher 1000 to set the permitted setting value. When the CPU 101 receives the permitted setting value from the switcher 1000 via the network I / F 105, the CPU 101 stores the permitted setting value in the RAM 102.

[0198] CPU 101 refers to the permission setting value, and if it is determined that the change in the angle of view of the main line video is permitted, proceeds to step S807, and if not, ends the processing shown in the flowchart of Fig. 13. In this way, if it is not determined that the change in the angle of view of the main line video is permitted, the zoom command is not executed. In this case, CPU 101 may notify the external device that transmitted the zoom command that zooming is not permitted.

[0199] In S807, the CPU 101 generates a control command for the specified sub-camera 400 based on the zoom position information, and stores the control command in the RAM 102.

[0200] In S808 , the CPU 101 reads out the control command from the RAM 102 , and transmits it to the specified sub-camera 400 via the network I / F 105 .

[0201] The photography control device 100 of this embodiment can control whether or not to permit a change in the angle of view of the sub-camera capturing the main line video (or a change in the angle of view of the main line video). When the photography control device 100 receives an instruction to change the angle of view of the sub-camera capturing the main line video, the photography control device 100 executes the change in the angle of view only if the change in the angle of view is permitted. This makes it possible to prevent the angle of view of the main line video from being changed unintentionally by the operator of the switcher.

[0202] In addition to the change in the angle of view, parameters (such as the focal length and exposure conditions) that affect the image of the sub-camera 400 and are controllable by the shooting control device 100 can also be controlled in the same manner as the change in the angle of view. Also, the permitted setting value can be set for each parameter.

[0203] (Other embodiments) In the above embodiment, the shooting control device 100 is an independent device. However, other devices (for example, the switcher 1000 or the sub-camera 400) may have the same functions as the shooting control device 100.

[0204] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0205] The disclosure of the present embodiment includes the following imaging control device, imaging control method, imaging system, and program. (Item 1) A determination means for determining a specific subject within a shooting range as a tracking subject; a control means for controlling a photographing direction of a camera so as to track and photograph the tracking subject, the determining means does not change the tracking subject to be tracked by the camera when the image captured by the camera is selected as a specific image by an external selection device; 13. An imaging control device comprising: (Item 2) 2. The imaging control device according to item 1, wherein the specific image is an image output by the selection device. (Item 3) 2. The imaging control device according to item 1, wherein the specific image is an image output by the selection device or an image that is scheduled to be output next by the selection device. (Item 4) The imaging control device according to any one of items 1 to 3, characterized in that the determination means enables the camera to change the tracking subject to be tracked when the image captured by the camera is not selected as the specific image by the selection device. (Item 5) 5. The photographing control device according to claim 1, wherein the determination unit determines the tracking subject of the camera according to a tracking subject of a camera other than the camera that photographs the photographing range. (Item 6) 6. The photographing control device according to item 5, wherein the control means controls the angle of view of the camera in accordance with the angle of view of a camera other than the camera that photographs the photographing range. (Item 7) 7. The photographing control device according to item 6, characterized in that how the tracking subject and angle of view of the other camera are used to control the camera differs depending on the type of role set for the camera. (Item 8) The camera further includes an identification unit for identifying a subject within the shooting range from an image of the shooting range, The determining means determines, from among the identified subjects, a subject determined as a target subject as the tracking subject. 8. The imaging control device according to any one of items 1 to 7, (Item 9) 9. The photographing control device according to item 8, wherein the subject of interest is determined by a user of the photographing control device. (Item 10) 9. The photographing control device according to item 8, wherein the subject of interest is determined by the photographing control device. (Item 11) 11. The shooting control device according to any one of items 8 to 10, wherein the image captured of the shooting range is an image captured of the entire shooting range in a fixed direction and angle of view. (Item 12) 11. The image capturing device according to any one of items 8 to 10, wherein the image captured within the shooting range is captured by a camera other than the camera and whose image capturing is not controlled by the image capturing control device. (Item 13) 13. The photographing control device according to any one of items 8 to 12, characterized in that an image in which an index indicating the target subject and the tracking subject is superimposed on an image photographed within the photographing range is displayed on a display device. (Item 14) Item 14. The photographing control device according to item 13, characterized in that the indicator is made visually different depending on whether the target subject and the tracking subject are the same or different. (Item 15) 15. The photographing control device according to any one of items 8 to 14, characterized in that, when the target subject changes, if the tracking subject is not changed because the image captured by the camera has been selected as the specific image by the selection device, the determination means determines the latest target subject as the tracking subject after the image captured by the camera is no longer selected as the specific image by the selection device. (Item 16) 16. The shooting control device according to any one of items 1 to 15, characterized in that when an instruction for a change that affects the image of the camera is received from the outside, the control means does not execute the instruction unless the change is permitted if the image shot by the camera has been selected as the specific image by the selection device. (Item 17) 17. The image capture control device according to item 16, wherein the change that affects the image of the camera is a change in any one of the angle of view, the focal distance, and the exposure conditions of the camera. (Item 18) A photographing control method executed by a photographing control device, determining a specific subject within a shooting range as a tracking subject; and controlling a photographing direction of a camera so as to track and photograph the tracking subject; The determining step includes not changing the tracking subject to be tracked by the camera when the image captured by the camera is selected as a specific image by an external selection device. 23. A photographing control method comprising: (Item 19) An imaging control device according to any one of items 1 to 17, One or more cameras that the photography control device controls to take pictures; The selection device; An imaging system comprising: the imaging control device, the camera, and a communication network that communicatively connects the imaging control device, the camera, and the selection device. (Item 20) A program for causing a computer to function as each of the means possessed by the imaging control device according to any one of items 1 to 17.

[0206] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0207] 100: control device, 300: overhead camera, 400: sub-camera, 1000: switcher, 101: CPU, 102: RAM, 103: ROM, 104: inference unit, 105: network I / F, 106: user input I / F

Claims

1. A determination means for determining a specific subject within a shooting range as a tracking subject; a control means for controlling a photographing direction of a camera so as to track and photograph the tracking subject, the determining means does not change the tracking subject to be tracked by the camera when the image captured by the camera is selected as a specific image by an external selection device; 13. An imaging control device comprising:

2. 2. The imaging control device according to claim 1, wherein the specific image is an image output by the selection device.

3. 2. The imaging control device according to claim 1, wherein the specific image is an image output by the selection device or an image that is scheduled to be output next by the selection device.

4. 2. The photographing control device according to claim 1, wherein the determination means enables the camera to change the tracking subject to be tracked when the image photographed by the camera is not selected as the specific image by the selection device.

5. 2 . The photographing control device according to claim 1 , wherein the determining unit determines the subject to be tracked of the camera in accordance with a subject to be tracked of a camera other than the camera that photographs the photographing range.

6. 6. The photographing control device according to claim 5, wherein the control means controls the angle of view of the camera in accordance with the angle of view of a camera other than the camera that photographs the photographing range.

7. 7. The photographing control device according to claim 6, wherein how the tracking subject and the angle of view of the other camera are used for controlling the other camera differs depending on the type of role set for the camera.

8. The camera further includes an identification unit for identifying a subject within the shooting range from an image of the shooting range, The determining means determines, from among the identified subjects, a subject determined as a target subject as the tracking subject.

2. The imaging control device according to claim 1 .

9. 9. The photographing control device according to claim 8, wherein the subject of interest is determined by a user of the photographing control device.

10. 9. The photographing control device according to claim 8, wherein the subject of interest is determined by the photographing control device.

11. 9. The image capture control device according to claim 8, wherein the image captured within the imaging range is an image captured within the entire imaging range in a fixed direction and angle of view.

12. 9. The image capture control device according to claim 8, wherein the image captured within the capture range is an image captured by a camera other than the camera, the image capture of which is not controlled by the image capture control device.

13. 9. The photographing control device according to claim 8, wherein an image obtained by photographing the photographing range and an index indicating the target subject and the tracking subject are superimposed on a display device.

14. 14. The photographing control device according to claim 13, wherein the indicator is made visually different depending on whether the target subject and the tracking subject are the same or different.

15. The photographing control device according to claim 8, characterized in that, when the target subject changes and the tracking subject is not changed because the image captured by the camera has been selected as the specific image by the selection device, the determination means determines the latest target subject as the tracking subject after the image captured by the camera is no longer selected as the specific image by the selection device.

16. The shooting control device according to claim 1, characterized in that when an instruction for a change that affects the image of the camera is received from the outside, the control means does not execute the instruction unless the change is permitted when the image captured by the camera has been selected as the specific image by the selection device.

17. 17. The image capture control device according to claim 16, wherein the change that affects the image captured by the camera is any one of a change in the angle of view, a focus distance, and an exposure condition of the camera.

18. A photographing control method executed by a photographing control device, determining a specific subject within a shooting range as a tracking subject; and controlling a photographing direction of a camera so as to track and photograph the tracking subject; The determining step includes not changing the tracking subject to be tracked by the camera when the image captured by the camera is selected as a specific image by an external selection device.

23. A photographing control method comprising:

19. The imaging control device according to any one of claims 1 to 17, One or more cameras that the photography control device controls to take pictures; The selection device; An imaging system comprising: the imaging control device, the camera, and a communication network that communicatively connects the imaging control device, the camera, and the selection device.

20. A program for causing a computer to function as each of the means included in the imaging control device according to any one of claims 1 to 17.

Images