Control device, learning device and control method

Abstract

To more preferably image a subject using one or more cameras.SOLUTION: A control device that controls one or more shooting units includes acquisition means that acquires the positions of a plurality of objects on the basis of an image taken by a shooting unit of a plurality of objects, and generation means that generates a control command for changing the posture of the shooting unit on the basis of at least the image, the positions of the plurality of objects, and the posture of the shooting unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for photographing a subject. [Background technology]

[0002] Automatic filming technologies have been proposed to enable low-cost filming and video distribution of sports and other events. Most of these technologies achieve automatic filming by acquiring the subject's position within the field where the event is taking place and controlling the filming camera based on the acquired subject position.

[0003] Patent Document 1 describes a photographing device that controls a photographing camera based on the subject position estimated by three-dimensional position estimation to achieve automatic photographing. Patent Document 2 describes a control device that acquires video for distribution by synthesizing a panoramic image covering the entire field from a multi-lens camera, and controlling a virtual camera based on the position of an object detected in the panoramic image to cut out the image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 3615867 [Patent Document 2] U.S. Patent No. 10,262,692 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-mentioned conventional technology requires the entire field to be targeted to acquire the subject position, which results in redundant image areas and the need for many unused cameras, which is an obstacle to building a low-cost automatic photography system.

[0006] The present invention has been made in view of the above problems, and aims to provide a technique that enables a subject to be more suitably photographed using one or more cameras. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the control device according to the present invention has the following configuration: Configured to allow at least one of the following controls: Pan, Tilt, and Zoom (PTZ) The control device that controls the imaging unit includes: an acquisition means for acquiring positions of a plurality of objects from the images captured by the imaging unit; Driving at least one of the PTZs to track a tracking target object included in the plurality of objects. generating means for generating a control command; With death, The generation means generates the control command such that an integrated score obtained from a first score calculated based on at least a position of the tracking target object in the image and a second score calculated based on detection reliability in detecting the plurality of objects increases. .

[0008] Alternatively, a learning device according to the present invention may have the following configuration: That is, a learning device that learns weight parameters of a learning model that outputs control information for changing the posture of an imaging unit, an output means for outputting the control information by inputting the postures of one or more of the image capturing units capturing images of the plurality of objects and the images of the plurality of objects into the learning model to which given weight parameters have been applied; a learning means configured to learn the weighting parameters for outputting the control information that maximizes a reward, the reward being a score estimated based at least on the attitude of the imaging unit controlled by the control information and images of the plurality of objects captured at the attitude of the imaging unit; and It has. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a technique that enables more suitable imaging of a subject using one or more cameras. [Brief explanation of the drawings]

[0010] [Figure 1] 1A and 1B are diagrams illustrating examples of camera and person placement and examples of viewpoint images. [Figure 2]FIG. 2 is a block diagram showing an example of a functional configuration of a control device. [Figure 3] 4 is a flowchart showing a process executed by a control device or a learning device. [Figure 4] FIG. 10 is a diagram illustrating an example of the configuration of a learning model. [Figure 5] FIG. 2 is a block diagram illustrating an example of a functional configuration of a learning device. [Figure 6] FIG. 2 is a diagram illustrating the relationship between a physical camera and a virtual camera. [Figure 7] FIG. 2 is a block diagram showing an example of a functional configuration of a control device. [Figure 8] 4 is a flowchart showing a process executed by a control device. [Figure 9] FIG. 10 is a diagram illustrating an example of the configuration of a learning model. [Figure 10] FIG. 10 is a diagram illustrating an example of camera placement. [Figure 11] FIG. 2 is a block diagram showing an example of a functional configuration of a control device. [Figure 12] FIG. 10 is a diagram illustrating an angle score. [Figure 13] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a control device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] (First embodiment) As a first embodiment of a control device and a learning device according to the present invention, a system that controls a pan-tilt-zoom (PTZ) video camera and automatically captures an image of a subject so as to satisfy set shooting conditions will be described below as an example.

[0013] <System configuration> In the first embodiment, as a specific example, a plurality of fixed PTZ video cameras are installed around a futsal pitch, which is a ball game, and these cameras are controlled to simultaneously / concurrently track the ball and people on the pitch and track and capture the ball or people closest to the ball.

[0014] Fig. 1(a) is a diagram showing an example of the arrangement of cameras and people in the first embodiment, and Fig. 1(b) is a diagram showing an example of a viewpoint image in the arrangement of Fig. 1(a).

[0015] Camera arrangement 100 shows the arrangement of cameras 101 to 104. Cameras 101 to 104 are fixedly installed PTZ cameras whose pan, tilt, and zoom can be driven by external commands. Origin 105 is the origin of three-dimensional space. Arrows 106, 107, and 108 respectively represent the X-axis, Y-axis, and Z-axis based on origin 105 in three-dimensional coordinates (world coordinates). The plane formed by the X-axis and Z-axis corresponds to the ground, and the Y-axis is the vertical direction. Rectangle 109 is the perimeter of the futsal pitch (the rectangle formed by the touchlines and goal lines). Each camera is fixed to a wall at a certain height above the ground and is installed so as to capture objects on the pitch.

[0016] Person arrangement 110 shows the arrangement of people (players) and the ball at a certain time. Rectangle 111 corresponds to the same pitch as rectangle 109. A center mark 112 and a halfway line 113 are positioned on the pitch. Team A players A0 to A4 and team B players B0 to B4 are shown separately as players. A ball S0 is also shown.

[0017] Images 120, 130, 140, and 150 show images of the person and ball arrangement shown in person arrangement 110 captured by cameras 101, 102, 103, and 104 shown in camera arrangement 100, respectively. Pitches 121, 131, 141, and 151 are the pitches shown by rectangle 109. As these images from each viewpoint show, the size of one or more objects that are hidden or visible changes depending on the positions of the camera and multiple objects. The size of the object that is visible can also be changed by changing the zoom of the camera. Furthermore, the field of view (FOV) can be changed by changing the camera's posture by driving the pan and tilt.

[0018] <Device configuration> FIG. 13 is a block diagram showing an example of the hardware configuration of a control device and a learning device. The central processing unit (CPU) H101 uses RAM H103 as a work memory to read and execute the OS and other programs stored in ROM H102 and storage device H104, controls each component connected to the system bus H109, ​​and performs calculations and logical judgments for various processes. The processes executed by the CPU H101 include the information processing of the embodiment. The storage device H104 is a hard disk drive, an external storage device, or the like, and stores programs and various data related to the information processing of the embodiment. The input unit H105 is an input device such as an imaging device such as a camera, buttons for inputting user instructions, a keyboard, or a touch panel. The storage device H104 is connected to the system bus H109 via an interface such as SATA, and the input unit H105 is connected to the system bus H109 via a serial bus such as USB, but details of these are omitted. The communication I / F H106 communicates with external devices via wireless communication. The display unit H107 is a display. The sensor H108 is an image sensor or a distance sensor.

[0019] FIG. 2(a) is a diagram showing the functional configuration of the device (control device 1000) during runtime. Runtime refers to the state in which automatic shooting (tracking shooting and tracking) operations are performed by cameras 101-104. FIG. 2(b) is a diagram showing the functional configuration of the device (learning device 2000) during learning. Note that although the control device 1000 and learning device 2000 are shown as separate components, they may also be configured as an integrated unit.

[0020] The imaging unit 1001 corresponds to cameras 101-104, which are PTZ cameras equipped with a drive mechanism including a pan mechanism, a tilt mechanism, and a zoom mechanism. The imaging unit 1001 can realize PTZ operation by controlling the drive mechanism (PTZ mechanism). Furthermore, the cameras 101-104 also have a function to realize pseudo PTZ operation by cutting out an image of a predetermined area from an image (wide-angle image) captured by controlling the zoom magnification to a wider angle than a predetermined magnification. The control device and the PTZ cameras (imaging devices) may be separate devices. The control device controls the attitude of at least multiple PTZ cameras and instructs each camera to track and capture a subject.

[0021] The control device 1000 includes an imaging unit 1001, an object detection unit 1002, a position estimation unit 1003, a command generation unit 1004, and a setting unit 1005. Although each functional unit is illustrated as being one, it may be multiple. For example, the imaging unit 1001 corresponds to the cameras 101 to 104 in FIG. 1(a). The learning device 2000 includes a learning unit 2100, a command generation unit 2005, and a setting unit 2007. The learning unit 2100 includes a simulation unit 2001, an imaging unit 2002, an object detection unit 2003, an update unit 2009, and a score calculation unit 2006. Although each functional unit is illustrated as being one, it may be multiple. For example, the imaging unit 2002 is one of four simulation cameras corresponding to the positions of the cameras 101 to 104 in the learning unit 2100.

[0022] As will be described in detail later, the learning device 2000 performs reinforcement learning on the weight parameters of a learning model that outputs control information for changing the attitude of the imaging unit. The control device 1000 applies the given weight parameters obtained by learning to a neural network, and uses the neural network to generate a PTZ command, which is a control command, based on the estimated position of the object.

[0023] <Device Operation> <Runtime processing> FIG. 3(a) is a flowchart showing the process at runtime in the first embodiment.

[0024] In step S1001, the setting unit 1005 sets a target object to be tracked and photographed. For example, the setting unit 1005 operates a graphical user interface (GUI) (not shown) to input a target object to be tracked and photographed into the setting unit 1005. Alternatively, a target object of a pre-specified category may be detected by image recognition and set as a target to be photographed.

[0025] Here, when setting the target for tracking and shooting, one of a ball and a person is set as the target for tracking and shooting. Next, the target to be tracked is set. The detection targets of the object detection unit 1002 used in S1003, which will be described later, are the candidates, and in this case, the person's head and the ball. One or more of these are selected. Note that the target set as the target for tracking and shooting is also set as the tracking target. Alternatively, if there are players, a ball, and a referee on the pitch and the object detection unit 1002 can detect them separately, the players, the ball, and the referee become candidates for tracking.

[0026] Details of the object detection unit 1002 will be described in detail in S1003. An ID is assigned to each instance of a candidate for tracking photography and a tracking target, and the target for tracking photography and tracking set here can be identified using that ID.

[0027] In step S1002, the command generation unit 1004 drives the PTZ of the imaging unit 1001 (cameras 101 to 104) to set it to its initial state. The command generation unit 1004 also loads weighting parameters corresponding to the target to be tracked and the target to be tracked set in S1001 into the neural network used in S1003 and S1005, which will be described later. The initialization step is performed at the start of imaging and tracking, and is therefore usually performed at the start of a match.

[0028] Loop L1001 is a time-related loop, and it is assumed that one loop will be executed at a frame rate of approximately 30 to 60 FPS. However, one loop is determined by the total throughput of each step in the loop, so it may run slower depending on the runtime environment. In that case, shooting will be performed at the normal video rate (approximately 30 to 60 FPS), images will be sampled according to the loop frame rate, and each step in the loop will be executed. This loop will also be executed repeatedly until tracking shooting and tracking are completed.

[0029] In step S1003, the object detection unit 1002 resizes the image captured by the image capture unit 1001 to an appropriate size and executes detection processing using an object detector. The object detector is assumed to be an object detector using a convolutional network disclosed in "Joseph Redmon, Ali, Farhadi, "YOLO9000: Better, Faster, Stronger", CVPR2017." This object detector outputs the bounding boxes (coordinates and width) and confidence scores of objects in a pre-trained detection target category, for each object present in the input image.

[0030] Here, the detection target categories are a ball and a person (head or whole body). If there is a difference in appearance between players and referees due to differences in clothing, etc., the learning device 2000 may be configured to pre-learn players and referees as separate categories, and the control device 1000 may be configured to detect each of them using weight parameters obtained by learning.

[0031] In step S1004, the position estimation unit 1003 performs 3D position estimation using a known technique based on the object detection results of each camera acquired in S1003 and the camera parameters of each camera. For example, it is possible to use the multi-view stereo technique described in "Matsushita, Furukawa, Kawasaki, Furukawa, Sagawa, Yagi, Saito, Computer Vision Frontier Guide 5 (2012)."

[0032] Furthermore, 3D tracking is performed by assigning an ID based on the consistency between the 3D position estimation results of the previous frame and the 3D position estimation results of the current frame. For ID assignment, a known combinatorial optimization method for solving assignment problems, such as the Hungarian algorithm, which uses Euclidean distance as a cost and achieves assignment that minimizes the sum of costs, is used. This ID is further assigned to the ID of the tracking target set in S1004. As a result, the detection result obtained in S1003 also has the ID of the tracking target assigned.

[0033] In step S1005, the command generating unit 1004 applies the weight parameters learned by reinforcement learning to the neural network, and generates a PTZ command using the neural network.

[0034] Figure 4 shows the configuration of a four-camera input actor-critic network 400 (learning model). Network 400 receives inputs of an image 401, tracking photography and the three-dimensional position 402 of the tracking target (ball / player), and camera posture and pan-tilt-zoom state 403. Network 400 outputs a policy output 404 and a value output 405.

[0035] Assume that image 401 is a three-channel image as shown in input image 410 in FIG. 4. Specifically, it is bounding box (BB) images 411 and 412 of the object detection result in S1003, and a background image 413. Note that BB image 411 is a BB image of the tracking and shooting target, and BB image 412 is a BB image of another tracking target. Note that the BB image is visualized using brightness values ​​as detection reliability, and background image 413 is a binarized image. Furthermore, images from four viewpoints taken by four cameras 101 to 104 are input to network 400 as image 401. For state 403, the attitudes of the four cameras (pan, tilt, and zoom control states) are also input.

[0036] The strategy output 404 corresponds to a command for the PTZ camera. Generally, a PTZ camera can set multiple speed levels for PTZ drive. In the simplest case, when controlling the PTZ, it is possible to set one speed level and use three discrete commands of {-1, 0, 1}. However, in this case, the PTZ movement is not smooth and is not suitable for images used for viewing or object detection. For this reason, here, a command with a continuous value of [-1, 1] is generated as the strategy output 404 to enable fine movements. This continuous command value is quantized to a predetermined gradation, and the numerical value is input to the PTZ camera to set multiple speed levels.

[0037] PTZ cameras also have a predetermined driving range for pan, tilt, and zoom. A PTZ command does not change the state within the predetermined driving range of the PTZ camera if the value is 0, changes the driving speed state by one step in the positive direction if the value is greater than 0, or in the negative direction if the value is less than 0.

[0038] 4, the policy output 404 is shown as outputting commands for four cameras, but it may be configured to output commands for one camera. In other words, a multi-agent configuration may be used in which actor-critic networks are prepared for the number of cameras.

[0039] The value output 405 outputs a value corresponding to the command in the policy output 404, but this output is not used in the runtime processing. Note that although the embodiment using a network reinforced by the Actor-Critic method is described here, other reinforcement learning methods such as Q-learning may also be applied.

[0040] In step S1006, the command generating unit 1004 transmits the PTZ command generated in S1005 to the imaging unit 1001 (cameras 101 to 104) and executes PTZ driving of the cameras.

[0041] As mentioned above, PTZ driving is performed within the PTZ camera's specified driving range. For example, the pan angle range is ±170°, the tilt angle range is -90° to +10° (horizontal direction is 0°), and the zoom (focal length) is 4.7mm to 94mm. For example, if the camera's pan state is +170°, which is the limit of this driving range, it will not drive any further in the positive direction, even if the next pan command is +X (≧0).

[0042] <Processing during learning> Figures 3(b) and 3(c) are flowcharts showing the learning process in the first embodiment. During learning, a virtual environment is constructed in which the cameras, pitch, people, and ball shown in Figure 1 are placed, and a futsal match is played in the virtual environment. Camera control is performed for the match, and the control results are scored and given to the network as a reward, thereby performing reinforcement learning for the network. Preferably, weight parameters are learned to output camera control that maximizes the reward (score). At this time, one futsal match being learned is called an episode.

[0043] In step S2001, the setting unit 2007 sets the target to be tracked and photographed. Learning is performed for all targets for tracking and photographing that can be selected in the processing at runtime. Therefore, in this step, if there are multiple options for the tracking target in the runtime processing, the options are set in order, and processing is performed multiple times thereafter.

[0044] Additionally, in this step, parameters for determining the behavior of tracking and shooting, which will be used in S3005 described later, may be set. The parameters include, for example, a parameter for determining the distance from the center of the image of the target to be tracked and shot, and a range of the apparent size of the target to be tracked and shot.

[0045] In step S2002, the setting unit 2007 performs initialization processing. Here, initialization of each module of the learning unit 2100 is performed. In addition, in the reinforcement learning of the actor-critic network used to generate PTZ commands, multiple episodes are learned in synchronization. In this step, multiple simulation environments used in this process are instantiated.

[0046] Loop L2001 is a loop related to episodes. As described above, this loop actually executes multiple episodes synchronously, so that learning processes related to multiple episodes are executed in parallel in one loop.

[0047] Loop L2002 is a time loop that is executed repeatedly from the start to the end of a match to be learned as one episode.

[0048] In step S2003, the learning unit 2100 updates the simulation environment. Figure 3(c) shows a subroutine in the environment update step (S2003). The subroutine includes the processes of S3001, S3002, S3003, S3004, and S3005.

[0049] In step S3001, the command generation unit 2005 generates a drive command for the imaging unit 2002. Note that the learning unit 2100 also has four cameras 101 to 104 corresponding to FIG. 1(a). Here, the PTZ command acquired in S2005 of the preceding loop is input to the camera in the virtual environment to drive the pan, tilt, and zoom of the camera. Note that in the first loop, zero is input, and the PTZ is not driven.

[0050] In step S3002, the simulation unit 2001 predicts and updates the movements of the people and ball in the virtual space. In the learning process, the people and ball shown on the pitch 111 in FIG. 1(b) are placed in the virtual space, and a futsal simulation is performed over a time period corresponding to the video frame rate (approximately 60 FPS). In this step, the state of the next game is calculated over a one-time period, and the state of each instance is updated.

[0051] In step S3003, the simulation unit 2001 performs an object detection simulation based on the state of the object detection target in virtual space and the state of the camera updated in S3002. First, the apparent size, occlusion rate, and speed of the object on the pixel coordinates of each camera are calculated based on the position and speed of the object detection target (a ball, a person's head, etc.) that have been set in advance, and the camera parameters. Next, the reliability is estimated using a multiple regression model with the apparent size, occlusion rate, and speed as explanatory variables and the reliability of object detection as a target variable. Here, the occlusion rate is defined as the ratio of the area of ​​the object occluded by a foreground object.

[0052] This multiple regression model is prepared by learning in advance using the above explanatory variables and objective variables, using a calibrated camera and actual object detection data of people and balls whose 3D positions have been obtained. This multiple regression model may also be a polynomial model of an appropriate degree.

[0053] Furthermore, ranges for the apparent size, occlusion rate, and speed of an object may be set based on examples of successful and unsuccessful object detection in real data to determine whether or not the object can be detected. In other words, if the size or speed of the object exceeds a predetermined range (for example, if it is too small or too fast), the object is deemed undetectable, and the reliability is set to 0. Furthermore, a probabilistic model for erroneous object detection (non-detection, over-detection, misaligned detection position) may also be created, and false detection (noise) may also be simulated. In this case, a random-number-driven probabilistic model simulates the occurrence of non-detection and over-detection, and the random-number-driven probabilistic model is also used to provide fluctuations in the reliability and detection position, thereby simulating false detections that may actually occur.

[0054] Here, we have explained a method for simulating reliability using a multiple regression model trained using actual object detection reliability data, but reliability can also be calculated using methods that do not use a multiple regression model. Generally, detection reliability is positively correlated with the apparent size of the object, and negatively correlated with the occlusion rate and speed. In other words, the larger the object, the higher the reliability, and information that shows the object is not occluded and is stationary is also highly reliable. It is also possible to create an appropriate function that can express these properties, and use that function to calculate and use a value corresponding to reliability from the apparent size, occlusion rate, and speed of the object.

[0055] In step S3004, the object detection unit 2003 estimates the 3D position of the detection target using the object detection results simulated in S3003. In this process, 3D tracking is performed on the object detection simulation results using the same method as described in S1004 of the runtime processing.

[0056] In step S3005, the score calculation unit 2006 calculates the scores for tracking and photographing and tracking based on the state of the tracking and photographing target and the state of the camera.

[0057] Tracking score (first criterion) In tracking photography, the standard is that the target to be tracked and photographed should be positioned as close to the center of the captured image as possible within the range where object detection is possible. Therefore, if the target is within the camera's FOV, the score for that camera is calculated by raising the cosine similarity between the vector from the camera center to the target in 3D space and the vector from the camera center to the center of the screen to an appropriate power (Equation (1)).

[0058]

number

[0059] Here, a is the vector from the center of the camera to the tracked target, b is the vector from the center of the camera to the center of the screen, and l is the degree. The cosine similarity has a wide range near its maximum value, and as it is, the score does not change much whether the target is in the center of the screen or on the periphery. Therefore, it is a good idea to raise it to a power of about 4 to create a function with a steeper slope near the maximum value. In this case, if the cosine similarity is zero or less, it is set to 0. k is the camera number.

[0060] Furthermore, a score related to the size of the appearance may be multiplied. For example, if the range of the size of the appearance is set in S2001, the value is close to 1 if the size of the appearance is within the range, and close to 0 if it is outside the range. The score is a value normalized to [0, 1], and r1 (k) Just multiply it by .

[0061] Furthermore, if the target exists outside the camera's FOV, it cannot be detected or photographed, so it is set to 0 (equation (2)).

[0062]

number

[0063] Finally, the maximum score of all cameras is taken and used as the score for tracking photography (Equation (3)).

[0064]

number

[0065] Tracking score (secondary criteria) In the first embodiment, three-dimensional positions are estimated using multi-view stereo, and three-dimensional tracking is performed. Therefore, it is desirable that all tracking targets can be detected by two or more cameras at all times. Furthermore, to improve detection accuracy, it is desirable that detection be performed with as high a detection reliability as possible. Therefore, in tracking, all targets are detected by two or more of K cameras, and the criterion is that the sum of the maximum and second-highest detection reliability is the better. Furthermore, it is desirable to calculate a score based on criteria including at least one of the occurrence rate of occlusion, the size of the target in the captured image, and the movement speed of the target.

[0066] This can be expressed as pseudo program code as follows: Here, the obtained score r2 is the tracking score.

[0067] Set N = number of targets Decleare r2 For n = 1 to N do Decleare accumulated_score Set num_detect = number of cameras detected target n If num_detect ≥ 2 then Set sorted_scores = sorted detection confidence array of target n Decleare value For index = 1 to 2 do value = value + sorted_scores[index] End do accumulated_score = value Else accumulated_score = 0 End If r2= r2+ accumulated_score End do r2= r2 / N Here, num_detect is the number of cameras that detected object n, and sorted_scores is an array of detection confidence scores sorted in descending order for object n (the number of elements is equal to the number of cameras).

[0068] Finally, the tracking shooting score and the tracking score are averaged to obtain a score (integrated score) determined when both tracking shooting and tracking are performed (Equation (4)).

[0069]

number

[0070] In step S2004, the command generation unit 2005 obtains input information for the Actor-Critic Network (FIG. 4(a)) used to generate PTZ commands from the simulation environment updated in S2003. As already described in the description of the processing at runtime, the Actor-Critic Network used in this embodiment includes the images of each camera, the three-dimensional position of the target, and the posture and pan-tilt-zoom state of each camera. Therefore, the relevant information is obtained from the updated simulation environment and input to the network. Here, the images of each camera are three-channel images, including two channels that visualize the target to be tracked and the BB and reliability of the tracking target, and one channel for a background image, as in the input image 410 shown in FIG. 4(b) described in the description of the processing at runtime.

[0071] In step S2005, the command generation unit 2005 executes forward propagation processing of the actor-critic network, and acquires the PTZ command for each camera from the strategy output of the actor-critic network.

[0072] In step S2006, the update unit 2009 updates the network. First, at each time instant of one episode executed in loop L1001, the output value of the value output of the Actor-Critic Network, the output value of the policy output, and the overall score of Equation (4) are obtained after forward propagation processing. Then, in accordance with the Actor-Critic Network learning method disclosed in the following reference 1, etc., the gain and value loss are calculated from the value output and the reward (integrated score), the policy loss is calculated from the policy output, and the network is updated based on both losses. (Reference 1: Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, "Asynchronous Methods for Deep Reinforcement Learning", ICML2016)

[0073] First, we calculate the loss of the value output using a value (Advantage) that takes into account the actions up to any k steps ahead, according to the method disclosed in Reference 1. The loss used in learning the value output is shown in Equation (5).

[0074]

number

[0075] where s t is the state at time t, and a t is the policy (action) at time t, π θ (a t |s t ) is the policy output at the state of the network θ of the policy output, and A(s t ,a t ) is the Advantage. The Advantage considering the k steps ahead is calculated by Equation (6).

[0076]

number

[0077] where γ is the discount rate, r t is the reward at time t, V φ (s t ) is the value output in the state. In this embodiment, the commonly used k=2 is used, and Advantage is set to take into account actions up to two steps ahead. In addition, for the loss of the policy function, Equation (7) is used, which adds an entropy term to mitigate falling into a local solution.

[0078]

number

[0079] where β is an appropriate coefficient, H(π t (s t )) is entropy, R t is a value calculated using the following pseudocode:

[0080] Set R = 0 for terminal s t or R = V φ (s t ) for non-terminal For t = t-1 to 0 do R = r t + γR End do

[0081] The two losses calculated as above, the value output and the policy output, are weighted with appropriate weights to calculate the final loss. Based on this final loss, the update unit 2009 updates the network.

[0082] As described above, according to the first embodiment, weight parameters learned by reinforcement learning are applied to a neural network, and PTZ commands are generated using the neural network. This makes it possible to detect and track a set target with higher detection reliability while performing automatic tracking and shooting of the target.

[0083] (Variation) In the first embodiment, reinforcement learning was performed by simulating the movement and object detection of people and a ball based on a sports game in a virtual environment and calculating the tracking shooting and tracking scores. The simulation performed here was intended to use the object detection results of multiple dynamically moving objects to learn the control for tracking shooting and tracking, and is not a perfect simulation of the movement and object detection of people and a ball in a real environment.

[0084] Therefore, in order to compensate for this difference, it is conceivable to also carry out learning in the real environment and improve the robustness of runtime processing in the real environment. Note that the configuration and operation of the device at runtime are the same as those in the first embodiment, so a description thereof will be omitted.

[0085] <Device configuration> 5 is a diagram showing the functional configuration of a modified example during learning in a real environment. The learning device 3000 includes an update unit 3001, a PTZ camera 3002, an object detection unit 3003, a position estimation unit 3004, a command generation unit 3005, a score calculation unit 3006, and a setting unit 3007. The PTZ camera 3002 may be located outside the learning device 3000.

[0086] That is, the main difference is that the estimation results of the object movements and positions simulated by the simulation unit 2001 in the first embodiment are replaced with the results estimated from the video of the actual subject in a futsal match in the second embodiment. Note that the futsal match has the same camera arrangement, player and ball arrangement as shown in Fig. 1, but is a futsal match played in a real environment.

[0087] <Device Operation> <Processing during learning> The operation of the device during learning according to the modified example differs in that an image of an actual futsal match is used in S2003, but the other steps are the same as those in the first embodiment. Therefore, the following will describe each step included in S2003.

[0088] In step S3001, the command generation unit 3005 generates a control command to drive the pan, tilt, and zoom of the PTZ camera 3002. In this modification, the PTZ camera 3002 performs the same processing as the learning unit 2100 in driving the pan, tilt, and zoom of the imaging unit 2002, except that the PTZ camera 3002 is an actual PTZ camera.

[0089] In step S3002, the PTZ camera 3002 acquires time-series images of the futsal match. Note that since the futsal match is a futsal match that is actually played, the images are input based on the progression of real time.

[0090] In step S3003, the object detection unit 3003 performs object detection on the image acquired by the PTZ camera 3002. This process is the same as S1003 at runtime in the first embodiment.

[0091] In step S3004, the position estimation unit 3004 estimates the three-dimensional positions of the tracking and photographing target and the tracking target based on the object detection result of S3003 and the camera parameters of the PTZ camera 3002. This process is the same as S1004 at runtime in the first embodiment.

[0092] In step S3005, the score calculation unit 3006 calculates a score based on the state of the tracking target and the tracking object, and the PTZ camera 3002. This process is almost the same as S3005 during learning in the first embodiment, except that detection results from actual images, rather than simulations, and 3D position estimation results based on the detection results are used.

[0093] As described above, according to the modified example, learning is performed in a real environment, which makes it possible to further improve the robustness of runtime processing in the real environment.

[0094] (Second embodiment) In the first embodiment described above, one object is treated as the tracking and photography target, and multiple objects are assumed to be tracked. However, if multiple objects can be set as tracking and photography targets, automatic photography such as tracking each player playing a sport becomes possible, expanding the range of applications. Therefore, in the second embodiment, a configuration will be described in which all players and the ball are tracked and photographed, and all players and the ball are tracked, in relation to a futsal match.

[0095] <System configuration> The system of the second embodiment is almost the same as that of the first embodiment (FIG. 1). That is, four PTZ cameras (cameras 101-104) are arranged around a futsal pitch 109. Also, there is one ball S0 and a total of ten players A0-A4, B0-B4 on the pitch. In the following description, the four PTZ cameras (cameras 101-104) are referred to as physical PTZ cameras. Furthermore, one physical PTZ camera contains multiple virtual shooting units (virtual cameras).

[0096] 6 is a diagram illustrating the relationship between a physical PTZ camera and two virtual cameras. A physical PTZ camera 600 is represented by an optical center 601 and an image plane 602. Virtual cameras 603 and 604 contained within the physical PTZ camera 600 are also shown as examples. The optical axis of the virtual camera is parallel to the optical axis of the physical PTZ camera, and the image plane is, in principle, contained within the image plane of the physical PTZ camera.

[0097] In other words, a virtual camera is a camera that can move parallel to the image plane of a physical PTZ camera and can zoom, capturing a partial area within the image plane of the physical PTZ camera. Each physical PTZ camera contains virtual cameras equivalent to the number of tracking and capturing targets set in S4001, which will be described later. Each virtual camera can be controlled independently of the other virtual cameras.

[0098] <Device configuration> Fig. 7(a) is a diagram showing the functional configuration of the device at runtime in the second embodiment, and Fig. 7(b) is a diagram showing the functional configuration of the device at learning time in the second embodiment.

[0099] The control device 4000 has a physical PTZ camera 4001, a virtual camera 4002, an object detection unit 4003, a position estimation unit 4004, a command generation unit 4005, a camera selection unit 4006, and a setting unit 4007. The physical PTZ camera 4001 has a different name but is the same as the imaging unit 1001 of the first embodiment. The learning device 5000 has a simulation environment 5100, a position estimation unit 5005, a command generation unit 5006, and a score calculation unit 5007. The simulation environment 5100 has a simulation unit 5001, a physical PTZ camera 5002, a virtual camera 5003, and an object detection unit 5004. The physical PTZ camera 5002 has a different name but is the same as the imaging unit 2002 of the first embodiment.

[0100] <Device Operation> <Runtime processing> 8(a) is a flowchart showing the process at runtime in the second embodiment. The following mainly describes the parts that are different from the first embodiment (FIG. 3(a)).

[0101] In step S4001, the setting unit 4007 sets the target for tracking and photography, as in S1001. However, in this embodiment, it is assumed that multiple targets can be set as targets for tracking and photography. An ID is assigned to each instance of a candidate target for tracking and photography, and the target for tracking and photography set here can be identified using that ID.

[0102] In step S4002, the command generation unit 4005 drives the PTZ of the physical PTZ camera 4001 (cameras 101 to 104) to bring it to an initial state, as in S1002. The command generation unit 4005 also initializes the virtual camera 4002. In this process, virtual cameras are generated for each physical PTZ camera, the number of which is the number of tracking and shooting targets set in S4001 (M: M is an integer of 2 or more), and these are initialized. For example, if 11 instances (one ball and 10 players) are set as tracking and shooting targets, one camera has 11 virtual cameras, which are initialized here. The initialization of the virtual cameras is the same process as for the physical PTZ cameras.

[0103] Loop L4001 is a loop related to time, and is the same as L1001 in the first embodiment, so a description thereof will be omitted.

[0104] In step S4003, the object detection unit 4003 resizes the image acquired from the physical PTZ camera 4001 to an appropriate size, as in S1003, and executes detection processing.

[0105] In step S4004, the position estimation unit 4004 estimates the three-dimensional position of each target and assigns an ID, similar to S1004. This ID is further assigned to the ID of the target to be tracked and photographed set in S4001. As a result, the detection result acquired in S4003 also has the ID of the target to be tracked assigned.

[0106] In step S4005, the command generation unit 1004 estimates control commands for the physical PTZ camera 4001 and the virtual camera 4002. In this embodiment, the same actor-critic network 400 (FIG. 4) as in the first embodiment is used to estimate the PTZ command for the physical PTZ camera 4001, and the input and output are also the same as in the first embodiment. That is, a PTZ command is output from the strategy output 404. On the other hand, the actor-critic network 900 in FIG. 9 is used to estimate the control command for the virtual camera.

[0107] Fig. 9 is a diagram showing the configuration of an actor-critic network of a virtual camera. As described with reference to Fig. 6, tracking shooting in this embodiment is achieved by cutting out an image of the virtual camera from an image of the physical PTZ camera. Since the size of the area from which the image is cut out is the same as the number of pixels, in order to maintain image quality at a viewing level, the zoom of the virtual camera is limited so that the size of the area from which the image is cut out is limited to a certain size or larger.

[0108] In addition, the virtual camera can move within the image plane of the physical PTZ camera to which it belongs. The image plane in which the virtual camera moves is a space with the center of the image as the origin, the horizontal rightward direction as +u, and the vertical upward direction as +v. However, the image plane of the virtual camera never extends beyond the image plane of the physical PTZ camera, and the virtual camera has a zoom state above a certain level, so the (u,v) state never reaches the edge of the space.

[0109] The action space of the virtual camera is three-dimensional, i.e., {u, v, Z} dimensions, and the policy output 904 of the actor-critic network 900 is a control command (uvZ command) on this action space.

[0110] The network 900 receives as input an image 901, the three-dimensional position 902 of the target to be tracked and photographed assigned to the virtual camera, the u, v and zoom states of the virtual camera, and the attitude and pan, tilt and zoom states 403 of the physical PTZ camera that has the virtual camera.

[0111] Assume that image 901 is a three-channel image such as the input image 910 in Fig. 9. Specifically, first channel image 911 is an image of a bounding box assigned with the ID of the tracking and shooting target from the detection results of the image of the physical PTZ camera to which the virtual camera belongs. Also, second channel image 912 is an image of the boundary of the cutout area, and third channel image 913 is an image of the binarized background.

[0112] As described above, the policy output 904 is a control command for {u, v, Z}. This command also generates a command with a continuous value of [-1, 1], similar to the policy output 404 of the actor-critic network 400 in the first embodiment.

[0113] The value output 905 outputs a value corresponding to the command of the policy output 904, but this output is not used for runtime processing, as in the first embodiment. Note that although a form using a network reinforced by the Actor-Critic method is described here, other reinforcement learning methods such as Q-learning may also be applied.

[0114] In step S4006, the command generation unit 4005 transmits the PTZ command generated in S4005 to the corresponding physical PTZ camera 5002 (cameras 101 to 104) to perform PTZ driving of the camera. Furthermore, the generated control command is transmitted to the corresponding virtual camera to drive the virtual camera (uvZ driving). The PTZ driving is performed within a predetermined driving range of the PTZ camera, as in the first embodiment. Similarly, the uvZ driving of the virtual camera is also performed within a predetermined driving range.

[0115] In step S4007, the camera selection unit 4006 performs processing to select the most appropriate clipped image from among clipped images of virtual cameras with the same ID in the four physical PTZ cameras. The selection criteria may be the same as the tracking and shooting score described in the first embodiment. That is, by using scores related to cosine similarity and appearance size, the distance from the center of the screen and the appearance size can be scored, and the clipped image is selected based on that score. Here, the value estimated in S4004 is used as the three-dimensional position of the tracking and shooting target used to calculate the score. Furthermore, to prevent frequent screen switching, the virtual camera ID to be selected may be smoothed using processing such as moving average.

[0116] This process is performed for all tracking and shooting targets to obtain cropped images of all tracking and shooting targets. The cropped images are then resized and other post-processing performed before being viewed.

[0117] <Processing during learning> The learning process is almost the same as in the first embodiment, so the differences from the first embodiment will be mainly described with reference to the flowcharts in FIGS. 3(b) and 3(c).

[0118] In step S2001, the setting unit 5008 sets multiple targets for tracking and photography and sets the tracking target. An ID is assigned to each instance of the candidate tracking and photography targets, and the tracking and photography targets set here can be identified using the ID. Similar to the first embodiment, parameters related to the criteria for tracking and photography are set.

[0119] In step S2002, the setting unit 5008 initializes the virtual camera in addition to the same initialization as in the first embodiment. That is, the setting unit 5008 instantiates and initializes the virtual camera 5003 in the simulation environment 5100.

[0120] In step S2003, the simulation unit 5001 updates the simulation environment. In S3001, the physical PTZ camera 5002 is driven in PTZ, and the virtual camera 5003 is driven in uvZ. In S3004, the tracking and photography and the three-dimensional position of the tracking target are estimated, and an ID is assigned. Furthermore, the ID here is assumed to be associated with the tracking target and the ID of the tracking target set in S2001. In S3005, the tracking and photography and tracking scores are calculated, as in the first embodiment.

[0121] However, as described above, in this embodiment, a plurality of tracking and shooting targets are set, and therefore, the tracking and shooting score is calculated for each of the one or more virtual cameras 5003 included in the physical PTZ camera 5002, with respect to the shooting targets assigned to each virtual camera.

[0122] As in the first embodiment, the tracking and photography score is calculated by scoring the distance from the center of the screen using formula (1). Furthermore, if a range is set for the size of the appearance, the size relative to the cropped image can be used to calculate a score for the size. Furthermore, if the target is outside the FOV of the virtual camera, the tracking and photography score can be set to 0 (formula (2)), as in the first embodiment. The tracking and photography score for a certain target obtained here is used as a reward for the corresponding virtual camera in reinforcement learning (learning corresponding to the virtual camera) of network 900 in S2006, which will be described later.

[0123] In this embodiment, a tracking and photography score is calculated for each of the four cameras for each target being tracked and photographed. The maximum value of the four tracking and photography scores for a given target is obtained and used as the tracking and photography score for that target. The scores for all tracking and photography targets are then averaged to use the overall tracking and photography score. The tracking score is calculated in the same way as in the first embodiment.

[0124] Finally, we use Equation (4) to obtain an integrated score by averaging the overall tracking and shooting score and the tracking score. The integrated score is then used as a reward for reinforcement learning in the network 400 (learning corresponding to the physical PTZ camera).

[0125] In step S2006, the command generation unit 2005 updates the network 400 using the acquired integrated score as a reward, and also updates the network 900 using the tracking and shooting score for each virtual camera as a reward.

[0126] As described above, according to the second embodiment, multiple tracking and photography targets are set, which makes it possible to more effectively perform the two tasks of acquiring automatically captured video and tracking data for all of the set tracking and photography targets simultaneously / in parallel.

[0127] (Third embodiment) In the third embodiment, a form applied to free viewpoint video generation will be described. More specifically, a shooting system will be described that simultaneously / parallelly performs creation of a 3D shape model using visual hull, and shooting and tracking. That is, while in the first and second embodiments, a video based on captured images is output, in the third embodiment, a video including generated free viewpoint video is output.

[0128] In the volume intersection method, one instance is photographed by multiple cameras arranged around the pitch, and a 3D shape model of the subject is reconstructed using foreground masks from multiple viewpoints. The camera control using reinforcement learning described in this embodiment performs optimal control to photograph all instances with more pixels and from more than the set number of viewpoints. This enables efficient generation of 3D shape models. Furthermore, since the ball and players moving around the pitch are all photographed, the method of this embodiment efficiently controls a limited number of cameras, enabling the generation of uniform free-viewpoint video footage covering every corner of the pitch. As with the first and second embodiments, the following description focuses on the photography of futsal.

[0129] <System configuration> 10 is a diagram showing an example of camera arrangement in the third embodiment. A camera arrangement 1000 shows an arrangement of a plurality of cameras 1002. In the following, the number of cameras is assumed to be K.

[0130] The K cameras 1002 are fixedly installed PTZ cameras, and their pan, tilt, and zoom can be controlled by external commands. Rectangle 1001 represents the perimeter of a futsal pitch. Each camera is installed on the same plane, at approximately the same height from the ground. Each camera is also calibrated in its initial state, so that even if the PTZ is driven, its posture can be accurately estimated based on the amount of change in its internal state, a 9-axis sensor (accelerometer, gyro, and geomagnetic sensor), images, etc.

[0131] <Device configuration> 11 is a diagram showing the functional configuration of a device at runtime in the third embodiment. The control device 6000 has a PTZ camera 6001, an object detection unit 6002, a position estimation unit 6003, a command generation unit 6004, a shape model generation unit 6005, and a setting unit 6006. The functional configuration of the device at the time of learning is the same as that of the first embodiment (FIG. 2(b)).

[0132] <Device Operation> <Runtime processing> 8(a) is a flowchart showing the processing at runtime in the third embodiment. In steps S6002, L6001, L6002, S6003, S6004, S6006, and S6007, the same processing as in the first embodiment (FIG. 3(a)) is performed for the number of cameras K. The following mainly describes the parts that differ from the first embodiment.

[0133] In step S6001, the setting unit 6006 sets the target of tracking photography and tracking. In addition, when applying the volume intersection method, the lower limit K of the number of cameras that capture one instance is set. v As mentioned above, the total number of cameras is K, so here, 2≦K v Set an integer that satisfies K.

[0134] In step S6005, the shape model generation unit 6005 creates a foreground mask for the K images captured by the K PTZ cameras and sets a region label for the foreground region of each image. Then, the visual volume intersection method is applied to the foreground region of each image to generate a 3D shape model of the subject. Further, as post-processing, noise may be removed using the size of the 3D shape model and the tracking information obtained in S6004. Furthermore, IDs may be associated with each 3D shape model using the tracking information.

[0135] <Processing during learning> The processing during learning is the same as in the first embodiment except for S3005, in which the number of cameras becomes K. Below, the processing of S3005, which differs from the first embodiment, will be described with reference to the flowcharts in Figures 3(b) and (c).

[0136] In step S3005, the learning unit 2100 calculates three scores and finally averages them to obtain an integrated score. Here, the three scores are the tracking and photography score, the tracking score, and a score indicating the degree of suitability for model generation by the visual volume intersection method (hereinafter referred to as the model creation score). Note that the tracking and photography score and the tracking score are the same as in the first embodiment, and therefore description thereof will be omitted.

[0137] The model creation score is calculated by dividing the camera into K v There is a constraint on the number of units to be constrained to more than 100. The angle score and size score are calculated separately to obtain the final model creation score.

[0138] Angle score 12 is a diagram illustrating the angle score. Triangles 1201, 1202, and 1203 each represent a camera. A vertex 1204 of the triangle represents the optical center of the camera represented by triangle 1201, and a base 1205 represents the image plane of the camera represented by triangle 1201. A circle 1206 is an instance that is the target of 3D shape model generation.

[0139] Axes 1212 and 1213 are the x-axis and z-axis when the instance is the origin, and the plane parallel to the x-axis and z-axis is the ground. As mentioned above, each PTZ camera is on the same plane, so as shown in this figure, the angle score is calculated in two dimensions parallel to the ground.

[0140] Circle 1207 is the unit circle in the xz coordinate system, and dotted line 1208 is the line of sight from the optical center of the camera (vertex 1204) to the instance (circle 1206). Arrow 1209 is a unit vector pointing from the instance to the optical center. Similarly, arrows 1210 and 1211 are unit vectors pointing to the optical centers of the two cameras corresponding to triangles 1202 and 1203.

[0141] In free viewpoint video generation, a 3D shape model is generated using the volume intersection method, and the image is then used as the texture of the 3D shape model. Therefore, when shooting an instance from multiple viewpoints, it is desirable that the angle from the instance to each viewpoint be equal. In other words, it is desirable that the magnitude of the sum of the unit vectors 1209, 1210, and 1211 from the instance toward the optical center of each viewpoint be close to 0. Therefore, it can be expressed as in equation (8).

[0142]

number

[0143] where K v is the number of viewpoints, and v k is the unit vector from the instance to each camera, and |||| is the Euclidean distance. v When images are taken from more than two viewpoints, the sum of the absolute values ​​of the angle differences between the vectors is the largest, K v Take vectors v k This means that K v If a combination of vectors can be realized that gives a high score in equation (8), then v This is because having more vectors does not have a negative effect on generating a 3D shape model.

[0144] Size score The size score can be calculated as in equation (9): f(x k ) is a function that outputs a value close to 1 if the apparent size of the instance is within an appropriate range, and a value close to 0 if it is outside the range.

[0145]

number

[0146] where x k is v in equation (8). k That is, the magnitude score is calculated for each viewpoint as f(x k ) and taking the average, the score will be close to 1 if the view from each viewpoint is within an appropriate range.

[0147] In the end, the 3D model creation score for a given instance is determined by the number of viewpoints taken, N. v In the above case, the equation becomes as shown in Equation (10).

[0148]

number

[0149] On the other hand, if the number of viewpoints to be photographed is N v If it is less than 1 / 2, it becomes as shown in equation (11).

[0150]

number

[0151] Furthermore, the average value across all instances is taken as the final 3D model creation score (Equation (12)).

[0152]

number

[0153] Finally, the tracking photography score, tracking score, and 3D model creation score, which are calculated in the same manner as in the first embodiment, are averaged to obtain an integrated score.

[0154] As described above, according to the third embodiment, in addition to capturing and tracking, a 3D shape model is generated using the volume intersection method in parallel, which enables automatic control of multiple cameras to generate free viewpoint video more efficiently.

[0155] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0156] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0157] 1000 control device; 1001 photography unit; 1002 object detection unit; 1003 position estimation unit; 1004 command generation unit; 1005 setting unit; 2000 learning device; 2100 learning unit; 2005 command generation unit; 2007 setting unit

Claims

1. A control device for controlling an imaging unit configured to be capable of controlling at least one of pan, tilt, and zoom (PTZ), an acquisition means for acquiring positions of a plurality of objects from an image captured by the imaging unit; A generating means for generating a control command for driving at least one of the PTZs in order to track a tracking target object included in the plurality of objects; having The generation means generates the control command such that an integrated score obtained from a first score calculated based on at least a position of the tracking target object in the image and a second score calculated based on a detection reliability in detecting the plurality of objects is increased. A control device comprising:

2. the first score is calculated based on a distance between a position of the tracking target object in the image and a predetermined position in the image; The second score is calculated based on an average value of the detection reliability of the plurality of objects. The control device according to claim 1 .

3. The first score is calculated by a function that rapidly approaches zero as the distance between the position of the tracking target object and the predetermined position increases.

3. The control device according to claim 2.

4. The distance between the position of the tracking target object and the predetermined position is calculated by exponentiating the inner product of a vector from the camera center of the shooting unit to the center of the image and a vector from the camera center to the tracking target object.

4. The control device according to claim 2 or 3.

5. If the inner product is equal to or less than zero, the inner product is set to zero.

5. The control device according to claim 4.

6. The first score is further based on a value based on a size of the tracked object in the image.

6. The control device according to claim 1, wherein the control device is a control unit.

7. The detection reliability is calculated based on a regression model that is trained in advance to estimate at least one of the size, the occlusion area ratio, and the moving speed of each object in the image as an explanatory variable.

7. The control device according to claim 1, wherein the control device is a control unit.

8. The integrated score is an average of the first score and the second score.

7. The control device according to claim 1, wherein the control device is a control unit.

9. a model generating unit for generating a shape model of a subject based on the image; The generating means further calculates a third score based on a third criterion indicating a degree of suitability for generating a three-dimensional shape model, based on the image, the positions of the plurality of objects, and the orientation of the image capturing unit, and generates the control command such that the integrated score determined based on the first score, the second score, and the third score becomes relatively large.

9. A control device according to any one of claims 1 to 8.

10. the generating means generates, as the control command, control information for changing the attitude of the imaging unit, the control information being obtained by inputting the image and the positions of the plurality of objects into a learning model to which given weight parameters are applied; The given weight parameters are generated by performing reinforcement learning using the integrated score as a reward when the image, the positions of the plurality of objects, and the orientation of the image capture unit are input to the learning model.

9. A control device according to any one of claims 1 to 8.

11. The imaging unit captures an image of a field where a ball game is being played, and the tracking target object includes a ball and / or a player closest to the ball in three-dimensional position.

11. A control device according to any one of the preceding claims.

12. The number of tracking target objects is M (M is an integer equal to or greater than 2), M virtual imaging units each of which is an imaging target for a partial area of ​​the image, The M virtual imagers are configured to track the M target objects.

12. The control device according to any one of the preceding claims.

13. The acquisition means acquires positions of the plurality of objects based on an image captured by at least one of the plurality of image capturing units, The generating means generates a control command for changing a posture of at least one of the plurality of image capturing units based on a score calculated from the positions of the plurality of objects acquired by the acquiring means.

13. A control device according to any one of the preceding claims.

14. A control method performed by a control device that controls an imaging unit configured to be capable of controlling at least one of pan, tilt, and zoom (PTZ), comprising: an acquisition step of acquiring positions of a plurality of objects from an image captured by the imaging unit; a generating step of generating a control command for driving at least one of the PTZs to track a tracking target object included in the plurality of objects; having In the generating step, the control command is generated so as to increase an integrated score obtained from a first score calculated based on at least a position of the tracking target object in the image and a second score calculated based on a detection reliability in detecting the plurality of objects. A control method comprising:

15. The first score is calculated based on a distance between a position of the tracking target object in the image and a predetermined position in the image; The second score is calculated based on an average value of the detection reliability of the plurality of objects.

15. The control method according to claim 14.

16. A program for causing a computer to execute the control method according to claim 14 or 15.

Images