Information processing device, control method, program

Abstract

This invention provides an information processing device, control method, and program that prevent overshoot and hunting of the tripod head and enable smooth tracking of the subject's movement. [Solution] When performing automatic tracking photography using a pan / tilt head with a shooting unit, the information processing device calculates the speed and direction of the moving object included in the video input from the pan / tilt head to predict the trajectory of the moving object, calculates the predicted trajectory of the pan / tilt head assuming that control command values ​​have been input to the pan / tilt head, calculates an evaluation value using an evaluation function with the difference between the trajectory of the moving object and the trajectory of the pan / tilt head, the speed of the moving object, and the system delay time, selects the evaluation value that minimizes the evaluation value from among the calculated multiple evaluation values, determines the control command value based on the selected evaluation value, and controls the pan / tilt head.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method, and a program, and more particularly to an information processing apparatus, a control method, and a program for controlling the movement of a pan-tilt unit to perform dynamic subject tracking control.

Background Art

[0002] Conventionally, a pan-tilt system in which a user remotely operates a camera from an operation device to acquire a desired video is widely known. For example, the video of an aircraft seen on TV news is taken by remotely operating a pan-tilt device installed on the airport rooftop from a broadcasting station.

[0003] In addition, an automatic tracking shooting system has been proposed in which an image recognition technology is mounted on this pan-tilt device to detect an object in a video and automatically operate and track pan, tilt, zoom, and focus in accordance with the movement of the object. Thereby, even if the user does not operate the operation device, a moving object can be automatically photographed.

[0004] In recent years, an example using artificial intelligence (hereinafter abbreviated as AI) and machine learning as an image recognition technology mounted on an automatic tracking shooting system is known. Particularly regarding object detection, it is known that a system using AI and machine learning can detect an object with high accuracy. As an example of using AI and machine learning in image recognition technology, the technology disclosed in Patent Document 1 estimates the control amount of an imaging device based on the result of object detection of a subject, evaluates the difference from the control amount instructed according to a user operation, and updates parameters required for estimation of the control amount. Thereby, the accuracy of estimation of the control amount for automatic shooting is improved.

[0005] In addition, the technology disclosed in Patent Document 2 is a technology for controlling an actuator by predicting the speed of a tracking target object and matching it with the speed of an imaging device. Thereby, a smooth tracking video can be obtained.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2018-113660 [Patent Document 2] Japanese Patent Publication No. 2008-61130 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, the prior art disclosed in Patent Document 1 only evaluates the control amount of the imaging device operated by the user, and does not consider overshoot, where the control amount exceeds the target value, or hunting, where the control amount moves up and down. When using only AI and machine learning, overshoot and hunting occur due to the delay in object detection and the error in estimating the control amount of the imaging device based on the object detection results. If the delay in object detection is large, the pan / tilt head device may not be able to smoothly track the movement of the subject, which may cause hunting. If the accuracy of the control amount estimation is low, there is a possibility of overshoot, such as losing track of the subject being tracked due to the field of view. In the prior art disclosed in Patent Document 2 as well, the delay in object detection and the error in estimating the control amount based on the object detection results lead to incorrect estimation of the control amount, resulting in overshoot and hunting.

[0008] Therefore, in automatic tracking shooting using AI and machine learning, there is a problem in that the pan / tilt head device cannot smoothly track the subject due to overshoot and hunting.

[0009] This invention has been made in view of the above-mentioned problems, and aims to provide an information processing device, control method, and program that can prevent overshoot and hunting of the tripod head and smoothly track the movement of the subject. [Means for solving the problem]

[0010] To solve the above problems, the information processing device according to claim 1 of the present invention is an information processing device that performs automatic tracking photography using a pan / tilt head having a shooting unit, and comprises: a video input means for receiving video from the pan / tilt head; a moving object trajectory calculation means for calculating the speed and direction of travel of a moving object included in the input video and calculating the predicted trajectory of the moving object; a pan / tilt head trajectory calculation means for calculating the predicted trajectory of the pan / tilt head assuming that a control command value has been input to the pan / tilt head; and an evaluation means for calculating an evaluation value using an evaluation function with respect to the difference between the trajectory of the moving object and the trajectory of the pan / tilt head, the speed of the moving object, and the delay time of the system, wherein the evaluation means selects the evaluation value that has the smallest evaluation value from among a plurality of evaluation values ​​calculated by the evaluation means, determines the control command value based on the selected evaluation value and controls the pan / tilt head. [Effects of the Invention]

[0011] According to the present invention, it is possible to prevent overshoot and hunting of the tripod head and smoothly track the movement of the subject. [Brief explanation of the drawing]

[0012] [Figure 1] This is a system configuration diagram of the automatic tracking and shooting system including the information processing device in Example 1. [Figure 2] This figure shows the hardware configuration of the automatic tracking and shooting system shown in Figure 1. [Figure 3] This figure shows the software configuration of the automatic tracking shooting system shown in Figure 1. [Figure 4] This is a conceptual diagram of the trained model and its input / output data in Example 1. [Figure 5] Figure 1 shows an example of the operation of the automatic tracking shooting system. [Figure 6] This is a flowchart of the control command value search process in Example 1. [Figure 7] This diagram shows the ideal predicted trajectory of the subject and the tripod head. [Figure 8] This figure shows the predicted trajectory of the subject and the pan / tilt head after the control cycle in Example 1.

Mode for Carrying Out the Invention

[0013] (Example 1) Hereinafter, Example 1 of the present invention will be described.

[0014] FIG. 1 is a system configuration diagram of an automatic tracking shooting system 1 including an information processing apparatus 100 in this embodiment.

[0015] In FIG. 1, the automatic tracking shooting system 1 is composed of an information processing apparatus 100, a pan-tilt device 200, an operation device 300, and a network 400.

[0016] The network 400 is a communication line such as a public telephone line or the Internet, and connects the information processing apparatus 100 and the operation device 300 communicably. In this embodiment, the information processing apparatus 100 and the pan-tilt device 200 are connected by two wired communication lines, a communication line 500a for video transmission and a communication line 500b for serial communication. However, it is not limited to this as long as these communications can be accurately performed. For example, the pan-tilt device 200 may also be connected to the network 400, and the information processing apparatus 100 and the pan-tilt device 200 may be communicably connected via the network 400.

[0017] When the user operates the operation device 300, a command corresponding to the operation is transmitted from the operation device 300 to the pan-tilt device 200 via the network 400 and the information processing apparatus 100. The pan-tilt device 200 performs internal control according to the content of the command from the operation device 300. Thereby, the user can remotely operate the pan-tilt device 200.

[0018] The pan-tilt device 200 includes a camera 201 and a drive unit 202 that turns the shooting direction thereof in the pan and tilt directions, and details will be described later in FIG. 2 and the like. The pan-tilt device 200 performs shooting of a video and adjustment of the shooting direction according to the remote operation.

[0019] The video captured by the pan / tilt head device 200 is also input to the information processing device 100. Based on the video input from the pan / tilt head device 200, the information processing device 100 performs various calculations and records necessary for automatic tracking shooting.

[0020] The pan / tilt head unit 200 and the information processing unit 100 are installed at locations such as airports, transmission towers, and television station rooftops, while the control unit 300 is installed inside the television station, etc. In the following description of this embodiment, we will explain using the example where the information processing unit 100 and the pan / tilt head unit 200 are installed at an airport and the subject to be automatically tracked and photographed is an aircraft.

[0021] Figure 2 shows the hardware configuration of the automatic tracking shooting system 1.

[0022] In Figure 2, the information processing device 100 consists of a RAM 101, a GPU 102, a CPU 103, an input unit 104, a storage unit 105, a serial communication unit 106, a network communication unit 107, and a UI unit 108, all of which are connected to each other via a bus.

[0023] The pan / tilt head device 200 consists of a camera 201, a CPU 204, and a drive unit 202, a serial communication unit 203, and a storage unit 205, all of which are connected to the CPU 204.

[0024] The operating device 300 consists of a network communication unit 301, an operating unit 302, a storage unit 303, a CPU 304, and a display unit 305, all of which are connected to each other via a bus.

[0025] The hardware configuration of the information processing device 100 will be described below.

[0026] RAM 101 is a volatile memory. The CPU 103 controls each part of the information processing device 100 using RAM 101 as work memory, for example, according to a program stored in the memory unit 105, and realizes the software configuration (Figure 3) described later.

[0027] GPU102 can perform parallel processing more efficiently than CPU103. Therefore, when performing multiple training processes using a learning model with many parameters, such as a deep neural network, as described later (Figure 4), CPU103 uses GPU102 for processing.

[0028] The input unit 104 (video input means) is an interface for inputting video signals captured by the camera 201 (shooting unit) to the information processing device 100 via the communication line 500a, and is one of various communication interfaces such as USB. The storage unit 105 is a non-volatile memory, in which image data, other data, and various programs for the operation of the CPU 103 are stored in predetermined areas. The storage unit 105 is composed of a storage medium such as an HDD or flash memory. The serial communication unit 106 is an interface for serial communication with the pan / tilt head device 200 via the communication line 500b, based on the control of the CPU 103. The network communication unit 107 (transmission means / acquisition means) is a communication interface for communicating with the operating device 300 via the network 400, based on the control of the CPU 103. The network communication unit 107 transmits video captured by the camera 201 and the status and warnings of the pan / tilt head device 200 (transmission means), and acquires automatic shooting start commands (Figure 5), which will be described later, from the operating device 300 (acquisition means). The UI unit 108 is a user interface that receives operation input from a user operating the information processing device 100 and displays information from the information processing device 100 to the user. The UI unit 108 consists of a keyboard, mouse, display touch panel, etc.

[0029] The following describes the hardware configuration of the tripod head unit 200.

[0030] Camera 201 is pan-tiltable around the mounting base 200 and captures images of a target subject. Camera 201 is equipped with an optical zoom lens that allows for changing the shooting magnification. Upon receiving a control command value from the CPU 204, including the zoom control amount, it changes the magnification of the captured image according to that control command value. Furthermore, Camera 201 is also equipped with a digital zoom function that locally enlarges a portion of the captured image. The digital zoom function is used when the magnification cannot be increased using only the optical zoom lens to the level of the captured image magnification corresponding to the control command value from the CPU 204, i.e., when it is desired to further enlarge the captured image. Camera 201 is also configured with a focus lens that allows for changing the focal position. Upon receiving a control command value from the CPU 204, including the focus control amount (described later), it changes the focal position of the captured image according to that control command value. In addition, Camera 201 is connected to the input unit 104 of the information processing device 100 via a wired communication line 500a through the pan-tilt device 200 and outputs the captured video signal to the information processing device 100.

[0031] The drive unit 202 includes circuits such as actuators for driving the pan / tilt mount 200 and rotating the camera 201 in the pan and tilt directions. When the pan / tilt mount 200 receives control command values, including the control amounts for pan and tilt, from the CPU 204 (described later), it can capture tracking footage by rotating the camera 201 in the pan and tilt directions relative to the target subject according to those control command values.

[0032] The serial communication unit 203 is connected to the serial communication unit 106 via the communication line 500b and is an interface for serial communication with the information processing device 100 based on the control of the CPU 204. The CPU 204 controls each part of the pan / tilt head device 200 according to a program stored in, for example, the memory unit 205. The memory unit 205 is a non-volatile memory, and setting data for the pan / tilt head device 200, other data, and various programs for the operation of the CPU 204 are stored in predetermined areas.

[0033] The hardware resources of the operating device 300 will be described below.

[0034] The network communication unit 301 is a communication interface for communicating with the information processing device 100 via the network 400, based on the control of the CPU 304. The operation unit 302 consists of a joystick, an operating lever, and various switches, and the user operates it to control the rotation of the pan / tilt head device 200, and adjust the zoom, focus, and gain of the camera 201. The memory unit 303 is a non-volatile memory, and setting data for the operation unit 300, other data, and various programs for the operation of the CPU 304 are stored in predetermined areas. The CPU 304 controls each part of the operation unit 300 according to the programs stored in the memory unit 303, for example. The display unit 305 is a touch panel display that notifies the user of the status and warnings of the pan / tilt head device 200.

[0035] Note that CPU103, CPU204, and CPU304 can each be configured with one or more processors.

[0036] Figure 3 shows the software configuration of the automatic tracking shooting system 1 having the hardware configuration shown in Figure 2.

[0037] The following describes an example of the software configuration of the information processing device 100.

[0038] The information processing device 100 consists of a learning unit 130, a data recording unit 131, an evaluation unit 132, an image processing unit 133, an estimation unit 134, a preprocessing unit 135, a moving object trajectory prediction unit 136, a pan / tilt head trajectory prediction unit 137, and a pan / tilt head control unit 138.

[0039] The learning unit 130 executes a learning process to enable the estimation unit 134 to perform estimation, and generates internal parameters for the trained model 143 (Figure 4). The details of the learning process will be described later in the explanation of Figure 4.

[0040] The data recording unit 131 performs the following processing: recording of images obtained by automatic tracking photography, recording of training data (described later in Figure 4), recording of control command values ​​used for prediction by the pan / tilt head trajectory prediction unit 137, and recording of evaluation values ​​calculated by the evaluation unit 132.

[0041] The evaluation unit 132 (evaluation means) uses the trajectory of the subject predicted by the moving object trajectory prediction unit 136 and the pan / tilt head trajectory predicted by the pan / tilt head trajectory prediction unit 137 to calculate an evaluation value using an evaluation function. The details of this evaluation value will be described later in the explanation from Figure 7 onwards.

[0042] The image processing unit 133 processes the video signal from the pan / tilt head device 200. Specifically, this involves resizing the image and adjusting its brightness.

[0043] The estimation unit 134 (estimation means) takes the output video from the preprocessing unit 135 as input data and inputs it into a trained model that uses internal parameters generated by the learning unit 130 to perform estimation.

[0044] The preprocessing unit 135 performs various noise processing and averaging processes and outputs the position of the subject in the video (the subject's current position). The video captured by the pan / tilt head device 200 contains noise other than the subject (if the subject is an aircraft, this noise includes other aircraft not being tracked, parts of the background, clouds, etc., that could be mistaken for an aircraft). The preprocessing unit 135 processes this noise and improves the reliability of the estimation results of the estimation unit 134.

[0045] The moving object trajectory prediction unit 136 (moving object trajectory calculation means) calculates the predicted trajectory of the subject. The predicted trajectory of the subject is calculated by using the output result of the estimation unit 134, calculating the subject's speed and direction of travel, and performing statistical processing. The calculation of the subject's speed and direction of travel is performed by combining not only the output of the estimation unit 134 but also the pan and tilt angles of the camera 201 provided by the pan / tilt head device 200. Alternatively, the subject's speed and direction of travel may be calculated using external sensors that acquire ADS-B information or other aviation radio signals.

[0046] The pan / tilt / zoom trajectory prediction unit 137 (pan / tilt / zoom trajectory calculation means) calculates the predicted trajectory of the pan / tilt / zoom. Here, the predicted trajectory of the pan / tilt / zoom represents the change in the tracking target position calculated from the control results of the pan, tilt, zoom, focus, and zoom of the pan / tilt / zoom device 200 based on the control command value, assuming that a control command value has been input to the pan / tilt / zoom device 200. The tracking target position here refers to the position in the shooting screen where the subject is to be held during tracking shooting. The predicted trajectory of the pan / tilt / zoom may also be inferred from past control information of the pan, tilt, zoom, and focus of the pan / tilt / zoom device 200. Alternatively, the predicted trajectory of the pan / tilt / zoom may be inferred from sensor information such as rotary encoders and other rotation sensors and position sensors connected to each of the mechanisms that perform the pan, tilt, zoom, and focus of the pan / tilt / zoom device 200. Furthermore, the predicted trajectory of the pan / tilt / zoom may be inferred by combining statistical processing such as a Kalman filter.

[0047] When the evaluation unit 132 receives the predicted trajectory of the subject and the predicted trajectory of the pan / tilt head calculated above, it determines whether the evaluation value based on these is optimal. If the evaluation value is optimal, it updates the control command value to the pan / tilt head device 200. The detailed method for updating the control command value to the pan / tilt head device 200 will be described later in Figure 6.

[0048] The pan / tilt head control unit 138 outputs the control command values ​​updated by the evaluation unit 132 to the pan / tilt head device 200. As a result, it is possible to automatically track and photograph the subject at a desired position within the field of view. Hereinafter, controlling the pan / tilt head device 200 and performing tracking photography will be referred to as automatic photography.

[0049] Furthermore, the learning unit 130 uses the GPU 102 in addition to the CPU 103 for processing. Specifically, when performing learning processing using a learning model, the CPU 103 and GPU 102 cooperate to perform calculations for learning. However, the processing of the learning unit 130 may also be performed by the CPU 103 or the GPU 102 alone. In addition, the estimation unit 134 and the pan / tilt control unit 138 may also use the GPU 102 in the same way as the learning unit 130.

[0050] The following describes an example of the software configuration for the tripod head device 200.

[0051] The pan / tilt head device 200 consists of a pan / tilt control unit 230, a camera control unit 231, a setting management unit 232, and a communication unit 233.

[0052] The pan-tilt control unit 230 outputs signals to the drive unit 202 for driving the pan and tilt of the camera 201 based on the drive commands received by the communication unit 233.

[0053] The camera control unit 231 outputs a signal to the camera 201 to control the camera 201 based on the command received by the communication unit 233.

[0054] The settings management unit 232 manages the settings of the operating device 300. Specific settings managed here include the maximum speed and drivable range for pan and tilt, respectively.

[0055] The communication unit 233 exchanges control commands and status information with the tripod head control unit 138 in accordance with predetermined communication rules (protocols).

[0056] The following describes an example of the software configuration for the operating device 300.

[0057] The operating device 300 consists of a communication unit 330 and a display unit 331.

[0058] The communication unit 330 exchanges control commands and status information with the pan / tilt head control unit 138 in accordance with predetermined communication rules (protocols). The communication unit 330 can also acquire moving images captured by the camera 201 via the information processing device 100.

[0059] The display unit 331 (display means / reception means) is a touch panel display that can display the status and warnings of the pan / tilt head device 200, as well as video footage acquired by the communication unit 330. When a user touches a subject they want to track from among the subjects included in the video footage displayed on the display unit 331, that touched subject is set as the subject to be tracked. The automatic shooting start command shown in Figure 5, which will be described later, also includes the position information of this set subject to be tracked. However, this is not limited to the above, as long as the subject to be tracked is selected by the user using the operating device 300. For example, a subject may be set as the subject to be tracked by user operation on the operating unit 302 instead of by touch operation on the display unit 331.

[0060] Figure 4 is a conceptual diagram of the trained model and its input / output data in this embodiment.

[0061] In this embodiment, a multi-layered deep neural network model is used as the learning model, in which the internal parameters have been adjusted by machine learning using AI.

[0062] The input data 140 is image data obtained by processing a single frame image (hereinafter referred to as "image") from a video captured by the camera 201 of the pan / tilt head device 200 in the image processing unit 133. Since the pan / tilt head device 200 captures video, the actual input data is a single frame from the video; however, for the sake of simplicity, the input data will be referred to as "image" from now on.

[0063] Output data 141 consists of the tags, position coordinates, and likelihood of objects present in input data 140. The tags to be output are selected from the tags included in the training data input during training. Two position coordinates, coordinate 1 and coordinate 2, are output. As shown in image 142 (Figure 4), in this embodiment, the top left of the circumscribing frame of the estimated object is coordinate 1, and the bottom right is coordinate 2. From these two coordinates, the object's size and center point coordinates can be calculated. The likelihood is a value between 0 and 1, and a higher value indicates a higher confidence in the estimation process for the output tag. Training data is data specified by an image, the name (tag) of the object in the image, and the coordinates of the object (coordinate 1, coordinate 2).

[0064] The trained model 143 is composed of a deep neural network, and its internal parameters are generated by the learning unit 130. The learning unit 130 may also include an error detection unit and an update unit. The error detection unit obtains the error between the output data output in response to the input data input to the trained model 143 and the training data. The error detection unit may use a loss function to calculate the error between the output data from the trained model 143 and the training data. The update unit updates the connection weight coefficients, etc., between the nodes of the trained model 143 based on the error obtained by the error detection unit, so as to reduce the error. This update unit updates the connection weight coefficients, etc., for example, using backpropagation. Backpropagation is a method for adjusting the connection weight coefficients, etc., between the nodes of the trained model 143 so as to reduce the above error.

[0065] Figure 5 shows an example of the operation of automatic tracking shooting by the automatic tracking shooting system 1. Note that the information processing device 100, pan / tilt head device 200, and operating device 300 in the figure are the same as in Figure 1, so their explanation is omitted.

[0066] First, the control device 300 sends an automatic shooting start command to the information processing device 100 (operation (1)). This is performed by the user operating a predetermined control unit 302 of the control device 300 to switch to automatic shooting. Next, the pan / tilt head device 200 sends a video signal to the information processing device 100 (operation (2)).

[0067] The information processing device 100 detects a subject from the video signal and performs an estimation process for the subject's current position (operation (3)). From the estimation result of the subject's position, it calculates the subject's speed and direction of travel and calculates the predicted trajectory of the moving object (operation (4)). The process of operation (4) is performed by the moving object trajectory prediction unit 136 described above in Figure 3. The predicted trajectory of the pan / tilt head is calculated assuming that multiple candidate control command values ​​are input, and an evaluation value is calculated for each of the calculated predicted trajectories of the pan / tilt head using an evaluation function (operation (5)). The process up to calculating the predicted trajectory of the pan / tilt head in operation (5) is performed by the pan / tilt head trajectory prediction unit 137 described above in Figure 3. The process of calculating the evaluation value using the evaluation function in operation (5) is performed by the evaluation unit 132 described above in Figure 3. The method for searching for multiple candidate control command values ​​will be described in detail from Figure 6 onwards. The control command value that will result in the pan / tilt head predicted trajectory with the optimal evaluation value is selected from among the multiple candidates (operation (6)) and transmitted to the pan / tilt head device 200 (operation (7)). The calculation method for the evaluation value of operation (6) and the optimal evaluation value will be described in detail in Figure 7 and subsequent figures. The pan / tilt head device 200 performs control according to the received control command value (operation (8)). When the control cycle has elapsed, the processes of operations (2) to (8) are executed to enable automatic tracking and shooting of the target subject.

[0068] Through the processes described above, the pan / tilt head device 200 is controlled, enabling smooth automatic tracking and shooting.

[0069] Next, using the flowchart in Figure 6, we will explain the detailed flow of the control command value update process described in operations (4) to (7) in Figure 5.

[0070] This process is executed by the CPU 103 and GPU 102 working together to load the program stored in the memory unit 105 into the RAM 101.

[0071] First, in step S600, the moving object trajectory prediction unit 136 calculates the predicted trajectory of the subject. The predicted trajectory of the subject is the movement trajectory in the camera's coordinate system, the movement trajectory in the polar coordinate system, or the movement trajectory in the 3D spatial coordinate system. The prediction method involves estimating the position in the camera's coordinate system using object detection AI, and then calculating the subject distance by estimating it from the zoom and focus positions.

[0072] In step S601, the pan / tilt / zoom trajectory prediction unit 137 calculates the predicted trajectory of the pan / tilt / zoom unit. The predicted trajectory of the pan / tilt / zoom unit is predicted from the control results of the pan, tilt, zoom, and focus of the pan / tilt / zoom unit 200. One prediction method is to predict the predicted trajectory of the pan / tilt / zoom unit from the predicted position of the pan / tilt / zoom unit 200, assuming that the past operating status of the pan / tilt / zoom unit 200 and control command values ​​have been output. The predicted position of the pan / tilt / zoom unit 200 is calculated according to the control model of the pan / tilt / zoom unit 200, which has been obtained in advance through experiments or simulations. The operating status of the pan / tilt / zoom unit 200 may be obtained from sensor information such as a rotary encoder or other rotation sensors and position sensors connected to the mechanism that performs the pan, tilt, zoom, and focus of the pan / tilt / zoom unit 200. In the initial state, the control command values ​​output here are control command values ​​with arbitrarily set initial values.

[0073] In step S602, the evaluation unit 132 calculates an evaluation value. The evaluation value is a numerical result of an evaluation function calculated using the predicted trajectory of the subject calculated in step S600 and the predicted trajectory of the pan / tilt head calculated in step S601. The evaluation function will be described later in Figure 7.

[0074] In step S603, the evaluation unit 132 compares the evaluation value calculated in step S602 with the evaluation value in the data recording unit 131 to determine whether it is more optimal. If the evaluation value is more optimal (YES in step S603), the process proceeds to step S604; otherwise, it proceeds to step S605. Here, an optimal evaluation value means that the evaluation value is either the minimum or maximum value. Whether the evaluation value is considered optimal (minimum or maximum value) depends on the evaluation function, which will be explained in detail in the description of the evaluation function in Figure 7.

[0075] In step S604, the data recording unit 131 swaps the evaluation value stored in the data recording unit 131 with the evaluation value calculated in step S602. Similarly, the control command value stored in the data recording unit 131 is swapped with the control command value used in step S601 to calculate the predicted trajectory of the pan / tilt head. One search for the control command value is performed from step S601 to step S604.

[0076] In step S605, the mobile body trajectory prediction unit 136 determines whether all control command values ​​to be searched have been searched. If all values ​​have been searched (YES in step S605), the process proceeds to step S606; otherwise, the process proceeds to step S608. "All values ​​searched" means that all control command values ​​to be searched have been searched. The control command values ​​to be searched here are control command values ​​that represent velocity and acceleration within an arbitrary range.

[0077] In step S606, the pan / tilt head control unit 138 outputs to the pan / tilt head device 200 a control command value for which the evaluation value stored in the data recording unit 131 has been optimized.

[0078] In step S607, the pan / tilt head control unit 138 determines whether a tracking end command has been output from the operating device 300. If a tracking end command has been output (YES in step S607), this process is terminated; otherwise, the process proceeds to step S609.

[0079] In step S608, the pan / tilt head trajectory prediction unit 137 updates the control command value to an unsearched control command value and proceeds to step S601. The method for updating the control command value involves adding or subtracting an arbitrary value from the control command value to be searched. This makes it possible to search for velocity and acceleration control command values ​​within an arbitrary range.

[0080] In step S609, the data recording unit 131 erases the evaluation values ​​within the data recording unit 131 and proceeds to step S600. This allows the control command values ​​to be updated to the optimal evaluation values, and the control amounts for pan, tilt, zoom, and focus of the tripod head device 200 can be changed.

[0081] In this embodiment, in step S604, the evaluation value and control command value in the data recording unit 131 are swapped. However, it is also acceptable to store all the calculated evaluation values ​​and control command values ​​in the data recording unit 131 and select from the stored values.

[0082] Figure 7 shows the ideal predicted trajectory of the subject and the tripod head.

[0083] To simplify the explanation, in Figure 7, the predicted trajectory of the subject and the predicted trajectory of the pan head are explained only in terms of the angle in the pan direction relative to the installation position of the pan head device 200. In Figure 7, the thick line shows the past trajectory of the subject, and the dotted line shows the predicted trajectory of the subject. Also, the double line shows the past trajectory of the pan head, and the dotted line shows the predicted trajectory of the pan head.

[0084] Let T0 be the current time, T1 be the time after the control cycle of T0, and Tend be the time interval from T0 to calculate the predicted trajectory of the subject and the predicted trajectory of the pan / tilt head. Tend is the time for calculating each predicted trajectory and is predetermined based on the subject's speed and the system's control characteristics. An example of Tend is 1.0 seconds.

[0085] An example of an evaluation function in such a situation is the following formula:

[0086]

number

[0087] Here, A and B are positive coefficients and are predetermined from the velocity of the subject and the delay time of the system. Also, y obj Let (t) be the predicted position of the subject at time t, and y ptz (t) is the predicted position of the pan / tilt head at time t, assuming that a control command value has been input. In this equation, the term with coefficient A calculates the maximum difference between the predicted speed of the subject and the predicted speed of the pan / tilt head, and the term with coefficient B calculates the difference between the position of the subject and the position of the pan / tilt head at the time of Tend.

[0088] When using this evaluation function, the optimal value in step S603 is the value that minimizes the evaluation value, and in the flowchart in Figure 6, the search will be conducted for an evaluation value that results in a lower value. By selecting a control command value that lowers this evaluation value, the speed of the pan / tilt head can be smoothly changed in accordance with the speed of the subject, the predicted trajectory of the subject and the pan / tilt head at the time of Tend can be matched, and the control can be made so that the subject reaches the center of the image field of view. As a result, it is possible to smoothly track the movement of the subject as shown in Figure 7.

[0089] Other examples of evaluation functions include the following formulas.

[0090]

number

[0091] When using this evaluation function, the optimal value for step S603 is the value that minimizes the evaluation value. By selecting a control command value that lowers this evaluation value, it is possible to control the subject and the trajectory of the tripod head to match during the time from T0 to Tend.

[0092] As an example, in this embodiment 1, the updated control command value is output to the pan / tilt head device 200 using the evaluation function of formula 1. This controls the predicted trajectory of the subject and the predicted trajectory of the pan / tilt head after the control cycle, as shown in Figure 8. Similarly, after the control cycle, the predicted trajectories of the subject and the pan / tilt head are calculated, evaluated using the evaluation function, and the control command value is updated. This makes it possible to evaluate the predicted trajectory for each control cycle, and to smoothly track the movement of the subject.

[0093] In this embodiment, an aircraft was used as an example of an object to be tracked, but the system is not limited to this; other objects may also be automatically tracked and photographed.

[0094] In this embodiment, the information processing device 100 and the pan / tilt head device 200 were connected via serial communication and a wired video signal line. However, the connection is not limited to these methods; it may also be made via a public telephone line or a communication line such as the Internet.

[0095] Of the processing units described above, the estimation unit 134 performs processing using a machine learning model, but rule-based processing such as a lookup table (LUT) may also be used. In that case, for example, the relationship between input data and output data is created in advance as a LUT. This created LUT may then be stored in the storage unit 105 of the information processing device 100. When performing processing with the estimation unit 134, the output data can be obtained by referring to this stored LUT. In other words, the LUT acts as a program to perform processing equivalent to that of the above-mentioned processing units, and works in cooperation with the CPU 103 or GPU 102, etc., to perform the processing of the above-mentioned processing units.

[0096] The present invention can also be realized by supplying a program that implements the functions of the above-described embodiments to a system or device via a network or storage medium, and by a computer in that system or device reading and executing the program. The computer has one or more processors or circuits and may include a plurality of separate computers or a network of a plurality of separate processors or circuits for reading and executing computer executable instructions.

[0097] (Other embodiments) A processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gateway (FPGA). It may also include a digital signal processor (DSP), a dataflow processor (DFP), or a neural processing unit (NPU).

[0098] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0099] This embodiment includes the following configurations, methods, and programs. (Configuration 1) An information processing device that performs automatic tracking photography using a pan / tilt head having a shooting unit, comprising: a video input means for receiving video from the pan / tilt head; a moving object trajectory calculation means for calculating the speed and direction of travel of a moving object included in the input video and calculating the predicted trajectory of the moving object; a pan / tilt head trajectory calculation means for calculating the predicted trajectory of the pan / tilt head assuming that a control command value has been input to the pan / tilt head; and an evaluation means for calculating an evaluation value using an evaluation function with respect to the difference between the trajectory of the moving object and the trajectory of the pan / tilt head, the speed of the moving object, and the delay time of the system, wherein the evaluation means selects the evaluation value that has the smallest evaluation value from among a plurality of evaluation values ​​calculated, determines the control command value based on the selected evaluation value and controls the pan / tilt head. (Configuration 2) The information processing device according to Configuration 1, characterized in that the evaluation function calculates the difference between the predicted position of the moving body and the predicted position of the pan / tilt head. (Configuration 3) The information processing device according to Configuration 1 or 2, characterized in that the evaluation function calculates the maximum difference between the predicted speed of the moving body and the predicted speed of the pan / tilt head. (Configuration 4) An information processing device according to any one of Configurations 1 to 3, characterized in that the control command value includes control amounts for pan, tilt, zoom, and focus. (Configuration 5) The information processing device according to any one of Configurations 1 to 4, characterized in that the pan / tilt / grip trajectory calculation means predicts the trajectory of the pan / tilt / grip from past control amounts of pan, tilt, zoom, and focus input to the pan / tilt / grip. (Configuration 6) The information processing device according to any one of Configurations 1 to 5, wherein the tripod head has a plurality of sensors connected to each of the mechanisms that perform pan, tilt, zoom, and focus, and the tripod head trajectory calculation means predicts the trajectory of the tripod head from the sensor information of the plurality of sensors. (Configuration 7) An information processing device according to any one of Configurations 1 to 6, further comprising an estimation means that, when one frame from the input video is input, estimates the position coordinates of the moving object in the video using a trained model as output data, and the moving object trajectory calculation means calculates the velocity and direction of travel of the moving object from the output data estimated by the estimation means. (Configuration 8) The information processing device according to Configuration 7, characterized in that the trained model is a trained model of a deep neural network. (Configuration 9) An information processing device according to any one of Configurations 1 to 8, characterized in that the speed and direction of travel of the moving body are estimated using an external sensor. (Configuration 10) An information processing device according to any one of Configurations 1 to 9, further comprising: a transmission means that is communicably connected to an operating device having a display unit and an operating unit, and transmits the input video to the operating device; and an acquisition means that acquires location information of the moving object from the operating device, wherein the operating device comprises a display means that displays the input video transmitted from the information processing device on the display unit; and a reception means that accepts the user selection of the moving object from among the subjects included in the input video displayed on the display unit at the operating unit. (Method 1) A control method for an information processing device that performs automatic tracking photography using a pan / tilt head having a shooting unit, comprising: a video input step in which video from the pan / tilt head is input; a moving object trajectory calculation step in which the speed and direction of travel of a moving object included in the input video are calculated to calculate the predicted trajectory of the moving object; a pan / tilt head trajectory calculation step in which the predicted trajectory of the pan / tilt head is calculated assuming that a control command value has been input to the pan / tilt head; and an evaluation step in which an evaluation function is used to calculate an evaluation value using the difference between the trajectory of the moving object and the trajectory of the pan / tilt head, the speed of the moving object, and the delay time of the system, wherein an evaluation value that minimizes the evaluation value is selected from among a plurality of evaluation values ​​calculated in the evaluation step, and the control command value is determined based on the selected evaluation value to control the pan / tilt head. (Program 1) A program for causing a computer to function as one of the means of an information processing device described in any one of Configurations 1 to 10. [Explanation of Symbols]

[0100] 100 Information Processing Devices 130 Learning Department 131 Data Recording Unit 132 Evaluation Department 133 Image Processing Unit 134 Estimation Department 135 Pre-processing section 136 Mobile object trajectory prediction unit 137 Mount Orbit Prediction Unit 138 Tripod Head Control Unit 140 Input Data 141 Output data 142 images 143 Pre-trained models

Claims

1. An information processing device that performs automatic tracking photography using a pan / tilt head having a shooting unit, A video input means that receives video from the aforementioned tripod head, A moving object trajectory calculation means calculates the speed and direction of travel of a moving object included in the input video and calculates the predicted trajectory of the moving object, A tripod head trajectory calculation means for calculating the predicted trajectory of the tripod head, assuming that a control command value is input to the tripod head, An evaluation means that calculates an evaluation value using an evaluation function with respect to the difference between the trajectory of the moving body and the trajectory of the pan / tilt head, the speed of the moving body, and the delay time of the system. It has, An information processing device characterized by selecting the evaluation value that has the smallest evaluation value from among a plurality of evaluation values ​​calculated by the evaluation means, determining the control command value based on the selected evaluation value, and controlling the pan / tilt head.

2. The information processing device according to claim 1, characterized in that the evaluation function calculates the difference between the predicted position of the moving body and the predicted position of the pan / tilt head.

3. The information processing device according to claim 1, characterized in that the evaluation function calculates the maximum difference between the predicted speed of the moving body and the predicted speed of the pan / tilt head.

4. The information processing apparatus according to claim 1, characterized in that the control command value includes control amounts for pan, tilt, zoom, and focus.

5. The information processing device according to claim 1, characterized in that the pan / tilt head trajectory calculation means predicts the trajectory of the pan / tilt head from past control amounts of pan, tilt, zoom, and focus input to the pan / tilt head.

6. The aforementioned tripod head has multiple sensors connected to each of the mechanisms that perform panning, tilting, zooming, and focusing. The information processing device according to claim 1, characterized in that the pan / tilt head trajectory calculation means predicts the trajectory of the pan / tilt head from the sensor information of the plurality of sensors.

7. When one frame from the input video is input, the system further includes estimation means that estimates the position coordinates of the moving object within the video using a trained model as output data. The information processing apparatus according to claim 1, characterized in that the moving body trajectory calculation means calculates the velocity and direction of travel of the moving body from the output data estimated by the estimation means.

8. The information processing apparatus according to claim 7, characterized in that the pre-trained model is a pre-trained model of a deep neural network.

9. The information processing apparatus according to claim 1, characterized in that the speed and direction of travel of the moving object are estimated using an external sensor.

10. It is connected to an operating device having a display unit and an operating unit in a communicative manner, A transmission means for transmitting the input video to the operating device, The operating device further comprises acquisition means for acquiring position information of the moving object, The aforementioned operating device is A display means for displaying the input video transmitted from the information processing device to the display unit, The information processing apparatus according to claim 1, further comprising a receiving means for receiving a user selection of a moving object from among the subjects included in the input video displayed on the display unit, via the operation unit.

11. A control method for an information processing device that performs automatic tracking photography using a pan / tilt head having a shooting unit, A video input step in which video from the aforementioned tripod head is input, A moving object trajectory calculation step calculates the speed and direction of travel of a moving object included in the input video and calculates the predicted trajectory of the moving object, A tripod head trajectory calculation step that calculates the predicted trajectory of the tripod head assuming that a control command value has been input to the tripod head, An evaluation step in which an evaluation function calculates an evaluation value using the difference between the trajectory of the moving body and the trajectory of the pan / tilt head, the speed of the moving body, and the delay time of the system, It has, A control method characterized by selecting the evaluation value that has the smallest evaluation value from among a plurality of evaluation values ​​calculated in the evaluation step, determining the control command value based on the selected evaluation value, and controlling the pan / tilt head.

12. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

Images