Information processing device, control method thereof, and program

Abstract

To provide an information processing device which can estimate a subject with proper accuracy in a proper cycle when tracking and imaging a moving subject.SOLUTION: An information processing device according to the present disclosure comprises: reception means which receives a moving image that is shot while controlling pan and tilt of imaging means; and estimation means which estimates a subject included in an image constituting the moving image. Here, the estimation means estimates the subject included in the image by using machine learning models with different properties according to the speed of the motion of pan or tilt of the imaging means when capturing the image.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method thereof, and a program. [Background technology]

[0002] Conventionally, remote shooting systems have been widely known in which a user remotely controls a camera from a controller to obtain a desired image. For example, an image of an aircraft can be captured by remotely controlling a pan head device permanently installed on the roof of an airport from a broadcasting station. In addition, an automatic tracking system is known in which such a pan head device is equipped with image recognition technology to estimate a subject appearing in the image, and automatically pans, tilts, and zooms the camera in accordance with the movement of the subject to track the subject. In such a system, a moving subject can be tracked and captured without the user having to operate a controller.

[0003] In recent years, image recognition technology using a machine learning model has become known. Patent Document 1 proposes a system that detects a subject using a machine learning model as an image recognition technology and changes the period of subject detection depending on the type of subject. The technology described in Patent Document 1 can detect a subject at an appropriate period while suppressing a decrease in detection accuracy. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-92354 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, when photographing a fast-moving subject, unless the period for estimating the subject (the time from inputting an image to completing the estimation of the subject) is shortened, a delay occurs in estimating the subject. On the other hand, when using a machine learning model, there is generally a trade-off between the period for estimating the subject and the accuracy. When the period for estimating the subject is long, a delay occurs in tracking, while when the accuracy of estimating the subject is low, the likelihood of the subject decreases, and there are cases where the subject cannot be estimated when the image is unclear or small. Therefore, in an automatic tracking system using a machine learning model, it is desirable to be able to estimate the subject with an appropriate period and accuracy. In this regard, the technology disclosed in Patent Document 1 takes into account only the type of subject, and does not take into account the estimation of a fast-moving subject.

[0006] The present invention has been made in view of the above-mentioned problems, and has as its object to realize a technique capable of estimating a subject with an appropriate period and accuracy when tracking and photographing a moving subject. [Means for solving the problem]

[0007] In order to solve this problem, for example, an information processing device of the present invention has the following configuration: That is, the information processing device has a receiving means for receiving a moving image captured while controlling the pan or tilt of an imaging means, and an estimating means for estimating a subject included in an image constituting the moving image, and the estimating means estimates a subject included in the image using a machine learning model with different characteristics according to the speed of the pan or tilt operation of the imaging means when capturing the image. The characteristics include a time from when the machine learning model inputs the image until the machine learning model completes the estimation of the object, and an accuracy of the machine learning model in estimating the object. , characterized in that Effect of the Invention

[0008] According to the present invention, when tracking and photographing a moving subject, it is possible to estimate the subject with an appropriate period and accuracy. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the configuration of an automatic tracking system according to the present invention. [Diagram 2] FIG. 1 is a block diagram showing a hardware configuration of an automatic tracking system according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a block diagram showing the software configuration of the automatic tracking system according to the present embodiment. [Figure 4] FIG. 1 is a diagram for explaining a learning model and input / output data according to the present embodiment; [Diagram 5] A flowchart showing a series of operations of an information processing device including a learning stage process according to the present embodiment. [Figure 6] FIG. 1 is a diagram showing an example of learning data according to the present embodiment; [Figure 7] FIG. 1 is a diagram for explaining the operation of the entire automatic tracking system according to the present embodiment. [Figure 8] 1 is a flowchart showing a series of operations in a tracking mode process according to the present embodiment. [Figure 9] FIG. 1 is a diagram for explaining a change in a learning model in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] <Shooting system configuration> FIG. 1 is a diagram showing a system to which the present invention can be applied, which is composed of an information processing device 100, a camera-head device 200, an operation device 300, and a network 150. The information processing device 100 is, for example, a personal computer capable of executing an image recognition technique using a machine learning model (sometimes simply referred to as a learning model). However, the information processing device 100 is not limited to a personal computer, and may be any other device capable of executing an image recognition technique using a machine learning model. In addition, although an example will be described in which the camera-head device 200 and the operation device 300 are each dedicated devices, each of the camera-head device 200 and the operation device 300 may be partially composed of electronic devices such as a personal computer or a tablet terminal.

[0012] When a user operates the operation device 300, a command corresponding to the operation is transmitted to the pan head device 200 via the network 150 and the information processing device 100. The pan head device 200 controls the shooting by the camera according to the contents of the command, so that the user can remotely operate the pan head device 200. The image captured by the pan head device 200 is transmitted to the information processing device 100, which performs various calculations and recording required for automatic tracking shooting. The network 150 is a communication line such as a public telephone line, a mobile communication network, or the Internet. For example, the pan head device 200 and the information processing device 100 are installed at spots such as an airport, a steel tower, or a TV station rooftop, and the operation device 300 is installed inside the TV station. In this embodiment, for example, a case will be described in which the information processing device 100 and the pan head device 200 are installed at an airport, and the object of automatic tracking shooting is an airplane.

[0013] Fig. 2 shows an example of the hardware configuration of each device constituting the system of Fig. 1. The information processing device 100 includes a RAM 101, a GPU 102, a CPU 103, an input unit 104, a storage unit 105, a serial communication unit 106, a network communication unit 107, and a UI unit 108. The pan head device 200 includes a camera 201, a drive unit 202, a serial communication unit 203, a CPU 204, and a storage unit 205. The operation device 300 includes a network communication unit 301, an operation unit 302, a storage unit 303, a CPU 304, and a display unit 305.

[0014] The configuration of the information processing device 100 will be described. The RAM 101 is, for example, a volatile memory, and is used as a temporary storage area such as a main memory or work area of ​​the CPU 103. The CPU 103 includes one or more processors, and controls each unit of the information processing device 100 using the RAM 101 as a work memory according to a program stored in, for example, a storage unit 105. The GPU 102 includes one or more processors, and can perform efficient calculations by processing more data in parallel. For this reason, calculations by the GPU 102 are effective when learning is performed multiple times using a learning model of a deep neural network (DNN) such as deep learning.

[0015] The input unit 104 is an interface for inputting a video signal to the information processing device 100, and is, for example, various communication interfaces such as a USB. The storage unit 105 is, for example, a non-volatile memory, and image data and other data, various programs for the CPU 103 to operate, and the like are stored in respective predetermined areas. The serial communication unit 106 is an interface for communicating with the pan head device 200 based on the control of the CPU 103. The network communication unit 107 is a communication interface for communicating with the operation device 300 via the network 150 based on the control of the CPU 103. The UI unit 108 is a user interface for accepting operation input from a user who operates the information processing device 100 and displaying information of the information processing device 100 to the user. The UI unit 108 may include a keyboard, a mouse, a display, and a touch panel.

[0016] Next, the hardware configuration of the pan head device 200 will be described. The camera 201 includes an imaging sensor, captures images of the surroundings where the pan head device 200 is installed, and sequentially outputs images showing target subjects (i.e., outputs video). The camera 201 includes an optical zoom lens capable of changing the imaging magnification, and the magnification of the captured video can be changed by sending a zoom control command from the CPU 204 to the camera 201. Furthermore, the camera 201 also has a digital zoom function that locally enlarges a part of the captured image. Digital zoom is performed when the magnification is insufficient with the optical zoom, that is, when it is desired to further enlarge the captured video. Moreover, the camera 201 is connected to the input unit 104 of the information processing device 100 by wire, and outputs a captured video signal to the information processing device 100.

[0017] The driving unit 202 includes an actuator for rotating the camera platform device 200 in the pan and tilt directions, a driving circuit for the actuator, and peripheral circuits. The camera platform device 200 rotates in the pan and tilt directions relative to a target subject, thereby capturing an image tracking the target subject. The serial communication unit 203 is connected to the serial communication unit 106 of the information processing device 100, and is an interface for communicating with the information processing device 100 under the control of the CPU 204. The CPU 204 includes one or more processors, and controls each part of the camera platform device 200 according to a program stored in the storage unit 205, for example. The storage unit 205 is a non-volatile memory, and setting data for the camera platform device 200, other data, various programs for the CPU 204 to operate, and the like are stored in respective predetermined areas.

[0018] Further, the hardware configuration of the operation device 300 will be described. The network communication unit 301 is a communication interface for communicating with the information processing device 100 via the network 150 based on the control of the CPU 304. The operation unit 302 is a joystick, an operation lever, and various switches, and the operator operates these to perform rotation control of the pan and tilt of the camera head device 200, zoom control, gain adjustment, and the like. The storage unit 303 is, for example, a non-volatile memory, and the setting data of the operation device 300 and other data, various programs for the CPU 304 to operate, and the like are stored in respective predetermined areas. The CPU 304 controls each part of the operation device 300 according to, for example, the program stored in the storage unit 303. The display unit 305 includes an LED, and notifies the user of the status of the camera head device 200, warnings, and the like. The display unit 305 may include a touch panel.

[0019] Next, the software configuration of the automatic tracking system will be described with reference to FIG.

[0020] First, a description will be given of the software configuration of the information processing device 100. The information processing device 100 includes a learning unit 150, a data storage unit 151, a learning model setting unit 152, a mode management unit 153, an image processing unit 154, an estimation unit 155, an estimation result processing unit 156, and a camera head control unit 157. Each of these units is realized by the CPU 103 or GPU 102 of the information processing device 100 executing a program, or may be realized in other configurations such as the storage unit 105 or a network communication unit.

[0021] The learning unit 150 executes a learning stage process for training the machine learning model before the estimation unit 155 executes an estimation stage process using the machine learning model (also simply referred to as a learning model). The learning stage process will be described later. The data storage unit 151 performs a recording process of the video captured with automatic tracking and a recording process of the learning data. The learning model setting unit 152 switches the learning model based on the control information of the pan, tilt, and zoom of the camera head output by the camera head control unit 157, and uses the learning model in the estimation unit 155. The learning models include a plurality of learning models with different characteristics such as processing time and estimation accuracy. The switching of the learning model by the learning model setting unit 152 and specific examples of the learning models will be described later.

[0022] The mode management unit 153 manages the operation modes of the information processing device 100. The operation modes include three modes: learning mode, automatic tracking mode, and manual mode. The detailed content of each mode will be described later. The image processing unit 154 receives video from the camera-to-camera device 200 and processes the received video. The video processing includes, for example, image resizing and brightness adjustment. The estimation unit 155 uses the output video of the image processing unit 154 as input data and a learning model that has been learned by the processing in the learning unit 150 to perform an estimation stage process.

[0023] The estimation result processing unit 156 performs various noise processing and averaging processing on the output of the estimation unit 155, and outputs the position in the image of the target subject to be tracked (current subject position). The video captured by the camera platform device 200 contains noise other than the tracking subject (aircraft other than aircraft, parts of the background, clouds, etc. that can be mistaken for aircraft). The estimation result processing unit 156 processes this noise and plays a role in outputting the current subject position with a high degree of reliability.

[0024] The tripod head control unit 157 calculates a control signal for controlling the tripod head device 200 according to the above-mentioned mode. When the current mode is the automatic tracking mode, the tripod head control unit 157 calculates and outputs a control signal for controlling the tripod head device 200 based on the current position of the subject and the tracking target position (the position where the subject is desired to be held within the shooting screen during tracking shooting). Furthermore, when the current mode is the manual mode, the tripod head control unit 157 outputs a command from the operation device 300.

[0025] For the processing by the learning unit 150, the GPU 102 may be used in addition to the CPU 103. For example, when executing the processing of the learning stage using a learning model, the CPU 103 and the GPU 102 cooperate to perform calculations. For the processing of the learning unit 150, calculations may be performed only by the CPU 103 or the GPU 102. Similarly to the learning unit 150, the estimation unit 155 may also use the GPU 102.

[0026] Next, the software configuration of the camera platform device 200 will be described. The camera platform device 200 includes a pan / tilt control unit 250, a camera control unit 251, a setting management unit 252, and a communication unit 253. The pan / tilt control unit 250 outputs a signal for driving the pan and tilt of the camera 201 to the driving unit 202 based on a driving command received by the communication unit 253. The camera control unit 251 outputs a signal for controlling the camera to the camera 201 based on a command received by the communication unit 253. The setting management unit 252 manages the settings of the operation device 300, and specific setting items include the maximum speed of pan and tilt, and a drivable range. The communication unit 253 exchanges control commands and status information with the camera platform communication unit 158 ​​of the information processing device 100 according to a predetermined communication rule (protocol).

[0027] Next, a description will be given of the software configuration of the operation device 300. The operation device 300 includes a communication unit 350 and a display unit 351. The communication unit 350 exchanges control commands and status information with the information processing device 100 in accordance with a predetermined communication rule (protocol).

[0028] One or more of the functional blocks in the above software configuration may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a combination of software and hardware.

[0029] Next, the input / output structure of the learning model in this embodiment will be described with reference to Fig. 4. Input data 400 is image data captured by the camera-to-camera device 200 and then processed by the image processing unit 154 of the information processing device 100. Note that since the camera-to-camera device 200 captures moving images, the image processing unit 154 actually outputs each frame of the moving images in sequence.

[0030] The output data 401 includes, for example, information on the tag, coordinates, and likelihood of an object present in the input data 400. The tag is selected from tags included in the learning data input during learning. The learning data will be described in detail later. The coordinates include information on two points, coordinate 1 and coordinate 2. For example, these coordinates indicate the top left (coordinate 1) and bottom right (coordinate 2) positions of the circumscribing frame of the estimated object, as shown in image 402 in FIG. 4. From the coordinates of these two points, the size and center point coordinates of the object are calculated. The likelihood is a value between 0 and 1, and a higher value indicates a higher reliability of the estimation of the output tag.

[0031] The learning model 403 is configured by, for example, a neural network, and the internal parameters of the neural network are generated or obtained (optimized) by the learning stage processing by the learning unit 150. The learning unit 150 may include an error detection unit and an update unit. The error detection unit obtains the error between the output data output from the output layer of the neural network according to the input data input to the input layer and the teacher data. The error detection unit may use a loss function to calculate the error between the output data from the neural network and the teacher data. The update unit updates the coupling weighting coefficients between the nodes of the neural network, etc., based on the error obtained by the error detection unit, so as to reduce the error. This update unit updates the coupling weighting coefficients, etc., using, for example, an error backpropagation method. The error backpropagation method is a method of adjusting the coupling weighting coefficients between the nodes of each neural network so as to reduce the above-mentioned error.

[0032] Next, a series of operations of the information processing device 100 including the processing in the learning stage will be described with reference to Fig. 5. Note that the CPU 103 and the GPU 102 execute a program stored in the storage unit 105 to realize the operations of the learning unit 150 and the like in this processing.

[0033] First, in step S500, the learning unit 150 determines whether or not the current mode is the learning mode. If the operation mode of the information processing device 100 is the learning mode, the learning unit 150 proceeds to step S501, and if not, the learning unit 150 proceeds to step S507.

[0034] In step S501, the learning unit 150 accepts a learning model to be learned from among a plurality of learning models. The plurality of learning models are configured with learning models that have different processing loads due to differences in at least one of the method of estimating an object, pre-processing before inference, post-processing, and the structure of a neural network, and thus have different periods and accuracies for estimating an object. The period for estimating an object is the time from when an image is input to when the learning model completes estimating the object (outputs the detection result). The accuracy of estimating an object (i.e., the accuracy of object detection) is the degree of agreement between the above-mentioned coordinates 1 and coordinates 2 and the coordinates where an actual object is located. A learning model with high accuracy for estimating an object has a high likelihood as described above even when an unclear image or an image of a small object is input. In other words, a learning model with high accuracy for estimating an object can estimate the object even when the size of the object in the image is smaller than a predetermined size.

[0035] Specific examples of learning models that differ in the method of estimating the subject include learning models such as YOLO and R-CNN. YOLO divides the input image into multiple regions and performs classification in the divided regions. This makes it possible to reduce the processing for position estimation and shorten the estimation cycle. In contrast, R-CNN performs classification after estimating the object's position. This allows for high inference performance and position detection.

[0036] Comparing YOLO and R-CNN, it is known that YOLO has a shorter period for estimating objects than R-CNN, but its accuracy is lower than R-CNN. On the other hand, R-CNN has a longer period for estimating objects than YOLO, but its accuracy is higher than YOLO.

[0037] Specific examples of multiple learning models with different pre-processing and post-processing include SSD300 and SSD512. Generally, when performing inference processing with a learning model, the input image is compressed in the pre-processing of inference. In this case, SSD300 compresses the input image to 300 pixels in length and width and inputs it to the learning model. On the other hand, SSD512 compresses the input image to 512 pixels in length and width and inputs it to the learning model. As a result, SSD300 has a shorter period for estimating the subject than SSD512, but is less accurate than SSD512. On the other hand, SSD512 has a longer period for estimating the subject than SSD300, but is known to have higher accuracy than SSD300.

[0038] As concrete examples of multiple learning models with different neural network structures, there are the above-mentioned learning models such as YOLO and Tiny-YOLO. Tiny-YOLO has the same pre-processing and post-processing of inference as YOLO, but the neural network structure is simpler than YOLO, which makes it possible to shorten the period for estimating the subject compared to YOLO.

[0039] In the following description, the case where YOLO and R-CNN are used will be described as an example. YOLO is an example of a learning model (learning model 1) that has relatively low accuracy but can execute object estimation in a shorter period. R-CNN is an example of a learning model (learning model 2) that has a relatively long period but can execute object estimation with higher accuracy. In the above example, YOLO, R-CNN, SSD300, SSD512, and Tiny-YOLO are described as examples of learning models for estimating objects, but the present invention is not limited to these, and other learning models that satisfy the relationship between accuracy and period may be used.

[0040] In step S502, the learning unit 150 accepts learning data. FIG. 6 shows an example of learning data in the learning stage. In the learning data, an image (input data) is linked to a tag (teaching data) of a target object contained in the image. It is preferable that the image, which is the input data, contains one target object (object), and the image size is preferably the same among a plurality of learning data. In the example of this embodiment, in order to perform automatic tracking and photographing of an aircraft at an airport, an image containing an aircraft as a subject as input data is used as the learning data to train a learning model. In addition, the image of the target object may be photographed and recorded in advance using the pan head device 200. The teaching data among the learning data is set by a person visually determining what the target object is from an image containing the above-mentioned target object.

[0041] Returning to FIG. 5 again, in step S503, the learning unit 150 judges whether the received learning data complies with a predetermined rule (for example, the format of the learning data shown in FIG. 6). If the learning unit 150 judges that the data complies with the rule, the process proceeds to step S504, otherwise the process returns to step S502. In step S504, the learning unit 150 inputs the received data learning data into a learning model, and in step S505, the learning model is trained. The learning of the learning model is executed as described with reference to FIG. 4.

[0042] In step S506, the learning unit 150 determines whether all the learning data has been learned, and if all the learning data has been learned, the process ends, otherwise the process returns to step S502. Through the above process, the internal parameters of the learning model 403 are determined, and a learning model that is the basis for the estimation process performed by the estimation unit 155 is generated.

[0043] The process from step S507 onwards is executed when the above-mentioned operation mode is the manual mode or the tracking mode. In step S507, the learning unit 150 determines whether the current operation mode is the manual mode or not, and if it is determined that the operation mode is the manual mode, the process proceeds to step S508, otherwise, the process proceeds to step S510.

[0044] In step S508, the camera platform control unit 157 receives a command from the operation device 300, and then in step S509, transmits the received command to the camera platform device 200, thereby terminating this process. In step S510, some processing units of the information processing device 100 execute a tracking mode process, which will be described later.

[0045] <Automatic tracking operation in the automatic tracking system> Next, with reference to Fig. 7, an automatic tracking operation in the case where the learning model shown in Fig. 4 is applied to an automatic tracking system will be described. First, in S701, a tracking start command is transmitted from the operation device 300 to the information processing device 100. The tracking start command is executed by a user operating a predetermined operation unit 302 of the operation device 300. Next, in S702, the information processing device 100 transmits a tracking start position command to the camera-to-camera device 200. The tracking start position is the pan, tilt, and zoom position of the camera-to-camera device 200 at which automatic tracking shooting starts, and is set in advance by the user operating the operation unit 302.

[0046] In S703, the camera platform 200 that has received the tracking start position command moves the pan, tilt, and zoom to the start positions. When the pan, tilt, or other positions reach the start positions, the camera platform 200 transmits, in S704, to the information processing device 100 a video signal indicating that the start positions have been reached.

[0047] In S705, the information processing device 100 starts a recording process for recording the video signal from the tripod head device 200 in the storage unit 105, and then in S706-1, an estimation process is performed from within the received video signal. Furthermore, in S706-2, the information processing device 100 calculates a tripod head control command using the estimation result processing unit 156 and the tripod head control unit 157, and in S707, transmits the calculated control command to the tripod head device 200.

[0048] In S708, the camera platform device 200 controls panning, tilting, and the like in accordance with the received control command, and in S709 transmits a captured image signal to the information processing device 100. Thereafter, the processes of S706-1 to S709 are executed, thereby enabling automatic tracking and capturing of a target subject.

[0049] Next, a tracking mode process (estimation process in the estimation stage) in the information processing device 100 will be described with reference to Fig. 8. Note that the CPU 103 and the GPU 102 execute a program stored in the storage unit 105 to realize the operation of the estimation unit 155 and the like in this process.

[0050] In step S800, the learning model setting unit 152 acquires the pan and tilt speeds transmitted to the camera-head device 200 from the camera-head control unit 157. Next, in step S801, the learning model setting unit 152 determines whether the pan or tilt speed of the camera-head device 200 is equal to or greater than a predetermined value. The predetermined value may be a numerical value determined in advance by experiments or the like, and is determined as the pan and tilt speeds at the time of switching of the tracking state by measuring the tracking state and the pan and tilt speeds during tracking. The relationship between the tracking state, the pan and tilt speeds, and the learning model to be used will be described later in the explanation of FIG. 9. If the pan and tilt speeds are equal to or greater than the predetermined value, the learning model setting unit 152 proceeds to step S802, and if not, proceeds to step S808.

[0051] In step S802, the learning model setting unit 152 switches to learning model 1. Meanwhile, in step S808, the learning model setting unit 152 switches to learning model 2. The learning model setting unit 152 selects the learning model to be switched to from among a plurality of learning models stored in the information processing device 100. That is, when the speed of the pan or tilt operation of the camera head device 200 is equal to or greater than a predetermined value, the learning model setting unit 152 uses a learning model (learning model 1) that takes less time to complete estimation of the subject. Also, when the speed of the pan or tilt operation of the camera head device 200 is smaller than a predetermined value, the learning model (learning model 2) that has a higher accuracy in estimating the subject is used.

[0052] In step S803, the image processing unit 154 changes (resizes) the size of the image received from the camera-to-camera device 200. By reducing the size of the input image, it is possible to reduce the processing time of the estimation unit 155. In step S804, the image processing unit 154 inputs the resized image data to the learning model.

[0053] In step S805, the estimation unit 155 performs estimation processing. In step S806, the estimation result processing unit 156 determines whether the estimation target is included based on the result of the estimation processing. In this embodiment, for example, the estimation target is an aircraft. If the estimation target is included, the estimation result processing unit 156 proceeds to step S807, and if not, the estimation result processing unit 156 proceeds to step S890. The processing of S807 will be described later.

[0054] In step S890, the camera-head control unit 157 transmits a stop command to the camera-head device 200, and the process proceeds to step S891. In step S891, the learning model setting unit 152 sets the current operation mode to manual mode, and ends this process. In this manner, when the estimation unit 155 does not output an estimation target, the information processing device 100 determines that the tracking target has disappeared from the photographable range, and ends the tracking mode.

[0055] Here, the change of the learning model will be described with reference to Fig. 9(a) to (c). Fig. 9(a) to (c) show several stages at which the learning model can be switched, taking as an example the case of tracking a landing aircraft. In the example of Fig. 9, the pan and tilt speed required for tracking, and the period and accuracy of the required subject estimation differ depending on the stage of the aircraft's landing. In the initial setting, when tracking the landing of an aircraft, the zoom is set to the telephoto side, and the pan and tilt are set so that the flight path of the aircraft is within the angle of view and the aircraft is stopped. At this time, a learning model (learning model 2) with a long period for estimating the subject but high accuracy for estimating the subject is set.

[0056] FIG. 9(a) shows the stage where an aircraft is approaching the airport to land on the runway. In this case, the aircraft is away from the camera platform on the roof of the airport, and even if the zoom is moved to the telephoto side, the aircraft is small within the angle of view of the camera platform. For this reason, a highly accurate learning model capable of estimating small objects is required, but a highly accurate learning model has a long period for estimating the object (i.e., it takes time to estimate). However, as described above, since the aircraft is small within the angle of view of the camera platform, the relative movement speed of the aircraft within the angle of view is small. Therefore, the pan and tilt speed required for tracking the object is low. At this stage, even if the period for estimating the object is long, the relative movement speed of the aircraft is small, so the impact on tracking is small. Therefore, by using a highly accurate learning model 2 at the stage where the aircraft is approaching the airport to land on the runway, it is possible to optimize the period and accuracy of estimating the object.

[0057] FIG. 9(b) shows the stage where the aircraft lands and runs on the runway. In this case, the aircraft is close to the camera platform on the roof of the airport, and even when the zoom is moved to the wide side, the relative speed of the aircraft within the angle of view is large. Therefore, the pan and tilt speed required for tracking is large. When the relative moving speed of the aircraft is large, the estimation delay has a large impact on tracking. For this reason, a learning model with a short period for estimating the subject is required. A learning model with a short period for estimating the subject has low accuracy in estimating the subject, and the estimation accuracy of the subject when the image is unclear or small decreases. However, as mentioned above, the aircraft is large within the angle of view of the camera of the camera platform, so the aircraft can be detected even when a learning model with relatively low accuracy in estimating the subject is used. Therefore, when the aircraft lands and runs on the runway, it is possible to optimize the period and accuracy of estimating the subject by using a learning model with a short period for estimating the subject (learning model 1).

[0058] Figure 9(c) shows the stage where an aircraft lands and moves from the runway to the airport terminal. In this case, the aircraft's physical speed is low, and the pan and tilt speed required for tracking is also low. Also, even when the zoom is moved to the wide side, the aircraft is large within the angle of view of the camera on the platform. At this stage, even if there is an estimation delay in the subject estimation result, the impact on tracking is small because the aircraft's speed is low. Therefore, a learning model with high accuracy for estimating the subject (learning model 2) can be used.

[0059] In this way, the pan and tilt speeds required for tracking and the period and accuracy of subject estimation differ depending on the stage of the landing of the aircraft. In this embodiment, the pan and tilt speeds are referenced, rather than the size and speed of the aircraft within the angle of view. This makes it easier to determine whether to switch the learning model, since the speed of the aircraft within the angle of view is a relative value, while the pan and tilt speeds are absolute values. When the camera platform device tracks an aircraft, the speed of the aircraft that can be calculated from the angle of view is a relative speed, and it is difficult to calculate the accurate speed of the aircraft. Although it is possible to calculate the speed from the background by performing background separation with the aircraft, in the case of shooting an aircraft, the background is often a blue sky or the like, which is difficult to separate. In contrast, the pan and tilt speeds are absolute values ​​and can be obtained from the operation device 300, so it is easy to determine whether to switch the learning model.

[0060] Returning to FIG. 8 again, in step S807, pan and tilt control commands are calculated based on the difference between the coordinates of the estimated result and the tracking target position. Then, a zoom control command is calculated based on the size of the estimated result and the tracking target size, and both signals are transmitted to the camera platform device 200. The tracking target position indicates a position in a predetermined image where the subject should be captured when tracking the subject. The tracking target size indicates a size in a predetermined image where the subject should be captured when tracking the subject. The tracking target position and the tracking target size may be registered in advance in the information processing device 100. As an example, the tracking target position may be set to the center coordinates in the screen, and the tracking target size may be set to 30% of the screen size, but this is not limited thereto. Also, the tracking target position and the tracking target size may be configured to be set in the information processing device 100 from the operation device 300.

[0061] Through the above processing, the pan head device 200 is rotated and the magnification is controlled according to the control command, and the image under control is input again to the information processing device 100, whereby the processing is repeated and automatic tracking shooting can be realized. In this way, in the processing of the tracking mode, the optimal learning model for executing the processing of the estimation stage can be changed based on the pan and tilt speed of the pan head device 200.

[0062] In the above embodiment, the case where an aircraft is automatically tracked and photographed while landing is described as an example. However, the present invention is not limited to this, and the aircraft may be automatically tracked and photographed while taking off.

[0063] In addition, the initial value of the learning model may be determined according to the calculated speed after calculating the speed of the aircraft from the frame difference of the video input before the pan, tilt, or zoom operation. Furthermore, the learning model may be switched by temporarily estimating the subject using two learning models, and switching may be performed when the estimation results of both models become approximately constant.

[0064] In the present embodiment, the case where the information processing device 100 switches the learning model by referring to the pan and tilt speed of the camera head device, for example, when an aircraft lands, has been described. However, the present invention is not limited to this, and the information processing device 100 may switch the learning model by referring to the zoom speed.

[0065] Furthermore, in this embodiment, the information processing device 100 and the pan head device 200 are connected by a wired video signal line, but the present invention is not limited to this and may be connected by a communication line such as a public telephone line or the Internet, or by wireless communication.

[0066] In addition, among the processing units of the information processing device 100 described above, the estimation unit 155 is configured to execute processing using a trained learning model. However, a predetermined deterministic rule-based processing may be performed using a look-up table (LUT) or the like. In that case, for example, the relationship between input data and output data is created in advance as an LUT. Then, the created LUT may be stored in the storage unit 105 of the information processing device 100. When performing processing of the estimation unit 155, the output data can be acquired by referring to the stored LUT. In other words, the LUT may be a program for performing the same processing, and may operate in cooperation with a CPU or a GPU to perform the processing of the estimation unit 155.

[0067] As described above, in this embodiment, the information processing device 100 receives a video captured while controlling the pan or tilt of the camera of the camera platform device 200, and estimates a subject included in an image constituting the video. At this time, the subject included in the image is estimated using a machine learning model (i.e., learning model 1 or learning model 2) with different characteristics according to the speed of the pan or tilt operation of the camera of the camera platform device 200 when capturing the image. In this way, it becomes possible to estimate the subject with an appropriate period and accuracy when tracking and capturing a moving subject.

[0068] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0069] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0070] 100: information processing device, 150: learning unit, 152: learning model setting unit, 153: mode management unit, 154: image processing unit, 155: estimation unit, 156: estimation result processing unit, 157: camera head control unit

Claims

1. A receiving means for receiving a moving image captured while controlling the pan or tilt of the imaging means; an estimation means for estimating a subject included in an image constituting the moving image, The estimation means estimates a subject included in the image by using a machine learning model having different characteristics according to a speed of a pan or tilt operation of the imaging means when capturing the image, The information processing device, characterized in that the characteristics include a time from when the machine learning model inputs the image to when it completes estimating the subject, and an accuracy with which the machine learning model estimates the subject.

2. The information processing device according to claim 1, characterized in that the machine learning models with different characteristics include a first machine learning model and a second machine learning model, and the first machine learning model has a higher accuracy in estimating the subject than the second machine learning model and takes a longer time to complete the estimation than the second machine learning model.

3. The information processing device described in claim 2, characterized in that the estimation means uses the second machine learning model that takes a shorter time to complete the estimation than the first machine learning model when the speed of the panning or tilting operation of the imaging means is greater than or equal to a predetermined value, and uses the first machine learning model that has a higher accuracy in estimating the subject than the second machine learning model when the speed is less than the predetermined value.

4. The information processing device according to claim 2 or 3, characterized in that the first machine learning model, which has a high accuracy in estimating the subject, is capable of estimating the subject even if the size of the subject in the image is smaller than a predetermined size.

5. The imaging means is further capable of controlling zoom, The information processing device according to any one of claims 1 to 4, characterized in that the estimation means estimates the subject included in the image using a machine learning model having different characteristics according to a zoom operation speed of the imaging means when capturing the image.

6. 6. The information processing device according to claim 1, further comprising a control unit that generates a control signal for controlling panning or tilting of the imaging unit based on a difference between the estimated position of the subject in the image and a predetermined position in the image at which the subject should be imaged when tracking the subject.

7. 6. The information processing device according to claim 5, further comprising a control means for generating a control signal for controlling a zoom of the imaging means based on a difference between the estimated size of the subject in the image and the size of the subject in a predetermined image in which the subject is to be imaged when tracking the subject.

8. 8. The information processing apparatus according to claim 6, further comprising a communication means for transmitting the control signal generated by the control means to the imaging means.

9. The information processing device according to claim 1 , wherein the machine learning model includes a model configured with a deep neural network.

10. a receiving step of receiving a moving image captured while controlling the pan or tilt of the imaging means; an estimation step of estimating a subject included in an image constituting the moving image, In the estimation step, a subject included in the image is estimated using a machine learning model having different characteristics according to a speed of a pan or tilt operation of the imaging means when capturing the image, A method for controlling an information processing device, characterized in that the characteristics include a time from when the machine learning model inputs the image to when it completes estimating the subject, and an accuracy with which the machine learning model estimates the subject.

11. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 9.

Images