Information processing device, control method thereof, and program

Abstract

To provide an information processing device, a control method thereof, and a program that can track a target subject during shooting without requiring special learning even in a case in which the detection accuracy of a subject is reduced.SOLUTION: A CPU 103 of an information processing device 100 acquires a frame of a moving image taken by a camera 201, and an estimation unit 155 estimates an object existing in the frame. In a case in which output data 401 from the estimation unit 155 has a preset tracking object tag, the CPU 103 calculates the position and object size of the tracking object in the frame on the basis of the output data 401, and resets the tracking object when the object size is less than or equal to a specified value after controlling the pan and tilt of the camera 201 from a difference between the calculated position and the tracking target position, and controlling the zoom of the camera 201 such that the object size approaches the tracking target size.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method thereof, and a program, and more particularly to an information processing device, a control method thereof, and a program for controlling tracking and photographing a target subject by a camera. [Background technology]

[0002] Conventionally, a pan-head system has been widely known, in which a user remotely controls a camera to obtain desired images. For example, the images of aircraft seen on television news are taken by remotely controlling a pan-head device permanently installed on the roof of an airport from a broadcasting station.

[0003] In addition, an automatic tracking system has been proposed that uses image recognition technology on the camera platform to recognize the subject in the image and automatically pan, tilt, and zoom to track the subject's movement. This makes it possible to automatically capture moving subjects without the user having to operate the controls.

[0004] In recent years, examples of image recognition technology using machine learning (AI) have become known. For example, Patent Document 1 proposes a system for detecting and monitoring a subject captured by a camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6184271 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the conventional technology disclosed in Patent Document 1, when the certainty (likelihood) of the machine learning processing result is low, the angle of the sub-camera is changed, so the main camera cannot track the subject and cannot capture continuous tracking video. If the subject detection accuracy can be improved and cases where the likelihood is reduced can be eliminated, tracking video can be captured by the main camera, but this requires a lot of learning.

[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an information processing device, a control method thereof, and a program that are capable of tracking a subject during shooting without requiring special learning, even in cases where the subject detection accuracy is reduced. [Means for solving the problem]

[0008] The information processing device according to claim 1 of the present invention controls pan, tilt, and zoom of an image capturing device that captures a moving image, thereby capturing a target subject by the image capturing device. of An information processing device for tracking, comprising: an acquisition means for acquiring frames of a moving image captured by the imaging device; an estimation means for estimating an object present in the acquired frames; and, when the estimated object is a preset tracking object, The imaging device is configured to track an object present in the acquired frame. and a control means for controlling the above-mentioned. When the size of the object present in the acquired frame is equal to or smaller than a first threshold value. , the tracking object For an object whose actual size is smaller than the preset tracking object Reset Characterized by . Effect of the Invention

[0009] According to the present invention, even in cases where the accuracy of detecting the subject is reduced, it is possible to track the subject during shooting without requiring special learning. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a configuration of an automatic tracking system including an information processing device according to an embodiment of the present invention. [Diagram 2]2 is a diagram showing the hardware configurations of an information processing device, a camera platform device, and an operation device appearing in FIG. 1. [Diagram 3] 2 is a diagram showing a software configuration realized by using hardware resources and programs included in the information processing device, the camera platform device, and the operation device in FIG. 1. FIG. [Figure 4] FIG. 4 is a conceptual diagram of an estimation process in an estimation unit in FIG. 3. [Diagram 5] 10 is a flowchart showing a detailed flow of processing in a learning phase according to the embodiment of the present invention. [Figure 6] 4 is a diagram showing an example of learning data input to a learning unit appearing in FIG. 3; FIG. [Figure 7] 5 is a diagram illustrating an automatic tracking and photographing operation in an automatic tracking system using the trained model in FIG. 4. FIG. [Figure 8] 10 is a flowchart showing a detailed flow of processing in an estimation phase according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] FIG. 1 is a diagram showing an automatic tracking system 1 including an information processing device 100 according to an embodiment of the present invention.

[0012] In FIG. 1, the automatic tracking system 1 includes an information processing device 100, a camera platform device 200, an operation device 300, and a network NET.

[0013] The operation device 300 is a device that accepts user operations, and transmits a command (control signal) corresponding to the accepted user operation to the pan head device 200 via the network NET and the information processing device 100.

[0014] The pan / tilt head device 200 operates according to the contents of the control signal transmitted from the operation device 300. This allows the user to remotely control the pan / tilt head device 200. Images captured by a camera 201 (FIG. 2) provided in the pan / tilt head device 200 are input to the information processing device 100, where various calculations and recordings required for automatic tracking shooting are performed.

[0015] The network NET is a communication line such as a public telephone line or the Internet. The camera platform device 200 and the information processing device 100 are installed at spots such as an airport, a steel tower, or the rooftop of a television station, and the operation device 300 is installed inside the television station, etc. In the following embodiment, a case will be described in which the information processing device 100 and the camera platform device 200 are installed at an airport, and the object to be automatically tracked and photographed by the camera platform device is an airplane.

[0016] FIG. 2 is a diagram showing the hardware configuration of the information processing device 100, the camera platform device 200, and the operation device 300. As shown in FIG.

[0017] The information processing device 100 includes a RAM 101, a GPU 102, a CPU 103, an input unit 104, a storage unit 105, a serial communication unit 106, a network communication unit 107, and a UI unit 108.

[0018] The camera head device 200 includes a camera 201 , a driving unit 202 , a serial communication unit 203 , a CPU 204 , and a storage unit 205 .

[0019] The operation device 300 includes a network communication unit 301 , an operation unit 302 , a storage unit 303 , a CPU 304 , and a display unit 305 .

[0020] The hardware resources of the information processing device 100 will be described in detail below.

[0021] The RAM 101 is a volatile memory, and is used as a temporary storage area such as a main memory and a work area for the CPU 103.

[0022] The CPU 103 controls each unit of the information processing device 100 according to a program stored in the storage unit 105, for example, using the RAM 101 as a work memory.

[0023] Since the GPU 102 can perform efficient calculations through parallel processing of data, the process of learning multiple times using a learning model through deep learning, as described below, is performed by the GPU 102.

[0024] The input unit 104 (acquisition means) is an interface for inputting a video signal to the information processing device 100 via a wired video signal line, and is, for example, one of various communication interfaces such as a USB.

[0025] The storage unit 105 is a non-volatile memory, and image data and other data, various programs for operating the CPU 103, and the like are stored in respective predetermined areas. The storage unit 105 is configured by, for example, a magnetic disk such as an HDD or a flash memory.

[0026] The serial communication unit 106 is an interface for performing serial communication with the camera platform device 200 under the control of the CPU 103 .

[0027] The network communication unit 107 is a communication interface for communicating with the operation device 300 via the network NET under the control of the CPU 103 .

[0028] The UI unit 108 is a user interface for accepting user operations on the information processing device 100 and displaying information of the information processing device 100 to the user. Specifically, the UI unit 108 is configured with a touch panel, but may further include a keyboard, a mouse, a display, and the like.

[0029] The hardware resources of the camera head device 200 will be described in detail below.

[0030] The camera 201 (imaging device) captures the surroundings around where the pan head device 200 is installed, and captures an image of a target subject. The camera 201 is equipped with an optical zoom lens capable of changing the imaging magnification. That is, the pan head device 200 has an optical zoom function that changes the magnification of the captured image by transmitting an optical zoom control command for the optical zoom lens from the CPU 204 to the camera 201. Furthermore, the pan head device 200 also has a digital zoom function that changes the magnification of the captured image by transmitting a digital zoom control command for locally enlarging a part of the captured image from the CPU 204 to the camera 201. The digital zoom function is used when the magnification is insufficient with the optical zoom function, that is, when it is desired to further enlarge the captured image. Moreover, the camera 201 is connected to the input unit 104 of the information processing device 100 by a wired video signal line, and outputs the captured video signal to the information processing device 100.

[0031] The driving unit 202 is an actuator and its driving circuit and peripheral circuits for rotating the camera platform device 200 in the pan and tilt directions. That is, the camera platform device 200 has a tracking shooting function for shooting a tracking image by transmitting a tracking control command from the CPU 204 to the driving unit 202 for rotating in the pan and tilt directions with respect to a target subject.

[0032] The serial communication unit 203 is connected to the serial communication unit 106 of the information processing device 100 , and is an interface for performing serial communication with the information processing device 100 based on the control of the CPU 204 .

[0033] The CPU 204 controls each part of the camera-head device 200 in accordance with a program stored in the storage unit 205, for example.

[0034] The storage unit 205 is a non-volatile memory, and stores setting data for the pan head device 200, other data, various programs for operating the CPU 204, and the like, in respective predetermined areas.

[0035] The hardware resources of the operation device 300 will now be described in detail.

[0036] The network communication unit 301 is a communication interface for communicating with the information processing device 100 via the network NET under the control of the CPU 304 .

[0037] The operation unit 302 is a joystick, an operation lever, and various switches, and the user operates the operation unit 302 to perform rotation control, zoom control, gain adjustment, etc. of the camera head device 200. Note that the zoom control here refers to zoom control using the optical zoom function and digital zoom function described above.

[0038] The storage unit 303 is a non-volatile memory, and stores setting data for the operation device 300, other data, various programs for the operation of the CPU 304, and the like in respective predetermined areas.

[0039] The CPU 304 controls each unit of the operation device 300 in accordance with a program stored in the storage unit 303, for example.

[0040] The display unit 305 includes an LED and a touch panel, and notifies the user of the status of the camera head device 200, warnings, and the like.

[0041] Each of the CPU 103, CPU 204, and CPU 304 may be configured with one or more processors.

[0042] FIG. 3 is a diagram showing a software configuration realized by using the hardware resources and programs provided in the information processing device 100, the camera head device 200, and the operation device 300 shown in FIG.

[0043] The software configuration of the information processing device 100 will be described below.

[0044] The information processing device 100 includes a learning unit 150 , a data storage unit 151 , an estimation target setting unit 152 , a mode management unit 153 , an image processing unit 154 , an estimation unit 155 , an estimation result processing unit 156 , and a camera head control unit 157 .

[0045] The learning unit 150 (learning means) executes a learning process for performing an estimation process in the estimation unit 155. The details of the learning process executed by the learning unit 150 will be described later.

[0046] The data storage unit 151 performs a process of recording images captured using automatic tracking and a process of recording learning data.

[0047] The estimation target setting unit 152 manages the tag of the object output by the estimation unit 155. Here, the tag is a part of the learning data input to the learning unit 150 and one of the data output by the estimation unit 155, and is a label indicating what the object is. Specific examples of tags include aircraft, dogs, cats, birds, etc. In the estimation target setting unit 152, the tag of the object that the user wants to automatically track and photograph can be set in advance, and it is also possible to set multiple tags. In this embodiment, in order to perform automatic tracking and photographing of an aircraft, the tag of the aircraft is set as the tag of the object to be automatically tracked and photographed.

[0048] The mode management unit 153 manages the mode of the information processing device 100, and manages three modes: a learning mode, an automatic tracking mode, and a manual mode. The details of each mode will be described later.

[0049] The image processing unit 154 processes the video signal (image; frame) received from the camera platform device 200. Specifically, the image processing unit 154 processes the received image such as resizing the image and adjusting the brightness.

[0050] The estimation unit 155 (estimation means) uses the image output from the image processing unit 154 as input data, inputs it to the trained model generated by the learning unit 150, and performs estimation processing. In this way, the subject (object) present in the frame received from the camera platform device 200 is estimated.

[0051] When the estimation result output from the estimation unit 155 includes a tracking target, the estimation result processing unit 156 performs various noise processing and averaging processing, and outputs the position of the tracking target in the image (current position of the subject). The image captured by the camera platform device 200 contains noise other than the tracking target (aircraft other than aircraft, parts of the background, clouds, etc. that can be mistaken for aircraft). The estimation result processing unit 156 has a role of processing these noises and outputting a highly reliable current position of the subject. Furthermore, if other tags are set in addition to the aircraft in the estimation target setting unit 152 described above, the noise processing in the estimation result processing unit 156 may be burdened, or the reliability of the current position of the subject to be output may decrease. For this reason, it is preferable to set the tags in the estimation target setting unit 152 to the minimum necessary.

[0052] The camera-head control unit 157 calculates a control signal for controlling the camera-head device 200 according to the above-mentioned mode, and outputs it to the camera-head device 200. When the current mode is the automatic tracking mode, the camera-head control unit 157 calculates a control signal for controlling the camera-head device 200 based on the current position of the subject and the tracking target position (the position where the subject is to be held within the shooting screen during tracking shooting), and outputs it to the camera-head device 200. When the current mode is the manual mode, the camera-head control unit 157 outputs a control signal transmitted from the operation device 300 to the camera-head device 200.

[0053] For processing by the learning unit 150, the GPU 102 is used in addition to the CPU 103. Specifically, when a learning program including a learning model is executed, the CPU 103 and the GPU 102 cooperate to perform calculations to perform the learning processing. Similarly to the learning unit 150, the estimation unit 155 also performs estimation processing by having the GPU 102 cooperate to perform calculations in addition to the CPU 103. For processing by the learning unit 150 and the estimation unit 155, calculations may be performed only by the CPU 103 or the GPU 102 depending on the calculation processing capacity of the CPU 103 or the GPU 102.

[0054] The software configuration of the camera head device 200 will be described below.

[0055] The camera platform device 200 includes a pan / tilt control unit 250 , a camera control unit 251 , a setting management unit 252 , and a communication unit 253 .

[0056] The pan / tilt control unit 250 outputs a signal for driving the pan and tilt to the driving unit 202 based on a control signal for controlling the driving of the driving unit 202 received by the communication unit 253 .

[0057] The camera control unit 251 outputs a signal for controlling the camera 201 to the camera 201 based on a control signal for controlling the camera 201 received by the communication unit 253 .

[0058] The setting management section 252 manages the settings of the operation device 300. Specific setting items managed by the setting management section 252 include drive parameters such as the maximum pan and tilt speeds and the drivable range.

[0059] The communication unit 253 exchanges control commands and status information with the camera platform control unit 157 in accordance with a predetermined communication rule (protocol).

[0060] The software configuration of the operation device 300 will now be described.

[0061] The operation device 300 includes a communication unit 350 and a display unit 351 .

[0062] The communication unit 350 exchanges control signals and status information with the camera platform control unit 157 in accordance with a predetermined communication rule (protocol).

[0063] FIG. 4 is a conceptual diagram of the estimation process in the estimation unit 155. As shown in FIG.

[0064] As shown in FIG. 4 , the estimation unit 155 inputs input data 400 to a trained model 403 to execute an estimation process, and outputs output data 401 from the trained model 403.

[0065] The input data 400 is image data captured by the camera 201 of the camera platform device 200 and processed by the image processing unit 154. Note that since the camera platform device 200 captures moving images, the data actually input to the trained model 403 as the input data 400 is one frame of the moving image.

[0066] The output data 401 includes tags, coordinates, and likelihoods of objects estimated to be present in the images that are the input data 400.

[0067] The tag of the estimated object is selected from tags included in the learning data input during learning. The learning data will be described in detail later.

[0068] The coordinates of the estimated object are the upper left coordinate (coordinate 1) and the lower right coordinate (coordinate 2) of the bounding box of the estimated object, as shown in image 402. From the coordinates of these two points, the object size and center point coordinates can be calculated.

[0069] The likelihood of an estimated object is a value between 0 and 1, and the larger the value, the higher the reliability of the estimation result by trained model 403 for the output tag.

[0070] The trained model 403 is a neural network, and its internal parameters are generated by machine learning of the training model by the training unit 150.

[0071] The learning unit 150 includes an error detection unit and an update unit.

[0072] The error detection unit obtains an error between the teacher data and output data output from the output layer of the neural network in response to input data input to the input layer. The error detection unit may use a loss function to calculate the error between the output data from the neural network and the teacher data.

[0073] The update unit updates the connection weighting coefficients between the nodes of the neural network based on the error obtained by the error detection unit so as to reduce the error. The update unit updates the connection weighting coefficients using, for example, an error backpropagation method. The error backpropagation method is a method of adjusting the connection weighting coefficients between the nodes of each neural network so as to reduce the error.

[0074] FIG. 5 is a flowchart showing a detailed flow of processing in the learning phase according to this embodiment.

[0075] First, in step S500, CPU 103 determines whether or not the current mode managed by mode management unit 153 is the learning mode. As a result, if CPU 103 determines that it is the learning mode, the process proceeds to step S501, and if not, the process proceeds to step S506.

[0076] In step S501, CPU 103 reads one piece of learning data from data storage unit 151, and proceeds to step S502. Here, the learning data will be described with reference to FIG.

[0077] FIG. 6 is a diagram showing an example of learning data input to the learning unit 150. As shown in FIG.

[0078] The learning data is data in which an image and a tag (teaching data) of an object contained in the image are linked. Note that an image preferably contains one object, and the image size is preferably the same among a plurality of learning data. In this embodiment, in order to perform automatic tracking and photographing of an aircraft at an airport, an image of the aircraft is prepared as input data, and in addition, an image of another object that is likely to be captured by the camera 201 of the camera platform 200 during automatic tracking and photographing is prepared. Specific examples of the other objects include birds and kites. Furthermore, images of these objects may be photographed and recorded in advance using the camera platform 200. Among the learning data, the teaching data is set by visually determining what the object is from the image of the object described above. FIG. 6 illustrates learning data 601 to 603, and shows examples in which an aircraft, a bird, and a kite are objects contained in the image that is the input data.

[0079] 5, in step S502, CPU 103 determines whether the learning data read from data storage unit 151 conforms to a predetermined rule (is regular). If CPU 103 determines that the learning data conforms to the rule, the process proceeds to step S503, and if not, the process returns to step S501.

[0080] In step S503, CPU 103 inputs the learning data read from data storage unit 151 to a learning model, which is a neural network in learning unit 150.

[0081] In step S504, the CPU 103 and the GPU 102 cooperate to perform a learning process in the learning unit 150, and update the internal parameters of the learning model.

[0082] Next, in step S505, CPU 103 determines whether learning based on all learning data recorded in data storage unit 151 has been completed, and if all learning has been completed, ends this process, otherwise returns to step S501.

[0083] Through the above processing, the internal parameters of the trained model 403 on which the estimation process is performed by the estimation unit 155 are determined.

[0084] The process from step S506 onwards shows the flow of the process when the above-mentioned mode is the manual mode or the automatic tracking mode.

[0085] In step S506, CPU 103 determines whether the current mode is the manual mode or not. As a result, if CPU 103 determines that the current mode is the manual mode, the process proceeds to step S507, and if not, the process proceeds to step S509.

[0086] In step S507, the CPU 103 receives a control signal (command) from the operation device 300, and proceeds to step S508.

[0087] In step S508, the CPU 103 transmits the received control signal to the camera-head device 200, and ends this process.

[0088] In step S509, the CPU 103 executes an automatic tracking mode process, which will be described later, and then ends this process.

[0089] FIG. 7 is a diagram for explaining the automatic tracking and photographing operation in the automatic tracking system 1 using the trained model 403 in FIG.

[0090] First, in step S701, a control signal for a tracking start command is transmitted from the operation device 300 to the information processing device 100. Note that this control signal is generated by the CPU 304 and transmitted to the information processing device 100 when a predetermined user operation is performed on the operation unit 302 of the operation device 300.

[0091] Next, in step S702, a control signal of a tracking start position command is transmitted from the information processing device 100 to the camera platform device 200. The tracking start position is a pan, tilt, and zoom position of the camera platform device 200 at which automatic tracking shooting starts, and is stored in advance in the storage unit 205 of the camera platform device 200 by the user operating the operation unit 302.

[0092] In step S703, the CPU 204 of the camera head device 200 that has received the tracking start position command moves the pan, tilt, and zoom positions to the tracking start positions.

[0093] In step S704, when the pan, tilt, and zoom positions reach the tracking start position, the CPU 204 of the camera head device 200 transmits to the information processing device 100 a signal indicating that the tracking start position has been reached and a video signal captured by the camera 201.

[0094] In step S 705 , the CPU 103 of the information processing device 100 that has received the video signal from the pan-and-tilt device 200 starts a recording process for recording the video signal from the pan-and-tilt device 200 in the storage unit 105 .

[0095] Next, in step S706-1, the CPU 103 cooperates with the GPU 102 to perform an estimation process for estimating a subject included in the received video signal (image).

[0096] Furthermore, in step S706-2, CPU 103 calculates a camera head control command using estimation result processing unit 156 and camera head control unit 157, and in step S707, CPU 103 transmits a control signal for the calculated camera head control command to camera head device 200.

[0097] In step S708, the CPU 204 of the camera head device 200 controls the pan and tilt positions in accordance with the control signal received by the drive unit 202, and also controls the zoom of the camera 201. After that, in step S709, the CPU 204 transmits a video signal captured by the camera 201 to the information processing device 100.

[0098] Thereafter, the processes of steps S706-1 to S709 are executed, thereby enabling automatic tracking and photographing of the target subject.

[0099] Fig. 8 is a flowchart showing a detailed flow of processing in the estimation phase according to this embodiment. Note that this processing is also the automatic tracking mode processing in step S509 in Fig. 5. Therefore, hereinafter, this processing will be referred to as the automatic tracking mode processing.

[0100] The automatic tracking mode starts, and the CPU 103 receives images (moving image frames) from the camera-head device 200 at the input unit 104 of the information processing device 100 in step S800a.

[0101] In step S800, the CPU 103 changes (resizes) the size of the received image in the image processing unit 154. The larger the size of the image data input to the trained model 403 as the input data 400, the longer the estimation process takes for the estimation unit 155. That is, the process of step S800 aims to reduce the time required for the estimation process by reducing the size of the input data 400. In this embodiment, the input data 400 is resized so that the time required for the estimation process after the size change in step S800 is about 50 ms. However, this is not limited to the time required for the estimation process, since it is related to the computational processing capabilities of the CPU 103 and the GPU 102.

[0102] Next, in step S801, the CPU 103 acquires a tag (tracking object) to be an estimation target from the estimation target setting unit 152. In this embodiment, since an aircraft is set as an initial value of the estimation target setting unit 152, in step S801, an aircraft is acquired as the tag.

[0103] Next, in step S802, the CPU 103 inputs the input data 400 resized in step S800 to the trained model 403.

[0104] In step S803, the CPU 103 and the GPU 102 cooperate to execute the estimation process in the estimation unit 155, and output the output data 401.

[0105] Next, the process proceeds to step S804, where CPU 103 determines whether the tag acquired in step S801 is included in output data 401. As a result, if CPU 103 determines that the tag acquired in step S801 is included in output data 401, the process proceeds to step S805, and if not, the process proceeds to step S890.

[0106] In step S890, the CPU 103 transmits a control signal for commanding the camera to stop to the pan-and-tilt head device 200, and the process proceeds to step S891.

[0107] In step S891, CPU 103 changes the current mode managed by mode management section 153 to the manual mode, and ends the automatic tracking mode process.

[0108] Thus, in step S804, if the output data 401 from the estimation unit 155 does not include the tag set by the estimation target setting unit 152, the CPU 103 determines that the tracking target has disappeared from the captureable range and terminates the automatic tracking mode.

[0109] In step S805, the CPU 103 (calculation means) calculates the object size and center point coordinates of the estimated object in the screen from the upper left coordinates and lower right coordinates of the circumscribing frame of the estimated object included in the output data 401. Next, the CPU 103 (control means) calculates a control command for panning and tilting based on the difference between the calculated center point coordinates and the tracking target position, and transmits a control signal of the command to the camera-head device 200. Similarly, the CPU 103 (control means) calculates a control command for zooming based on the calculated object size and tracking target size, and transmits a control signal of the command to the camera-head device 200. The tracking target position and the tracking target size may be registered in advance in the storage unit 105 of the information processing device 100. In this embodiment, the center coordinates in the screen are registered in advance as the tracking target position, and a size of 30% of the screen is registered in advance as the tracking target size, but this is not limited thereto. The tracking target position and tracking target size may be configured to be set in the information processing device 100 from the operation device 300 .

[0110] Through the above-described processing, the pan head device 200 performs rotation and magnification control of the camera 201 in accordance with control signals of pan and tilt control commands and zoom control commands transmitted from the information processing device 100. In addition, an image captured by the camera 201 during such rotation and magnification control is transmitted again to the information processing device 100. Except for the cases described below (NO in step S808), the image transmitted again from the pan head device 200 is used as input data 400, and the processing from step S800a is repeated. This allows the CPU 103 to control automatic tracking shooting of the camera 201 based on the received frame every time the CPU 103 receives a frame from the camera 201.

[0111] Incidentally, when an aircraft taking off is photographed from a permanently installed pan head device 200, the aircraft moves away from the pan head device 200, so in the automatic tracking mode, the zoom control command in step S805 controls the aircraft to approach the tracking target size. However, if the camera 201 continues such automatic tracking and photographing, the size of the aircraft in the input data 400 becomes smaller, and image quality degradation occurs. As a result, the likelihood of the aircraft tag in the output data 401 from the estimation unit 155 decreases. In addition, if the camera 201 exceeds the optical zoomable area of ​​the camera 201 and zooms in on the aircraft in the input data 400 by digital zoom, image quality degradation occurs. This also reduces the likelihood of the aircraft tag in the output data 401 from the estimation unit 155. In such a case, even if an aircraft is present in the subject included in the image transmitted from the pan head device 200, the accuracy of estimating the subject as an aircraft by the trained model 403 decreases, making it difficult to continue automatic tracking and photographing.

[0112] Therefore, in the processing from step S806 onwards, as described below, the estimation target is reset according to at least one of the calculation result in step S805, the output data 401 obtained in step S803, and the meta information of the frame received in step S800a. This enables the information processing device 100 to control automatic tracking and photographing of the aircraft using the camera platform device 200.

[0113] In step S806, CPU 103 determines whether the object size calculated in step S805 is equal to or smaller than a predetermined value (equal to or smaller than a first threshold value). If CPU 103 determines that the object size calculated is equal to or smaller than the predetermined value, the process proceeds to step S809; otherwise, the process proceeds to step S807.

[0114] In step S807, CPU 103 acquires the likelihood of the object from output data 401, and if the likelihood is equal to or less than a predetermined value (equal to or less than a second threshold), proceeds to step S809, otherwise proceeds to step S808.

[0115] In step S808, CPU 103 determines whether or not there is a factor that degrades image quality in camera 201. As a result, if CPU 103 determines that there is a factor that degrades image quality in camera 201, the process proceeds to step S809, and if not, the process returns to step S800a. In this embodiment, CPU 103 determines that there is a factor that degrades image quality if camera 201 used digital zoom when capturing the frame received in step S800a. When acquiring a frame from camera 201, information on whether or not digital zoom was used when capturing the frame is also acquired as meta information for the frame.

[0116] The presence or absence of a factor of image quality degradation may be determined based on the luminance of the captured frame, the gain of the camera 201 at the time of capturing the frame, and the time of capturing the frame, either alone or in combination. For example, when acquiring a frame from the camera 201, the luminance, gain, and capture time at the time of capturing the frame are also acquired as meta information of the frame. This allows the CPU 103 to determine from the acquired meta information that there is a factor of image quality degradation if the luminance at the time of capturing the frame is low, the gain of the camera 201 is high, and the capture time is at night.

[0117] In step S809, the CPU 103 resets the estimation target managed by the estimation target setting unit 152. Specifically, in addition to the initial value of aircraft, a tag that is easily mistaken for an aircraft (in this embodiment, tags of birds and kites whose actual size is smaller than that of an aircraft) is set, and the process returns to step S800a. For tags that are easily mistaken for aircraft, a tracking target size smaller than that of an aircraft, here 10% of the screen size, is registered in advance. As a result, from the input data 400 based on the image (images from the next frame onwards) newly received from the camera platform device 200, the estimation unit 155 outputs the output data 401 including tags of birds and kites in addition to aircraft in step S803. Therefore, even if the estimation unit 155 outputs a tag other than an aircraft in step S804, that is, the estimation result is a result of misidentifying an aircraft as a bird or kite, it is possible to continue automatic tracking and photographing of the aircraft.

[0118] In this embodiment, when the estimation target managed by the estimation target setting unit 152 is reset, the CPU 103 (notification means) may notify the camera head device 200 of the reset via the serial communication unit 106. In this case, when the camera head device 200 receives such a notification, it changes the drive parameters stored internally. Specifically, when the estimation target is reset to a bird or a kite, its actual size will be smaller than that of an airplane, which is the initial value of the estimation target, and the amount of movement within the screen will also be smaller. Therefore, the camera head device 200 changes the drive parameters of pan, tilt, and zoom managed by the setting management unit 252, and realizes finer control than before the estimation target was reset.

[0119] In this embodiment, the determinations of steps S806, S807, and S808 are executed in this order, but it is also possible to determine at least one of these, or to execute them in a different order.

[0120] In the present embodiment, an example has been described in which automatic tracking and shooting is performed on a scene in which an aircraft is taking off. However, the present invention is not limited to this, and automatic tracking and shooting may be performed on a scene in which an aircraft is landing.

[0121] In this embodiment, the information processing device 100 and the camera platform device 200 are connected by serial communication and a wired video signal line, but the connection is not limited to this and may be made by a communication line such as a public telephone line or the Internet. For example, the information processing device 100 may be a cloud server.

[0122] Furthermore, in this embodiment, the learning unit 150 is also present in the information processing device 100, but the learning unit 150 may be present in an external device connected to the information processing device 100. In this case, the information processing device 100 acquires the internal parameters of the trained model 403 from the external device.

[0123] Among the above-mentioned processing units, the estimation unit 155 executes the estimation process using the trained model 403 that has been machine-learned, but rule-based estimation process such as a look-up table (LUT) may be performed. In that case, for example, the relationship between the input data 400 and the output data 401 is created in advance as an LUT. Then, the created LUT may be stored in the storage unit 105 of the information processing device 100. When the estimation unit 155 executes the estimation process, the output data can be acquired by referring to the stored LUT. That is, the LUT executes the estimation process of the estimation unit 155 by operating in cooperation with a CPU or a GPU as a program for performing the same process as the trained model 403.

[0124] Furthermore, the learning unit 150 may also learn by using as learning data object images linked to tags of objects other than the tracking target and objects that are easily mistaken for the tracking target (e.g., clouds; hereinafter, referred to as "incorrect tags"). In this case, when the estimation target is reset in step S809, incorrect tags are also added. Thereafter, if in step S804 the only tags included in the output data 401 are incorrect tags, the process proceeds to step S890, where the information processing device 100 determines that the tracking target has disappeared from the captureable range and ends the automatic tracking mode.

[0125] In this embodiment, the learning model for which the learning unit 150 performs the learning process and the trained model 403 for use in the estimation process of the estimation unit use a neural network algorithm, but other algorithms may be used. For example, a decision tree or an SVM (Support Vector Machine) algorithm may be used.

[0126] The present invention can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having a computer in the system or device read and execute the program. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.

[0127] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gateway (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU). [Explanation of symbols]

[0128] 100 Information processing device 103,204 CPU 106,203 Serial communication section 150 Learning Department 152 Estimation target setting unit 155 Estimation Department 157 Head control unit 200 Panhead device 201 Camera 202 Drive unit 300 Operating device NET Network

Claims

1. An information processing device that tracks a target subject by controlling pan, tilt, and zoom of an imaging device that captures a moving image, comprising: an acquisition means for acquiring frames of a moving image captured by the imaging device; An estimation means for estimating an object present in the acquired frame; a control unit that controls the imaging device so as to track an object present in the acquired frame when the estimated object is a preset tracking object; The information processing device is characterized in that the control means resets the tracking object to an object whose actual size is smaller than the predetermined tracking object when the size of an object present in the acquired frame is equal to or smaller than a first threshold.

2. The information processing apparatus according to claim 1 , wherein the control means resets the tracking object when the estimated object likelihood is equal to or smaller than a second threshold value.

3. 3 . The information processing apparatus according to claim 1 , wherein the control means resets the tracking object when it is determined that there is a factor causing deterioration in image quality in the imaging device based on meta information of the frame. 4 .

4. the imaging device performs the zooming using at least one of an optical zoom and a digital zoom; 4. The information processing device according to claim 3, wherein the control means determines that the factor of image quality degradation exists when the meta information includes information indicating that the digital zoom was used when the frame was captured by the imaging device.

5. The information processing device according to claim 3 or 4, characterized in that the control means determines that there is a factor of image quality degradation based on at least one of the brightness of the frame, the gain of the imaging device when the frame was photographed, and the time when the frame was photographed, which are included in the meta information.

6. 6. The information processing apparatus according to claim 1, further comprising a notification unit that, when the control unit resets the tracking object, notifies the imaging device of that effect.

7. 7. The information processing apparatus according to claim 6, wherein the imaging device changes driving parameters of pan, tilt and zoom when the imaging device receives the notification from the notification means.

8. The information processing apparatus according to claim 1 , further comprising: a learning unit that generates internal parameters of a trained model used for estimating the object in the estimation unit.

9. 9. The information processing apparatus according to claim 8, wherein the learning means uses learning data in which an image and teacher data indicating an object contained in the image are linked to each other.

10. The information processing device according to claim 8 or 9, characterized in that the estimation means, each time it receives a frame of the moving image from the imaging device, resizes the received frame as input data to the trained model, and outputs information about the object from the trained model as output data.

11. A method for controlling an information processing device to track a target subject by controlling pan, tilt, and zoom of an imaging device that captures a moving image, comprising: An acquisition step of acquiring frames of a moving image captured by the imaging device; an estimation step of estimating an object present in the acquired frame; and a control step of controlling the imaging device so as to track an object present in the acquired frame when the estimated object is a preset tracking object, A control method characterized in that the control step resets the tracking object to an object whose actual size is smaller than the predetermined tracking object when the size of an object present in the acquired frame is equal to or smaller than a first threshold.

12. A computer-executable program that causes a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 10.

Images