Information processing device, imaging system, information processing method, and program

Abstract

To enable accurate detection of the subject being tracked. [Solution] An information processing device (100) for detecting a subject to be tracked from a video image captured by an imaging device (200) and performing automatic tracking photography, comprising: an obstacle detection means (153) for detecting obstacles that may be mistaken for the subject from the video image by AI object detection; a registration means (154) for registering the video region of the obstacle detected by the obstacle detection means (153); an exclusion means (155) for excluding the video region of the obstacle registered by the registration means (154) from a detection region for detecting the subject in the video image; and a subject detection means (156) for detecting the subject from the detection region excluding the video region of the obstacle by AI object detection.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a photographing system, an information processing method, and a program suitable for use in performing automatic tracking photography.

Background Art

[0002] Conventionally, there has been widely known a photographing system in which a user remotely operates a pan-tilt device equipped with a camera from an operation device to acquire a desired video. For example, the video of an airplane seen on the news of a TV is photographed by remotely operating a pan-tilt device permanently installed on the roof of an airport from a broadcasting station. In this type of photographing system, an automatic tracking photographing system has been proposed that detects a subject appearing in the video and performs pan, tilt, and zoom operations in accordance with the movement of the subject to perform automatic tracking photography. Thereby, even if the user does not operate the operation device, it is possible to automatically track and photograph a moving subject. As a technique related to automatic tracking photography, Patent Document 1 proposes a technique for improving the tracking performance of an object when the load of the object detection process is reduced by restricting the area for detecting the object. In recent years, as a technique for detecting a subject appearing in a video, object detection using AI (Artificial Intelligence) (referred to as "AI object detection" in the present application) is known, and high-precision object detection has become possible.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] While AI object detection enables highly accurate object detection, it cannot distinguish between multiple objects with similar characteristics, and may detect obstacles that could be mistaken for the target object in addition to the target object. Even when the object detection area is limited, as disclosed in Patent Document 1, in the unrestricted area, obstacles that could be mistaken for the target object may be detected in addition to the target object.

[0005] This invention has been made in view of the above-mentioned points, and aims to enable accurate detection of the subject being tracked. [Means for solving the problem]

[0006] The present invention is an information processing device for detecting a subject to be tracked from a moving image captured by an imaging device and performing automatic tracking photography, and is characterized by comprising: an obstacle detection means for detecting an obstacle that may be mistaken for the subject from the moving image by AI object detection; a registration means for registering the video region of the obstacle detected by the obstacle detection means; an exclusion means for excluding the video region of the obstacle registered by the registration means from a detection region for detecting the subject in the moving image; and a subject detection means for detecting the subject from the detection region from which the video region of the obstacle has been excluded by AI object detection. [Effects of the Invention]

[0007] According to the present invention, it becomes possible to detect the subject being tracked with high accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing the configuration of the automatic tracking and shooting system according to the embodiment. [Figure 2] This figure shows the hardware configuration of the automatic tracking and shooting system according to the embodiment. [Figure 3] This figure shows the functional configuration of the automatic tracking and shooting system according to the embodiment. [Figure 4] This is a conceptual diagram showing the input / output structure using a learning model. [Figure 5] This diagram illustrates how AI object detection can detect a target subject from an AI object detection region that excludes the video area of ​​obstacles. [Figure 6] This is a diagram illustrating how to register the video region of an obstacle. [Figure 7] This is a flowchart showing the learning process. [Figure 8] This is a diagram to explain the training data. [Figure 9] This flowchart shows the processing flow in tracking mode in the first embodiment. [Figure 10] This flowchart shows the processing flow in tracking mode in the second embodiment. [Figure 11] This diagram illustrates a method for detecting a target subject to be tracked from an AI object detection area, which excludes a portion of the video field containing obstacles, using AI object detection. [Figure 12] This is a flowchart showing the processing flow in tracking mode in the third embodiment. [Figure 13] This is a diagram to explain the training data. [Figure 14] This is a conceptual diagram showing the input / output structure using a learning model. [Figure 15] This flowchart shows the processing flow in tracking mode in the fourth embodiment. [Modes for carrying out the invention]

[0009] Preferred embodiments of the present invention will be described below with reference to the attached drawings. [First Embodiment] The first embodiment will be described with reference to Figures 1 to 9. FIG. 1 is a diagram showing the configuration of an automatic tracking shooting system according to an embodiment. The automatic tracking shooting system includes an information processing device 100, a pan-tilt device 200 equipped with a camera, an operating device 300, and a network 131. When the user operates the operating device 300, a command corresponding to the operation is transmitted to the pan-tilt device 200 via the network 131 and the information processing device 100. By controlling the pan-tilt device 200 according to the content of the command, the user can remotely operate the pan-tilt device 200. The video captured by the pan-tilt device 200 is input to the information processing device 100, and various arithmetic processes and recording processes necessary for automatic tracking shooting are performed.

[0010] For example, the pan-tilt device 200 and the information processing device 100 are installed at spots such as airports, towers, and television station rooftops, and the operating device 300 is installed inside a television station or the like. The network 131 is a communication line such as a public telephone line or the Internet. Hereinafter, in this embodiment, a case where the information processing device 100 and the pan-tilt device 200 are installed at an airport and the subject to be tracked is an aircraft will be taken as an example.

[0011] FIG. 2 is a diagram showing the hardware configuration of an automatic tracking shooting system according to an embodiment. The information processing device 100 includes a RAM 101, a GPU 102, a CPU 103, an input unit 104, a storage unit 105, a serial communication unit 106, a network communication unit 107, and a UI (user interface) unit 108. The RAM 101 is a volatile memory and is used as a temporary storage area such as the main memory and work area of the CPU 103. The CPU 103 controls each part of the information processing device 100 using the RAM 101 as a work memory according to a program stored in the storage unit 105, for example. Note that the CPU 103 can be constituted by one or more processors. Since the GPU 102 can perform efficient calculations by processing more data in parallel, it is effective to perform processing with the GPU 102 when performing learning multiple times using a learning model with a large number of parameters such as deep learning.

[0012] The input unit 104 is an interface for inputting a video signal into the information processing apparatus 100, and is various communication interfaces such as USB, for example. The storage unit 105 is a non-volatile memory, and image data, other data, various programs for the operation of the CPU 103, etc. are stored in predetermined areas respectively. The storage unit 105 is constituted by a storage medium such as an HDD or a flash memory, for example. The serial communication unit 106 is a communication interface for communicating with the pan-tilt device 200 under the control of the CPU 103. The network communication unit 107 is a communication interface for communicating with the operation device 300 via the network 131 under the control of the CPU 103. The UI unit 108 is a user interface for receiving operation inputs from a user who operates the information processing apparatus 100 and displaying various information to the user. These can be operated using a keyboard, a mouse, a display touch panel, etc.

[0013] The pan-tilt device 200 includes a camera 201, a drive unit 202, a serial communication unit 203, a CPU 204, and a storage unit 205, and functions as an imaging device in the present invention. The camera 201 photographs the surroundings and captures a moving image of a subject to be tracked. The camera 201 is equipped with an optical zoom lens whose shooting magnification can be changed, and the magnification of the captured video can be changed by transmitting a zoom control command from the CPU 204 to the camera 201. Further, the camera 201 also has a digital zoom function for locally magnifying a part of the captured video. The digital zoom is executed when the magnification is not sufficient with the optical zoom, that is, when a more magnified video is desired. Also, the camera 201 is connected by wire to the input unit 104 of the information processing apparatus 100 via the pan-tilt device 200, and outputs the captured video signal to the information processing apparatus 100. The drive unit 202 includes an actuator for turning the pan-tilt device 200 in the pan and tilt directions, its drive circuit, and peripheral circuits. By turning the pan-tilt device 200 in the pan and tilt directions, tracking shooting becomes possible.

[0014] The serial communication unit 203 is connected to the serial communication unit 106 and is a communication interface for communicating with the information processing device 100 under the control of the CPU 204. The CPU 204 controls the pan / tilt head device 200 according to a program stored, for example, in the memory unit 205. The CPU 204 can be composed of one or more processors. The memory unit 205 is a non-volatile memory, and setting data for the pan / tilt head device 200, other data, and various programs for the operation of the CPU 204 are stored in predetermined areas.

[0015] The operating device 300 comprises a network communication unit 301, an operating unit 302, a storage unit 303, a CPU 304, and a display unit 305. The network communication unit 301 is a communication interface for communicating with the information processing device 100 via the network 131 under the control of the CPU 304. The operation unit 302 consists of a joystick, operating levers, and various switches, which the user can use to control the rotation, zoom, gain, etc., of the pan / tilt head device 200. The memory unit 303 is a non-volatile memory, where setting data for the operation device 300, other data, and various programs for the operation of the CPU 304 are stored in predetermined areas. The CPU 304 controls each part of the operation device 300 according to, for example, the programs stored in the memory unit 303. The CPU 304 can be configured with one or more processors.

[0016] Figure 3 is a diagram showing the functional configuration of the automatic tracking shooting system according to this embodiment. The information processing device 100 includes a learning unit 150, a mode management unit 151, a communication control unit 152, an obstacle detection unit 153, a memory control unit 154, an image processing unit 155, a subject detection unit 156, and a pan / tilt head control unit 157. The learning unit 150 performs training to perform AI object detection and generates a trained model. Details of the training will be described later. The mode management unit 151 manages three modes: learning mode, tracking mode, and manual mode. Details of each mode will be described later. The communication control unit 152 controls communication with the pan / tilt head device 200 via the serial communication unit 106 and communication with the operating device 300 via the network communication unit 107, and exchanges control signals and status information in accordance with predetermined communication rules (protocols).

[0017] The obstacle detection unit 153 uses AI object detection based on a learning model generated by the learning unit 150 to detect obstacles (hereinafter simply referred to as "obstacles") that could be mistaken for the subject being tracked, from the video captured by the pan / tilt head device 200. The memory control unit 154 uses the memory unit 105 to perform tasks such as recording automatically tracked video footage, recording learning data, recording detection results of the tracked subject and obstacles, and recording command values. In this embodiment, the memory control unit 154 plays the role of a registration means for registering the video region of obstacles detected by the obstacle detection unit 153.

[0018] The image processing unit 155 performs image processing on the video captured by the pan / tilt head device 200. In this embodiment, the image processing unit 155 plays the role of an exclusion means that excludes the video area of ​​an obstacle registered by the memory control unit 154 from the detection area (hereinafter referred to as the AI ​​object detection area) for detecting the subject to be tracked in the video captured by the pan / tilt head device 200. Details of this process will be described later. The subject detection unit 156 detects the subject to be tracked from the AI ​​object detection area, which is created by the image processing unit 155 by excluding the video area of ​​obstacles, using AI object detection based on the learning model generated by the learning unit 150.

[0019] The pan / tilt head control unit 157 calculates and outputs a control signal to control the pan / tilt head device 200 according to the mode managed by the mode management unit 151. In tracking mode, it calculates and outputs a control signal to control the pan / tilt head device 200 so that the subject being tracked is centered in the image. In manual mode, it calculates and outputs a control signal in response to commands from the operating device 300. The functions of each unit 150 to 157 are realized, for example, by the CPU 103 executing a predetermined program. The learning unit 150 uses a GPU 102 in addition to the CPU 103. Specifically, when executing a learning program that includes a learning model, the CPU 103 and GPU 102 work together to perform calculations to carry out learning. The processing of the learning unit 150 may also be performed by the CPU 103 or the GPU 102 alone. Furthermore, the detection units 153 and 156 may also use the GPU 102, similar to the learning unit 150.

[0020] The pan / tilt mount device 200 includes a pan / tilt control unit 250, a camera control unit 251, a setting management unit 252, and a communication control unit 253. The pan-tilt control unit 250 outputs signals for pan and tilt driving to the drive unit 202 based on control signals received under the control of the communication control unit 253. The camera control unit 251 outputs a signal to the camera 201 for controlling the camera based on the control signal received under the control of the communication control unit 253. The settings management unit 252 manages the settings of the operating device 300. Specific setting items include the maximum speed of pan and tilt drive, and the driveable range. The communication control unit 253 controls communication with the information processing device 100 via the serial communication unit 203 and exchanges control signals and status information in accordance with predetermined communication rules (protocols). The functions of each part 250-253 are realized, for example, by the CPU 204 executing a predetermined program.

[0021] The operating device 300 comprises a communication control unit 350 and a display control unit 351. The communication control unit 350 controls communication with the information processing device 100 via the network communication unit 301 and exchanges control signals and status information in accordance with predetermined communication rules (protocols). The display control unit 351 performs display processing using the display unit 305. The functions of each part 350 and 351 are realized, for example, by the CPU 304 executing a predetermined program.

[0022] This section will explain the sequence of events in tracking mode within such an automatic tracking shooting system. The user operates the control device 300 and sends a signal to the information processing device 100 to switch to tracking mode. Subsequently, the pan / tilt head device 200 takes a picture, and the pan / tilt head device 200 transmits the captured image to the information processing device 100. The information processing device 100 detects the subject to be tracked from the image and controls the pan / tilt head device 200 so that the subject to be tracked is in the center of the image. A specific example of tracking mode will be described later.

[0023] Refer to Figure 4 to explain the input / output structure using the learning model. Figure 4 is a conceptual diagram showing the input / output structure using the learning model, illustrating the method for detecting objects and obstacles. The input data 400 is video data captured by the pan / tilt head device 200. Since the pan / tilt head device 200 captures moving images, the actual input data is one frame from the moving image, and for the sake of simplicity, the input data 400 will be referred to as image data. In the illustrated example, the image data 400 contains three objects (an aircraft in flight 406, an aircraft parked 407, and a transmission tower 408 that has similar characteristics to the aircraft).

[0024] The learning model 401 is composed of a neural network, and its internal parameters are determined by the learning unit 150. The learning unit 150 may include an error detection unit and an update unit. The error detection unit obtains the error between the output data output from the output layer of the neural network and the training data, depending on the input data input to the input layer. The error detection unit may calculate the error between the output data from the neural network and the training data using a loss function. The update unit updates the connection weight coefficients, etc., between the nodes of the neural network based on the error obtained by the error detection unit, so as to reduce the error. This update unit updates the connection weight coefficients, etc., for example, using backpropagation. Backpropagation is a method of adjusting the connection weight coefficients, etc., between the nodes of each neural network so as to reduce the above error.

[0025] Output data 402-404 contains the tags, coordinates, and likelihoods of three objects present in image data 400 (aircraft in flight 406, parked aircraft 407, and transmission tower 408). Output data 402 represents aircraft in flight 406, output data 403 represents parked aircraft 407, and output data 404 represents transmission tower 408. The tags are selected from the tags included in the training data input during training. Two coordinate points are output. These two points are the upper left and lower right corners of the circumscribing frame of the estimated object, as shown in the image region 405. The likelihood is a value between 0 and 1, and a larger value indicates a higher confidence level for the detected tag. By setting a threshold for likelihood and detecting subjects with a likelihood above the threshold, it is possible to output only those with a certain level of confidence. The coordinates of the output data are the coordinates within the captured image data, but it is possible to calculate the coordinate position after the pan, tilt, and zoom positions at the time of coordinate output and the amount of drive of the pan / tilt

[0026] Referring to Figure 5, we will explain how to detect a subject to be tracked from an AI object detection area that excludes the video region of obstacles using AI object detection. Figure 5 is a diagram illustrating how to detect a subject to be tracked from an AI object detection area that excludes the video region of obstacles using AI object detection. The memory control unit 154 pre-registers the video regions of obstacles, and based on this information, the image processing unit 155 processes the image data 500 that is input to the learning model 401. Image data 500 will be described as being the same as image data 400. The method by which the memory control unit 154 registers the video regions of obstacles will be described later. We will explain the case where an aircraft in flight 406 is the subject to be tracked, and a parked aircraft 407 and a transmission tower 408 are obstacles. Input data 501 shows the processed image data. Input data 501 is the image data 500 with the video regions of obstacles blacked out as excluded regions. A rectangle with the line connecting two coordinate points of the obstacle as its diagonal is defined as the video region of the obstacle and is blacked out as such. By blacking out the video region of the obstacle, unlike the result described in Figure 4, only output data 402 can be output, and as shown in video region 503, it becomes possible to detect only the aircraft in flight, which is the subject to be tracked.

[0027] Refer to Figure 6 to explain how to register the video region of an obstacle. Figure 6 is a diagram illustrating how to register the video region of an obstacle. Image data 600 is a single frame from a video captured by the pan / tilt head device 200. Image data 601 is also a single frame from a video captured by the pan / tilt head device 200, and was captured at a later time than image data 600. Image data 601 will be described as being the same as image data 400. The parked aircraft 407 and transmission tower 408 remain in the same position in image data 600 and image data 601. The aircraft 406 in flight is not present in image data 600, but moves and becomes present in image data 601, which was captured at a later time than image data 600.

[0028] Following the procedure described in Figure 4, when image data 600, which does not contain object 406, is input to the learning model 401, output data 403 and 404 are obtained. Based on these output data 403 and 404, a rectangle with the line connecting two coordinate points as its diagonal is calculated and registered as the video region of the obstacle. In addition, when registering the video region of the obstacle, the pan, tilt, and zoom information from when the image data 600 was captured is also registered.

[0029] Figure 7 is a flowchart showing the learning process during the learning phase. In step S700, the information processing device 100 determines whether the current mode managed by the mode management unit 151 is the learning mode. If the information processing device 100 determines that it is the learning mode, the process proceeds to step S701; otherwise, the process proceeds to step S706.

[0030] In step S701, the learning unit 150 accepts input of training data. Now, with reference to Figure 8, the training data will be explained. Figure 8 is a diagram illustrating the training data. The training data is created by linking an image of an aircraft (input data) 801 with the tag "aircraft" 802 (training data) of an object contained in the image 801, and this is used as one training data. Preferably, the input data, image 801, contains only one object, and the size of image 801 is the same across multiple training data. In this embodiment, since automatic tracking and shooting of an aircraft is performed at an airport, video footage of an aircraft is prepared as input data.

[0031] Returning to the explanation in Figure 7, in step S702, the learning unit 150 determines whether the data received in step S701 is legitimate (conforms to the established rules). If the learning unit 150 determines that the received data is legitimate, the process proceeds to step S703; otherwise, the process returns to step S701. In step S703, the learning unit 150 inputs the data received in step S701 into the learning model. In step S704, the learning unit 150 performs learning. In step S705, the learning unit 150 determines whether or not it has learned all of the training data. If the learning unit 150 determines that it has learned all of the data, this flowchart ends; otherwise, the process returns to step S701. Through the above process, the internal parameters of the learning model are determined, and a learning model for AI object detection is generated.

[0032] The processing steps from step S706 onward show the processing flow in manual mode or tracking mode. In step S706, the information processing device 100 determines whether the current mode managed by the mode management unit 151 is manual mode. If the information processing device 100 determines that it is manual mode, the process proceeds to step S707; otherwise, the process proceeds to step S709. In step S707, the information processing device 100 receives a command from the operating device 300. In step S708, the information processing device 100 transmits a control signal to the operating device 300 corresponding to the command received in step S707, thereby ending this flowchart. In step S709, the information processing device 100 performs tracking mode processing. Details of the tracking mode processing will be described later.

[0033] Figure 9 is a flowchart showing the processing flow in tracking mode. The flowchart in Figure 9 is executed by the information processing device 100 after a signal to switch to tracking mode is sent to the information processing device 100 by an operation on the operating device 300. This embodiment is an example in which the area of ​​the moving image captured by the pan / tilt head device 200 is fixed when an obstacle is detected and its video area is registered. In step S900, the pan / tilt control unit 157 transmits a control signal to the pan / tilt device 200 to move it to the home position. The home position is the pre-set pan and tilt position of the pan / tilt device 200. In step S901, the pan / tilt head control unit 157 transmits a control signal to the pan / tilt head device 200 to perform wide-area shooting, and receives the resulting moving image from the pan / tilt head device 200.

[0034] In step S902, the obstacle detection unit 153 uses the image in the video received in step S901 as input data and detects obstacles in the manner described in Figure 4. In step S903, the memory control unit 154 registers the video area of ​​the obstacle detected in step S902. In step S904, the information processing device 100 determines whether or not it is the update time, which is a preset time. If the information processing device 100 determines that it is the update time, the process returns to step S902. If the information processing device 100 determines that it is not the update time, the process proceeds to step S905. There is not just one update time. Since the state of an obstacle may change over time, the detection of an obstacle and the registration of the obstacle's video area are updated at preset intervals.

[0035] In step S905, the image processing unit 155 removes the video region of the obstacle from the AI ​​object detection region using the method described in Figure 5. In step S906, the subject detection unit 156 detects an aircraft that will be the subject of tracking from the AI ​​object detection area, which was created by excluding the video area of ​​obstacles in step S905, using the method described in Figure 5. If the information processing device 100 determines that an aircraft has been detected, the process proceeds to step S907; otherwise, the process returns to step S904.

[0036] In step S907, the information processing device 100 starts aircraft tracking and photography. In step S908, the image processing unit 155 excludes the video region of the obstacle from the AI ​​object detection region, similar to step S905. In step S909, the subject detection unit 156 detects an aircraft that will be tracked from the AI ​​object detection area, which was created by excluding the video area of ​​obstacles in step S908, using the method described in Figure 5. In step S910, the pan / tilt head control unit 157 calculates a control signal to control the pan / tilt head device 200 so that the aircraft is centered in the image, based on the coordinates of the aircraft in the moving image detected in step S909, and transmits it to the pan / tilt head device 200. In step S911, the information processing device 100 determines whether or not to continue tracking. For example, the tracking duration may be predetermined, and tracking may end once the time has elapsed. If the information processing device 100 determines to continue tracking, the process returns to step S908; otherwise, this flowchart terminates.

[0037] As described above, steps S900 to S906 detect obstacles and register the image area of ​​the obstacles before starting automatic tracking shooting. Then, steps S907 to S911 exclude the image area of ​​the obstacles from the AI ​​object detection area during automatic tracking shooting and detect the subject to be tracked. This makes it possible to detect the subject to be tracked accurately without detecting obstacles during automatic tracking shooting.

[0038] [Second Embodiment] A second embodiment will be described with reference to Figure 10. The schematic configuration of the automatic tracking shooting system according to the second embodiment is the same as that described in the first embodiment. Hereafter, we will omit explanations of matters common to the first embodiment and focus on matters that differ from the first embodiment.

[0039] Figure 10 shows a flowchart illustrating the processing flow in tracking mode. The flowchart in Figure 10 is executed by the information processing device 100 after a signal to switch to tracking mode is transmitted to the information processing device 100 by an operation on the operating device 300. This embodiment is an example in which the area of ​​the moving image captured by the pan / tilt head device 200 is changed when an obstacle is detected and its video area is registered. In step S1000, the pan / tilt control unit 157 calculates a control signal and transmits it to the pan / tilt device 200 to photograph the movement path of the aircraft that is the subject to be tracked. In response, the pan / tilt device 200 photographs the movement path by panning, tilting, and zooming. At this time, the obstacle detection unit 153 uses the image in the moving image captured by the pan / tilt device 200 as input data and detects obstacles in the manner described in Figure 4. In step S1001, the memory control unit 154 registers the video area of ​​the obstacle detected in step S1000.

[0040] In step S1002, the pan / tilt control unit 157 transmits a control signal to the pan / tilt device 200 to move it to the home position. The home position is a preset pan and tilt position of the pan / tilt device 200. In step S1003, the pan / tilt head control unit 157 transmits a control signal to the pan / tilt head device 200 to perform wide-area shooting, and receives the resulting moving image from the pan / tilt head device 200. In step S1004, the information processing device 100 determines whether or not it is the update time, which is a preset time. If the information processing device 100 determines that it is the update time, the process returns to step S1000. If the information processing device 100 determines that it is not the update time, the process proceeds to step S1005. There may be more than one update time. The steps S1005 to S1011 are the same as the steps S905 to S911 in Figure 9.

[0041] As described above, steps S1000 to S1006 detect obstacles and register the image area of ​​the obstacles before starting automatic tracking shooting. Then, steps S1007 to S1011 exclude the image area of ​​the obstacles from the AI ​​object detection area during automatic tracking shooting and detect the subject to be tracked. This makes it possible to detect the subject to be tracked accurately without detecting obstacles during automatic tracking shooting.

[0042] [Third Embodiment] A third embodiment will be described with reference to Figures 11 and 12. The schematic configuration of the automatic tracking shooting system according to the third embodiment is the same as that described in the first embodiment. Hereafter, we will omit explanations of matters common to the first embodiment and focus on matters that differ from the first embodiment. In the first and second embodiments, the entire area of ​​the obstacle in the image is excluded from the AI ​​object detection area. However, if there are many parked aircraft, for example, within the shooting angle of view, the excluded area increases and occupies a wide area of ​​the AI ​​object detection area. As a result, the area in which the subject to be tracked can be detected becomes narrower, which may affect the detection of the subject to be tracked. Therefore, in this embodiment, a portion of the video area containing obstacles is excluded from the AI ​​object detection area to prevent the area in which the subject to be tracked can be detected from becoming narrower.

[0043] Referring to Figure 11, a method for detecting a subject to be tracked from an AI object detection area, which excludes a portion of the video area containing obstacles, will be explained. Figure 11 is a diagram illustrating a method for detecting a subject to be tracked from an AI object detection area, which excludes a portion of the video area containing obstacles, using AI object detection. In the first embodiment, the rectangle formed by the line connecting the two coordinate points of the obstacle as its diagonal is defined as the excluded area (the area to be blacked out), whereas in this embodiment, only a portion of the rectangle formed by the line connecting the two coordinate points of the obstacle as its diagonal is defined as the excluded area. Specifically, a portion of the quadrilateral region formed by the diagonal of the two coordinate points of the obstacle is defined as the excluded area. For example, only the central 30 percent portion of the rectangle in the direction of the shorter side is defined as the partial area. As a result, unlike the results described in Figure 4, the output data 1103 for the parked aircraft 407 and the output data 1104 for the transmission tower 408 lose their aircraft-likeness compared to output data 403 and output data 404, and the likelihood decreases from 0.8 to 0.6.

[0044] As a result, if the likelihood of output data 1103 and output data 1104 falls below the threshold for the likelihood of detecting the subject being tracked (hereinafter referred to as the "subject detection threshold"), it will no longer be mistakenly identified as the subject being tracked. Currently, in the partial region, if the likelihood of an obstacle is greater than or equal to the subject detection threshold, the partial region is expanded by a predetermined amount. In this way, it is possible to set a portion of the video region of the obstacle according to the likelihood of the obstacle. This is repeated until the likelihood of the obstacle falls below the subject detection threshold. As a result, as shown in video region 1105, it becomes possible to detect only the aircraft 406 in flight, which is the subject being tracked, similar to the video region 503 in Figure 5.

[0045] Furthermore, by reducing the likelihood of obstacles, it becomes possible to readjust the subject detection threshold to a lower value. By adding the process of readjusting the subject detection threshold to a lower value, the detection accuracy of the subject being tracked can be improved. For example, in the example in Figure 11, the likelihood has decreased to 0.6 in output data 1103 and output data 1104. Therefore, if the subject detection threshold is greater than 0.6, the parked aircraft 407 and the transmission tower 408 will not be detected. Lowering the subject detection threshold makes it easier to detect subjects, so by performing this process, the detection accuracy of the subject being tracked can be improved.

[0046] Figure 12 is a flowchart showing the processing flow in tracking mode. The flowchart in Figure 12 is executed by the information processing device 100 after a signal to switch to tracking mode is sent to the information processing device 100 by an operation on the operating device 300. The procedure in steps S1200 to S1204 is the same as the procedure in steps S900 to S904 in Figure 9.

[0047] In step S1212, the image processing unit 155 excludes a portion of the obstacle's image area (initial partial area) from the AI ​​object detection area using the method described in Figure 11. For example, the initial partial area is defined as only the central 30 percent portion in the direction of the shorter side of a quadrilateral area formed by the coordinates of two points of the obstacle. In step S1213, the information processing device 100 determines whether the likelihood of the obstacle is less than or equal to the subject detection threshold. If the information processing device 100 determines that the likelihood of the obstacle is less than or equal to the subject detection threshold, the process proceeds to step S1215; otherwise, the process proceeds through step S1214 and returns to step S1213. In step S1214, the image processing unit 155 performs a process to expand the partial area by a predetermined amount, thereby increasing the proportion of the excluded area. In step S1215, the information processing device 100 resets and updates the subject detection threshold to a value smaller than the likelihood of the obstacle.

[0048] The steps S1206 to S1211 are the same as the steps S906 to S911 in Figure 9. However, in step S1208, the image processing unit 155 excludes a portion of the obstacle's video area from the AI ​​object detection area. This portion is the excluded area finally determined through steps S1212 to S1214. Then, in step S1208, the subject detection unit 156 detects the aircraft to be tracked from the AI ​​object detection area, which has had a portion of the obstacle's video area excluded in step S1208, based on the subject detection threshold updated in step S1215, using the method described in Figure 11.

[0049] As described above, steps S1200 to S1206 detect obstacles and register the image area of ​​the obstacles before starting automatic tracking shooting. Then, steps S1207 to S1211 exclude a portion of the image area of ​​the obstacles from the AI ​​object detection area during automatic tracking shooting to detect the subject to be tracked. In this embodiment, by excluding a portion of the image area of ​​the obstacles from the AI ​​object detection area, it is possible to prevent the area in which the subject to be tracked can be detected from becoming narrower. As a result, it becomes possible to detect the subject to be tracked with high accuracy without detecting obstacles during automatic tracking shooting.

[0050] [Fourth Embodiment] A fourth embodiment will be described with reference to Figures 13 to 15. The schematic configuration of the automatic tracking shooting system according to the fourth embodiment is the same as that described in the first embodiment. Hereafter, we will omit explanations of matters common to the first embodiment and focus on matters that differ from the first embodiment. When AI object detection detects an aircraft, and simultaneously detects specific parts of the aircraft, excluding these specific parts can also cause the aircraft to lose its characteristic appearance. In other words, by excluding specific parts of the aircraft that are detected as obstacles, it is possible to reduce the likelihood that non-tracked obstacles will be detected as aircraft. This embodiment describes how to do this.

[0051] Figure 13 is a diagram illustrating the training data. In addition to the training data linking aircraft image 801 and tag "aircraft" 802, similar to that described in the first embodiment, training data linking images and tags of specific parts of the aircraft is prepared and trained as separate objects. For example, training data of the head, which is the front part of the aircraft shown in image 801 (head image 1301 and tag "head" 1302), and training data of the tail fin, which is the rear part of the aircraft shown in image 801 (tail fin image 1303 and tag "tail fin" 1304) are trained. By performing the training described in the first embodiment for each object, a learning model for each object is generated.

[0052] Refer to Figure 14 to explain the input / output structure using the learning model. Figure 14 is a conceptual diagram showing the input / output structure using the learning model. The input data 1400 is video data captured by the pan / tilt head device 200. Since the pan / tilt head device 200 captures moving images, the actual input data is one frame from the moving image. For simplicity of explanation, the input data 1400 will be referred to as image data. In the illustrated example, the image data 400 contains three objects (aircraft, head, and tail fin).

[0053] Output data 1401-1403 contains the tags, coordinates, and likelihood of the objects (aircraft, head, tail) present in image data 1400. Figure 13 illustrates the case where the tags are "aircraft," "head," and "tail" as an example of training data; however, the tags are selected from the tags included in the training data input during training. Two coordinate points are output. These two points are the upper left and lower right corners of the estimated object's bounding box, as shown in image region 1405. If the training data tags include parts that constitute an object (hereinafter referred to as "object parts"), the coordinates of the object parts lie inside the object's bounding box. For output data 1401, coordinates 1 and 2 correspond to these coordinates; for output data 1402, coordinates 3 and 4 correspond to these coordinates; and for output data 1403, coordinates 5 and 6 correspond to these coordinates. The object's size and center coordinates can be calculated from the coordinates of each pair of points. The likelihood is a value between 0 and 1; a higher value indicates higher confidence in the detection result for the outputted tag.

[0054] Thus, by using AI object detection to detect an aircraft, and simultaneously detecting specific parts of the aircraft, it is possible to perform exclusion processing for each specific part, thereby reducing the likelihood of it being an obstacle and preventing it from being mistaken for the target subject being tracked. In the flowchart explained in Figure 12, step S1212 may be used to exclude a specific part of the obstacle's image region, and step S1213 may be used to determine the likelihood of it being an aircraft.

[0055] By setting specific parts according to the state of the subject being tracked, highly accurate automatic tracking becomes possible. For example, if you want to track a landing aircraft, the camera composition would likely involve shooting with the aircraft facing you. In this case, as shown in Figure 13, the head of the aircraft is the easiest part to capture. Therefore, by setting the exclusion area (part of the image area of ​​an obstacle) to the head, and further adding the aircraft's head to the tracking target, highly accurate automatic tracking that avoids obstacles becomes possible. The above explanation describes a method for setting specific parts depending on the state of the subject being tracked. However, it is also possible to set parts that will not be excluded and exclude the remaining parts. For example, if you want to detect and track the tail fin of an aircraft, you can choose not to exclude the tail fin and exclude the other parts.

[0056] Figure 15 is a flowchart showing the processing flow in tracking mode. The flowchart in Figure 15 is executed by the information processing device 100 after a signal to switch to tracking mode is sent to the information processing device 100 by an operation on the operating device 300. In step S1501, the information processing device 100 receives the status setting for the subject to be tracked. In this embodiment, a landing aircraft is set as the subject to be tracked. For the setting, the pan / tilt head device 200 receives a command from the operating device 300 and manages it in the mode management unit 151. The procedure for steps S1200 to S1204 is as described in Figure 12. In step S1502, a specific part to be excluded is determined according to the state of the subject to be tracked, as set in step S1501. If the subject to be tracked is a landing aircraft, the head is set as the specific part to be excluded. In step S1503, the image processing unit 155 excludes the image region of a specific part from the AI ​​object detection area. The steps S1206 to S1211 are as shown in Figure 12.

[0057] Although the present invention has been described above along with its embodiments, these embodiments are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features. In this embodiment, an example of blacking out the area of ​​an obstacle in the video feed was described as a process to exclude the area of ​​an obstacle from the AI ​​object detection area, but this is not the only method. Any method that applies some kind of image processing to the area of ​​an obstacle in the video feed to prevent it from being mistaken for the subject being tracked is acceptable. (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0058] This embodiment includes the following configuration. (Composition 1) An information processing device for detecting a subject to be tracked from a moving image captured by an imaging device and performing automatic tracking photography, An obstacle detection means that detects obstacles that may be mistaken for the subject from the video image using AI object detection, A registration means for registering the video region of the obstacle detected by the obstacle detection means, An exclusion means for excluding the video region of the obstacle registered by the registration means from the detection region for detecting the subject in the video image, An information processing apparatus characterized by comprising: subject detection means for detecting the subject from the detection region, which excludes the video region of the obstacle, by AI object detection. (Configuration 2) Before starting automatic tracking shooting, the obstacle detection means detects the obstacle, and the registration means registers the video area of ​​the obstacle. The information processing device according to Configuration 1, characterized in that, during automatic tracking shooting, the exclusion means excludes the image area of ​​the obstacle and the subject detection means detects the subject. (Composition 3) The information processing device according to configuration 2, characterized in that the obstacle detection means detects the obstacle and the registration means registers the video area of ​​the obstacle, and updates this at a predetermined time interval. (Composition 4) The imaging device has pan, tilt, and zoom functions. The information processing device according to any one of configurations 1 to 3, characterized in that the registration means also records pan, tilt, and zoom information when registering the video area of ​​the obstacle. (Composition 5) An information processing device according to any one of configurations 1 to 4, characterized in that when detecting an obstacle with the obstacle detection means and registering the video region of the obstacle with the registration means, the region of the video image captured by the imaging device is fixed. (Composition 6) An information processing device according to any one of configurations 1 to 4, characterized in that the obstacle detection means detects the obstacle and the registration means registers the video region of the obstacle, and the region of the video image captured by the imaging device is changed. (Composition 7) The exclusion means excludes a portion of the video area of ​​the obstacle registered by the registration means from the detection area. The information processing device according to any one of configurations 1 to 6, characterized in that the subject detection means detects the subject by AI object detection from the detection area excluding a part of the video area of ​​the obstacle. (Composition 8) The information processing device according to configuration 7, characterized in that a portion of the video region of the obstacle can be set according to the likelihood of the obstacle. (Composition 9) The information processing device according to configuration 7 or 8, characterized in that the threshold for the likelihood of detecting a subject to be tracked can be changed. (Composition 10) The information processing device according to configuration 7, characterized in that a portion of the video area of ​​the obstacle can be set according to the state of the subject. (Claim 11) An information processing device according to any one of configurations 7 to 10, characterized in that it enables tracking of a specific part of the subject. [Explanation of Symbols]

[0059] 100: Information processing unit, 150: Learning unit, 151: Mode management unit, 152: Communication control unit, 153: Obstacle detection unit, 154: Memory control unit, 155: Image processing unit, 156: Subject detection unit, 157: Pan / tilt head control unit, 200: Pan / tilt head device, 300: Operating device

Claims

1. An information processing device for detecting a subject to be tracked from a moving image captured by an imaging device and performing automatic tracking photography, An obstacle detection means that detects obstacles that may be mistaken for the subject from the video image using AI object detection, A registration means for registering the video region of the obstacle detected by the obstacle detection means, An exclusion means for excluding the video region of the obstacle registered by the registration means from the detection region for detecting the subject in the video image, An information processing apparatus characterized by comprising: subject detection means for detecting the subject from a detection area that excludes the video area of ​​the obstacle by AI object detection.

2. Before starting automatic tracking shooting, the obstacle detection means detects the obstacle, and the registration means registers the video area of ​​the obstacle. The information processing device according to claim 1, characterized in that, during automatic tracking shooting, the exclusion means excludes the image area of ​​the obstacle and the subject detection means detects the subject.

3. The information processing apparatus according to claim 2, characterized in that the obstacle detection means detects the obstacle and the registration means registers the video area of ​​the obstacle, and updates this at a predetermined time interval.

4. The imaging device has pan, tilt, and zoom functions. The information processing apparatus according to any one of claims 1 to 3, characterized in that the registration means also records pan, tilt, and zoom information when registering the video area of ​​the obstacle.

5. The information processing apparatus according to any one of claims 1 to 3, characterized in that when detecting an obstacle with the obstacle detection means and registering the video region of the obstacle with the registration means, the region of the video image captured by the imaging device is fixed.

6. The information processing apparatus according to any one of claims 1 to 3, characterized in that when detecting the obstacle with the obstacle detection means and registering the video region of the obstacle with the registration means, the region of the video image captured by the imaging device is changed.

7. The exclusion means excludes a portion of the video area of ​​the obstacle registered by the registration means from the detection area. The information processing apparatus according to any one of claims 1 to 3, characterized in that the subject detection means detects the subject by AI object detection from the detection area excluding a part of the video area of ​​the obstacle.

8. The information processing apparatus according to claim 7, characterized in that a portion of the video region of the obstacle can be set according to the likelihood of the obstacle.

9. The information processing device according to claim 7, characterized in that the threshold for the likelihood of detecting a subject to be tracked can be changed.

10. The information processing device according to claim 7, characterized in that a portion of the video area of ​​the obstacle can be set according to the state of the subject.

11. The information processing device according to claim 7, characterized in that it enables tracking of a specific part of the subject.

12. A shooting system that detects a subject to be tracked from a moving image captured by an imaging device and performs automatic tracking shooting, An obstacle detection means that detects obstacles that may be mistaken for the subject from the video image using AI object detection, A registration means for registering the video region of the obstacle detected by the obstacle detection means, An exclusion means for excluding the video region of the obstacle registered by the registration means from the detection region for detecting the subject in the video image, A shooting system characterized by comprising: a subject detection means for detecting the subject from a detection area that excludes the video area of ​​the obstacle by AI object detection.

13. An information processing method for detecting a subject to be tracked from a moving image captured by an imaging device and performing automatic tracking photography, A procedure for detecting obstacles that may be mistaken for the subject from the video image using AI object detection, A procedure for registering the video area of ​​the detected obstacle, A procedure for excluding the video region of the registered obstacle from the detection region for detecting the subject in the aforementioned video image, An information processing method characterized by comprising the steps of detecting the subject from the detection region, which is obtained by excluding the video region of the obstacle, using AI object detection.

14. A program for detecting a subject to be tracked from a moving image captured by an imaging device and performing automatic tracking photography, An obstacle detection means that detects obstacles that may be mistaken for the subject from the video image using AI object detection, A registration means for registering the video region of the obstacle detected by the obstacle detection means, An exclusion means for excluding the video region of the obstacle registered by the registration means from the detection region for detecting the subject in the video image, A program for causing a computer to function as a subject detection means that detects the subject from the detection area, which is obtained by excluding the video area of ​​the obstacle, using AI object detection.

Images