Information processing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

An information processing apparatus, a method for controlling the information processing apparatus, and a storage medium are disclosed. The information processing apparatus includes: an image acquisition unit configured to acquire an image including a subject that is acquired by imaging the subject; a detection unit configured to detect one or more subject regions from the image, each subject region being a region of the subject that is a tracking target; and a setting unit configured to set a subject region of the subject that is the tracking target based on a likelihood of the subject in the subject region, wherein, in a case where at least a part of the subject is occluded by an obstacle in the image and a plurality of subject regions of the subject are detected by the detection unit, the setting unit sets the subject region among the plurality of subject regions based on the likelihood being higher than a likelihood of other subject regions among the plurality of subject regions.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a method for controlling the information processing apparatus, and a storage medium. Background Technology

[0002] In conventional technologies, the following method is well-known: a user acquires a desired video image (picture) by remotely operating a camera from a console device. For example, video images (pictures) of an aircraft broadcast on television news can be captured by remotely operating a camera platform device, including a camera permanently mounted on an airport roof, from a broadcasting station. Furthermore, there exists an automatic tracking and imaging system in which a camera platform device equipped with image recognition technology identifies a subject in a video image (picture) and, in response to the subject's movement, tracks the subject by automatically panning, tilting, and zooming the camera. This allows for capturing images of moving subjects (such as aircraft) while automatically tracking the subject, without requiring user operation of the console device.

[0003] Furthermore, the aforementioned automatic tracking and photography systems can utilize technologies designed to improve automatic tracking performance. For example, in recent years, technologies using machine learning and artificial intelligence (hereinafter referred to as "AI") have become well-known as image recognition technologies, and automatic tracking and photography systems using machine learning and AI are able to perform subject detection (object detection) with high accuracy.

[0004] However, for example, obstacles may pass in front of the target subject, which may cause the target subject to be lost and no longer within the desired field of view (hereinafter referred to as "lost"). In this regard, Japanese Patent Application Publication Nos. 2015-104050 and 2006-311099 describe a technique for stably performing automatic tracking while reducing erroneous decisions, even when obstacles are present during subject tracking and photography.

[0005] Specifically, Japanese Patent Application Publication No. 2015-104050 describes a technique that lowers the confidence threshold for subject detection (object detection) when an obstacle passes in front of the subject, thereby making it easier to identify the subject compared to the normal time when no obstacle passes in front of the subject. The technique described in Japanese Patent Application Publication No. 2015-104050 is based on the premise of continuously detecting automatically tracked subjects without loss. Therefore, the technique described in Japanese Patent Application Publication No. 2015-104050 may experience loss of the tracked target subject in situations such as when an obstacle passes in front of the target subject and obscures part of the subject, and multiple subject regions appearing in a manner where the obstacle is located are split.

[0006] Furthermore, Japanese Patent Application Publication No. 2006-311099 describes a technique in which different processing is applied to the interior and exterior of a specific region (obstacle) when a moving object (subject) being tracked overlaps with the specific region. However, the technique described in Japanese Patent Application Publication No. 2006-311099 may result in the loss of the tracked subject in cases such as when an obstacle passes in front of the tracked subject and obscures part of the subject, and when multiple subject regions are split in such a way that the obstacle is located in between. Summary of the Invention

[0007] This disclosure addresses the aforementioned problems and aims to reduce the loss of the tracked target even when at least a portion of the tracked target is obscured by an obstacle and multiple target areas are detected.

[0008] According to one aspect of this disclosure, an information processing apparatus includes: an image acquisition unit configured to acquire an image including the subject acquired by photographing the subject; a detection unit configured to detect one or more subject regions from the image, each subject region being a region of the subject serving as a tracking target; and a setting unit configured to set a subject region of the subject serving as the tracking target based on the likelihood of the subject in the subject regions, wherein, in the case where at least a portion of the subject is occluded by an obstacle in the image and multiple subject regions of the subject are detected by the detection unit, the setting unit sets the subject regions among the multiple subject regions based on the likelihood of the subject regions being higher than the likelihood of other subject regions among the multiple subject regions.

[0009] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of the embodiments is by way of example. Attached Figure Description

[0010] Figure 1 This is a diagram illustrating a schematic configuration example of an automatic tracking and photography system according to the first embodiment.

[0011] Figure 2 This is a diagram illustrating an example of the hardware configuration of each device in the automatic tracking and photography system according to the first embodiment.

[0012] Figure 3 This is a diagram illustrating an example of the software configuration of each device in the automatic tracking and photography system according to the first embodiment.

[0013] Figure 4This is a diagram illustrating an example of the processing performed by the tracking target detection unit of the information processing apparatus according to the first embodiment, using a trained model generated by the training unit, to detect and track a target subject and misidentified obstacles.

[0014] Figure 5 This is a diagram illustrating an example of an information processing apparatus according to the first embodiment performing automatic tracking of an aircraft as a tracking target.

[0015] Figure 6 This is a flowchart illustrating an example of a processing procedure in a method for controlling an information processing apparatus according to a first embodiment.

[0016] Figure 7 This is a diagram illustrating an example of an information processing apparatus according to the second embodiment performing automatic tracking of an aircraft as a tracking target.

[0017] Figure 8 This is a flowchart illustrating an example of a processing procedure in a method for controlling an information processing apparatus according to a second embodiment.

[0018] Figure 9 This is a diagram illustrating an example image when an obstacle is within the camera's field of view according to the third embodiment.

[0019] Figure 10 This is a diagram illustrating a first example of automatic tracking of an aircraft as a tracking target subject by an information processing apparatus according to a third embodiment.

[0020] Figure 11 This is a diagram illustrating a second example of automatic tracking of an aircraft as a tracking target by an information processing apparatus according to a third embodiment. Detailed Implementation

[0021] The forms (executives) used to implement this disclosure will be described with reference to the accompanying drawings.

[0022] First Embodiment First, the first embodiment will be described.

[0023] Figure 1 This is a diagram illustrating a schematic configuration example of the automatic tracking and imaging system 10 according to the first embodiment. (See diagram for reference.) Figure 1 As shown, the automatic tracking and photography system 10 includes an information processing device 100, a camera platform device 200, a console device 300, and a network 400.

[0024] Information processing device 100 controls the entire operation of automatic tracking and imaging system 10. Camera platform device 200 includes a camera configured to capture images of a tracked target subject. Furthermore, camera platform device 200 performs subject tracking and imaging by controlling the pan, tilt, and zoom of the camera under the control of information processing device 100. Control console device 300 is operated by a user to remotely manipulate camera platform device 200 or perform other operations. Operational inputs to control console device 300 are input to information processing device 100 via network 400. Network 400 connects information processing device 100 and control console device 300, enabling them to communicate with each other. Network 400 is a communication line such as a public telephone line or the Internet.

[0025] According to this embodiment, when a user operates the console device 300, the instruction corresponding to the operation is sent to the camera platform device 200 via the network 400 and the information processing device 100.

[0026] The camera platform device 200 executes control based on instructions corresponding to user operations, thereby allowing the user to remotely operate the camera platform device 200. Video images (pictures) captured by the camera of the camera platform device 200 are input to the information processing device 100. The information processing device 100 then performs various calculations and recording processes on the images input from the camera platform device 200, necessary for automatic tracking and photography. For example, the camera platform device 200 and the information processing device 100 are installed in locations such as airports, steel towers, or on the roof of a television station, and the control console device 300 is installed inside the television station or a similar location. This embodiment will describe an example assuming the camera platform device 200 and the information processing device 100 are installed at an airport and the aircraft is the target subject during automatic tracking and photography.

[0027] Figure 2 This is a diagram illustrating an example of the hardware configuration of each device in the automatic tracking and photography system 10 according to the first embodiment. Figure 2 In, with Figure 1 Each component in the figure is given the same reference numerals, and its detailed description is omitted.

[0028] First, the hardware configuration of the information processing device 100 will be described.

[0029] like Figure 2 As shown, the information processing device 100 has a hardware configuration including random access memory (RAM) 101, graphics processing unit (GPU) 102, central processing unit (CPU) 103, input unit 104, storage unit 105, serial communication unit 106, network communication unit 107 and user interface (UI) unit 108.

[0030] RAM 101 is volatile memory, used as the main memory of CPU 103 and temporary storage areas such as the working area. GPU 102 can perform computations efficiently by processing more data in parallel. Therefore, it is efficient to use GPU 102 to perform processing when performing machine learning multiple times using a trained model with a large number of parameters (such as deep learning). For example, CPU 103 uses RAM 101 as working memory to control each component of information processing device 100 based on the program stored in storage unit 105, and performs various types of control and processing of information processing device 100.

[0031] Input unit 104 is an interface for inputting information (e.g., image data from camera platform device 200) from an external device to information processing device 100. Examples include various communication interfaces, such as a Universal Serial Bus (USB) communication interface. Storage unit 105 is a non-volatile memory such as a hard disk drive (HDD) or flash memory. Storage unit 105 stores image data, other data, programs executed by CPU 103 to perform various types of control and processing on information processing device 100, etc., in its predetermined area. In addition, storage unit 105 stores various data and other data acquired by CPU 103 through performing various types of control and processing on information processing device 100. Serial communication unit 106 is an interface for serial communication with camera platform device 200 under the control of CPU 103. Network communication unit 107 is an interface for communication with console device 300 via network 400 under the control of CPU 103. UI unit 108 is a user interface for receiving operation input from a user operating information processing device 100 and displaying information about information processing device 100 to the user. The UI unit 108 can be operated using, for example, a keyboard, mouse, or display touch panel.

[0032] Next, the hardware configuration of the camera platform device 200 will be described.

[0033] like Figure 2 As shown, the camera platform device 200 has a hardware configuration including a camera 201, a drive unit 202, a CPU 204, a storage unit 205, and a serial communication unit 206. Furthermore, the drive unit 202 includes a windshield wiper 203.

[0034] Camera 201 is an image capturing unit configured to capture images of the area surrounding camera platform device 200 under the control of CPU 204, and to capture motion images of a target subject. Camera 201 includes an optical zoom lens with variable image magnification, and optical zoom can be performed to change the magnification of the image to be captured by driving the optical zoom lens based on zoom control instructions from CPU 204. Furthermore, camera 201 also includes a digital zoom function for partially magnifying a portion of the captured image. This digital zoom function is performed when the magnification achieved by optical zoom is insufficient (i.e., when further magnification of the captured image is desired). Additionally, camera 201 is connected to the input unit 104 of information processing device 100 via cable using camera platform device 200, and the captured image data is output to information processing device 100.

[0035] The drive unit 202 includes an actuator, drive circuitry for the actuator, and peripheral circuitry. The actuator is configured to rotate the camera platform assembly 200 (more specifically, the camera 201) in the pan and tilt directions under the control of the CPU 204. The drive unit 202 rotates the camera platform assembly 200 (more specifically, the camera 201) relative to the tracked target subject in the pan and tilt directions, thereby enabling the capture of an image of the tracked subject. Furthermore, a windshield wiper 203 belonging to the drive unit 202 is configured to remove water droplets and other substances from the image of the camera 201 on the camera platform assembly 200.

[0036] For example, CPU 204 controls each component of camera platform device 200 based on programs stored in storage unit 205, and performs various types of control and processing of camera platform device 200. Storage unit 205 is non-volatile memory. Storage unit 205 stores setting data and other data of camera platform device 200, programs executed by CPU 204 to perform various types of control and processing of camera platform device 200, etc., in its predetermined area. In addition, storage unit 205 stores various data and other data acquired by CPU 204 by executing various types of control and processing of camera platform device 200. Serial communication unit 206 is connected to serial communication unit 106 of information processing device 100, so that serial communication unit 206 can communicate with serial communication unit 106. Serial communication unit 206 is an interface for communicating with information processing device 100 under the control of CPU 204.

[0037] Next, the hardware configuration of the console device 300 will be described.

[0038] like Figure 2As shown, the console device 300 has a hardware configuration including a network communication unit 301, a console unit 302, a storage unit 303, a CPU 304, and a display unit 305.

[0039] The network communication unit 301 is an interface for communicating with the information processing device 100 via the network 400 under the control of the CPU 304. The control console unit 302 includes, for example, joysticks, joysticks, and various switches. Users can control rotation and zoom, and adjust the gain of the camera platform device 200, etc., by operating the control console unit 302.

[0040] Storage unit 303 is a non-volatile memory. Storage unit 303 stores setting data and other data of the console device 300, programs executed by CPU 304 to perform various types of control and processing of the console device 300, etc., in its predetermined area. Furthermore, storage unit 303 stores various data and other data acquired by CPU 304 through executing various types of control and processing of the console device 300. For example, CPU 304 controls each component of the console device 300 based on the programs stored in storage unit 303 and executes various types of control and processing in the console device 300. Display unit 305 is, for example, a light-emitting diode (LED) or a display touch panel. Display unit 305 displays, for example, the status and warnings of camera platform device 200 to the user.

[0041] It should be noted that, according to this embodiment, each of CPUs 103, 204 and 304 can be composed of one or more processors.

[0042] Figure 3 This is a diagram illustrating an example of the software configuration of each device in the automatic tracking and photography system 10 according to the first embodiment. Figure 3 In, with Figure 1 or Figure 2 Each component in the figure is given the same reference numerals, and its detailed description is omitted.

[0043] First, the software configuration of the information processing device 100 will be described.

[0044] like Figure 3 As shown, the information processing device 100 has a software configuration including a training unit 110, a data storage unit 120, a mode management unit 130, an image processing unit 140, a target tracking and detection unit 150, a video region calculation unit 160, and a camera platform control unit 170.

[0045] Training unit 110 performs machine learning processing to enable target detection unit 150 to detect the target subject from images (image data) captured by camera 201. In this machine learning process performed by training unit 110 according to this embodiment, besides... Figure 2 In addition to the CPU 103 shown, a GPU 102 is also used. Specifically, when the training unit 110 executes a training procedure including a trained model, the CPU 103 and GPU 102 cooperate to perform computations, thereby performing machine learning. It should be noted that it is possible to perform machine learning using only the CPU 103 and GPU 102. Figure 2 The CPU 103 or GPU 102 shown performs computations on the machine learning processing performed by the training unit 110.

[0046] The data storage unit 120 performs processing such as storing images (image data) captured by automatic subject tracking, storing training data, storing detection results of the tracked target subject, storing control command values, and other processing. According to this embodiment, the data storage unit 120 also stores and registers the video area of ​​the windshield wiper 203, which may become an obstacle.

[0047] The mode management unit 130 manages the operating modes of the information processing device 100. According to this embodiment, the mode management unit 130 manages the following three modes: training mode, automatic tracking mode, and manual mode.

[0048] The image processing unit 140 processes images (image data) received from the camera platform device 200 based on the data storage unit 120. According to this embodiment, the image processing unit 140 performs processing such as excluding the video area of ​​the windshield wiper 203, which is an obstacle, from the area where objects are detected by artificial intelligence (AI), and other processing.

[0049] The target detection unit 150 takes the output image from the image processing unit 140 as input data and feeds it into the trained model generated by the training unit 110. It then performs processing to detect the target subject and to detect misidentified obstacles. Furthermore, the target detection unit 150 detects the target subject from regions where objects are detected by AI, excluding video regions of obstacles stored in the data storage unit 120. It should be noted that, like the training unit 110, the target detection unit 150 can use... Figure 2 The GPU 102 shown.

[0050] The video region calculation unit 160 calculates the video regions of the tracked target and misidentified obstacles based on the detection results output from the tracking target detection unit 150.

[0051] The camera platform control unit 170 calculates control signals configured to control the camera platform device 200 based on the operating mode of the information processing device 100. For example, if the current operating mode of the information processing device 100 is automatic tracking mode, it calculates and outputs control signals configured to control the camera platform device 200 to center the tracked target subject in the image. Furthermore, for example, if the current operating mode of the information processing device 100 is manual mode, it outputs drive commands from the console device 300 to the camera platform device 200.

[0052] Next, the software configuration of the camera platform device 200 will be described.

[0053] like Figure 3 As shown, the camera platform device 200 has a software configuration including a pan / tilt control unit 210, a camera control unit 220, a settings management unit 230, and a communication unit 240.

[0054] The pan / tilt control unit 210 outputs signals to the drive unit 202 for driving the camera platform device 200 (more specifically, the camera 201) in the pan and tilt directions based on drive commands received from the information processing device 100 via the communication unit 240. It should be noted that if the drive command received from the information processing device 100 via the communication unit 240 is a drive command for the windshield wiper 203, the pan / tilt control unit 210 outputs signals to the windshield wiper 203 for driving the windshield wiper 203. The camera control unit 220 outputs signals to the camera 201 for controlling the camera 201 based on commands received from the information processing device 100 via the communication unit 240. The settings management unit 230 manages the settings configured on the console device 300. Specific examples of settings managed by the settings management unit 230 include the maximum speed and range of motion for pan and tilt drives. The communication unit 240 sends control commands and status information to the camera platform control unit 170 of the information processing device 100 according to predefined communication rules (protocols), and receives control commands and status information from it.

[0055] Next, the software configuration of the console device 300 will be described.

[0056] like Figure 3 As shown, the console device 300 has a software configuration including a communication unit 310 and a display unit 320.

[0057] The communication unit 310 sends control commands and status information to the camera platform control unit 170 of the information processing device 100 according to predefined communication rules (protocols), and receives control commands and status information from it. Figure 2Similar to the display unit 305 shown, the display unit 320 displays, for example, the status and warnings of the camera platform device 200 to the user.

[0058] Next, the operation sequence of the information processing device 100, camera platform device 200 and console device 300 in automatic tracking mode will be described.

[0059] The user operates the console device 300, which sends a signal to the information processing device 100 to switch to automatic tracking mode. Subsequently, under the control of the information processing device 100, the camera platform device 200 uses the camera 201 to capture an image of the target subject. Then, the camera platform device 200 sends the image (image data) captured by the camera 201 to the information processing device 100. The information processing device 100 then detects the target subject from the image (image data) sent by the camera platform device 200 and controls the camera platform device 200 to center the target subject in the image.

[0060] Figure 4 This is a diagram illustrating an example of the processing performed by the tracking target detection unit 150 of the information processing apparatus 100 according to the first embodiment using a trained model 420 generated by the training unit 110 to detect the tracking target subject and misidentified obstacles.

[0061] exist Figure 4 In this context, the input data 410 input to the trained model 420 is image data captured by the camera 201 of the camera platform device 200. It should be noted that since the camera 201 of the camera platform device 200 captures moving images, the actual input data 410 input to the trained model 420 is data corresponding to a single frame in the moving image, but for the sake of simplicity, the input data 410 will be described as an image below.

[0062] exist Figure 4In this model, the trained model 420 includes, for example, a neural network, and the internal parameters of the trained model are generated by the training unit 110. It should be noted that the training unit 110 may include an error detection unit and an update unit. In this case, the error detection unit of the training unit 110 obtains the error between the training data and the output data output from the output layer of the neural network based on the input data input to the input layer of the neural network. Furthermore, the error detection unit can use, for example, a loss function to detect the error between the output data from the neural network and the training data. For example, the update unit of the training unit 110 updates the coefficients or other coefficients used to weight the connections between nodes in the neural network based on the error obtained by the error detection unit, thereby reducing the error. The update unit uses, for example, a backpropagation method to update the coefficients or other coefficients used to weight the connections. Here, the backpropagation method is a scheme for adjusting the coefficients or other coefficients used to weight the connections between nodes in the neural network to reduce the error obtained by the error detection unit.

[0063] exist Figure 4 In the output data 431 to 433 from the trained model 420, there is information about the label, coordinates, and likelihood of each object present in the input data 410.

[0064] In each of the output data 431 to 433, the object is labeled "airplane," which is an example of tracking a target subject. The object label is selected from the labels included in the training data (the data used for training) input during training.

[0065] In each of the output data 431 to 433, the coordinates of two points are output as the object's coordinates. Specifically, in each of the output data 431 to 433, the coordinates of two points are output as the object's coordinates, such as... Figure 4 Image 440, corresponding to input data 410, shows the coordinates of the upper left and lower right corners of the estimated bounding box of the object. Although the coordinates in output data 431 to 433 are coordinates in the captured image data, the position coordinates of the camera platform device 200 after the drive can be calculated based on the pan, tilt, and zoom positions of the camera platform device 200 during coordinate output and the drive amount of the camera platform device.

[0066] In each of the output data 431 to 433, the object likelihood value is between 0 and 1, with larger values ​​indicating higher confidence in the detection result relative to the output label. Specifically, according to this embodiment, the object likelihood represents the AI's confidence in the object detection (the detection of the aircraft specified by the label). Setting a threshold for the likelihood allows only data with a confidence level greater than or equal to a certain level to be output.

[0067] exist Figure 4In the diagram, output data 431 represents an aircraft in flight, output data 432 represents a parked aircraft, and output data 433 represents a steel tower with similar features to an aircraft. Figure 4 In the case where the aircraft in flight is planned to track the target subject, in addition to the tracked target subject, obstacles that are not tracked may also be detected as having the same label or similar likelihood.

[0068] Figure 5 This is a diagram illustrating an example of the information processing apparatus 100 according to the first embodiment performing automatic tracking processing on an aircraft as a tracking target. Furthermore, Figure 5 An example is shown in which the windshield wiper 203 of the camera platform device 200 is designated as an obstacle 540, which exists between the camera 201 and the aircraft, which is the subject being tracked, and obscures at least a portion of the aircraft in the image captured by the camera 201. Furthermore, the windshield wiper 203, acting as obstacle 540, moves left and right on the screen of the camera 201 tracking the subject.

[0069] exist Figure 5 In the image 510, the image was captured by camera 201 before the windshield wiper 203, which acts as an obstacle 540, obstructed the aircraft, which is the subject (i.e., before the operation of the windshield wiper 203). The tracking target detection unit 150 of the information processing device 100 detects an object detection result region 501 from image 510 as the subject region, which is the region of the tracking target subject (the current example aircraft). Furthermore, in acquiring... Figure 5 When the image 510 is shown, for example, the camera platform control unit 170 of the information processing device 100 performs control to set the object detection result area 501 as the subject area for tracking the target subject and performs automatic tracking.

[0070] Subsequently, the user operates the console unit 302 of the console device 300 to drive the windshield wiper 203, and the information processing device 100 receives the operation command. Next, the information processing device 100 sends the user operation command to the camera platform device 200. When the camera platform device 200 receives the user operation command, the CPU 204 controls the drive unit 202 to drive the windshield wiper 203. At this time, the CPU 103 of the information processing device 100, while receiving the user operation command from the console device 300, has already acquired information about when the windshield wiper 203 will be driven (as the windshield wiper 203, acting as an obstacle 540, passes in front of the camera platform device 200). In some embodiments, the wiper operation can be automatic, that is, if the wiper operation can be indicated, for example, by sensors configured to monitor parameters or environmental conditions (e.g., dirt, liquid, or the like), such as starting or stopping.

[0071] exist Figure 5 In the image 520, at least a portion of the aircraft, which is the subject, is obstructed by the windshield wiper 203, which acts as an obstacle 540. Figure 5 In the example shown, the image was captured by camera 201 in a partially obscured state (i.e., during windshield wiper 203 operation). The tracking target detection unit 150 of the information processing device 100 detects multiple object detection result regions 501 and 502 containing the obstacle 540 from the image 520, which are designated as the subject region of the tracking target (aircraft). Furthermore, in acquiring... Figure 5 When the image 520 is displayed, the likelihood of the subject (airplane) in object detection result region 501 decreases from high to low, while the likelihood of the subject (airplane) in object detection result region 502 increases from low to high. In other words, when acquiring... Figure 5 In the image 520 shown, the likelihood of the subject (aircraft) in object detection result region 502 is higher than the likelihood of the subject (aircraft) in object detection result region 501. In this case, for example, the camera platform control unit 170 of the information processing device 100 performs control to set the subject region of the tracking target subject based on the likelihood of the subject (aircraft) and performs automatic tracking. Specifically, according to this embodiment, for example, the camera platform control unit 170 of the information processing device 100 performs control to set the object detection result region 502 with the highest likelihood of the subject (aircraft) as the subject region of the tracking target subject and performs automatic tracking.

[0072] A more detailed description will follow. Figure 5 Image 520 is shown.

[0073] like Figure 5 As shown in image 520, due to the obstruction of the windshield wiper 203 (which acts as an obstacle 540), the subject area of ​​the object (aircraft) is segmented, resulting in the tracking target detection unit 150 detecting multiple object detection result areas 501 and 502. In this case, located in Figure 5 The object detection result area 501 in the direction of movement of the wiper 203, indicated by the hollow (unfilled) arrow, may decrease, and the likelihood of the subject (aircraft) may also decrease. At this time, for example, the camera platform control unit 170 (CPU 103) continuously monitors the likelihood of the subject (aircraft) in the object detection result area 501.

[0074] Subsequently, for example, if the likelihood of the subject (airplane) in the object detection result region 501, which is the first subject region automatically tracked from image 510, becomes less than or equal to a predetermined threshold, the camera platform control unit 170 (CPU 103) can perform the first and second example processes.

[0075] Specifically, in the first example processing, for example, the camera platform control unit 170 switches the subject region of the tracked target subject (aircraft) from the object detection result region 501 to the object detection result region 502, which is a second subject region with a likelihood greater than or equal to a predetermined threshold, and sets the object detection result region 502.

[0076] Specifically, in the second example processing, for example, the camera platform control unit 170 sets the subject region of the tracking target subject based on the likelihood of the subject (aircraft) and at least one of the degree of overlap with the object detection result region 501 and the degree of adjacency with the object detection result region 501. According to this embodiment, for example, the camera platform control unit 170 switches the subject region of the tracking target subject (aircraft) from the object detection result region 501 to the object detection result region 502 based on the likelihood of the subject and at least one degree, and sets the object detection result region 502.

[0077] In the second example processing described above, the degree of overlap and adjacency can be defined using the following methods. That is, the degree of overlap and / or adjacency can be defined using one or a combination of the following methods: Use AI to define the degree of overlap between the detection bounding box of the subject (airplane) and the area of ​​the object detection result; The degree of overlap with the object detection result area is determined based on the degree of overlap between the detection boxes of the subject (aircraft) and the detection boxes. In this case, the degree of overlap between the detection boxes is determined based on the area of ​​the contact box and / or the area of ​​the overlapping detection boxes. The degree of overlap with the detected object region is determined based on the degree of adjacency with the detected object region. In this case, the degree of adjacency can be determined by the distance between the center points of the detection box of the object (aircraft), the shortest distance from the edge of the detection box to the edge of the detection box, or equivalent methods. The detection box with the largest overlap or adjacency value is designated as the adjacent detection box, and this adjacent detection box is set as the subject area of ​​the target subject (aircraft).

[0078] For example, the camera platform control unit 170 (CPU 103) sends control information to the camera platform device 200 to apply automatic tracking to the object detection result area 502, which is the subject area set as the tracking target. Then, the CPU 204 of the camera platform device 200 controls and drives the drive unit 202 to automatically track the object detection result area 502 based on the control information received from the information processing device 100.

[0079] exist Figure 5 In the image 530, the image is captured by the camera 201 in a state where the windshield wiper 203, acting as an obstacle 540, temporarily obstructs the aircraft (i.e., the state after the operation of the windshield wiper 203). For example, in acquiring... Figure 5 When the image 530 is displayed, the camera platform control unit 170 of the information processing device 100 performs control to set the object detection result area 502 as the subject area for tracking the target subject and performs automatic tracking.

[0080] Figure 6 This is a flowchart illustrating an example of a processing procedure in a method for controlling an information processing apparatus 100 according to a first embodiment. Specifically, Figure 6 This is a flowchart illustrating an example of the processing procedure when the operating mode of the information processing device 100 is automatic tracking mode, and automatic tracking is applied to the detection frame in the subject area of ​​the target subject.

[0081] exist Figure 6 In step S600, when automatic tracking is applied, firstly, in Figure 6 In step S601, for example, the camera platform control unit 170 (CPU 103) determines whether the likelihood of the subject in the detection frame within the subject region where tracking is being applied is less than or equal to a predetermined threshold. When Figure 5 When the windshield wiper 203, which is shown as an obstacle 540, appears in image 520, execution is performed. Figure 6 The process in step S601. Furthermore, according to this embodiment, in... Figure 6 In step S601, when determining the likelihood of the subject, the detection box in the tracked subject region is, for example, Figure 5 The object detection result region 501 in image 520.

[0082] exist Figure 6 In step S601, for example, if the camera platform control unit 170 determines that the likelihood of the subject in the detection frame in the subject area where tracking is being applied is not less than or equal to a predetermined threshold ("No" in step S601), the process remains in step S601.

[0083] On the other hand, Figure 6 In step S601, for example, if the camera platform control unit 170 determines that the likelihood of the subject in the detection frame in the subject area where tracking is being applied is less than or equal to a predetermined threshold ("Yes" in step S601), the process proceeds to step S602.

[0084] For example, in Figure 6 In step S602, the camera platform control unit 170 extracts adjacent detection frames from the detection frames detected by the tracking target detection unit 150 based on at least one of the degree of overlap and adjacency with the detection frames in the subject region where tracking is being applied. According to this embodiment, in Figure 6 The adjacent detection boxes extracted in step S602 are, for example, Figure 5 The object detection result region 502 in image 520.

[0085] Next, in Figure 6 In step S603, for example, the camera platform control unit 170 (CPU 103) determines whether the likelihood of the subject in the adjacent detection frame extracted in step S602 is greater than or equal to a predetermined threshold. Figure 6 The predetermined threshold used in step S603 can be related to Figure 6 The same threshold is used in step S601.

[0086] exist Figure 6 In step S603, for example, if the camera platform control unit 170 determines that the likelihood of the subject in the adjacent detection frame extracted in step S602 is not greater than or equal to a predetermined threshold ("No" in step S603), the process returns to step S601.

[0087] On the other hand, Figure 6 In step S603, for example, if the camera platform control unit 170 determines that the likelihood of the subject in the adjacent detection frame extracted in step S602 is greater than or equal to a predetermined threshold ("Yes" in step S603), the process proceeds to step S604.

[0088] exist Figure 6 In step S604, for example, the camera platform control unit 170 (CPU 103) switches the detection box in the currently applied tracking area of ​​the subject to the target subject from the detection box in the subject area to the adjacent detection box extracted in step S602, and sets the adjacent detection box. In other words, in Figure 5 In image 520, the subject region of the tracked target subject is switched from object detection result region 501 to object detection result region 502, and object detection result region 502 is set.

[0089] exist Figure 6 After the processing in step S604 is terminated, the process returns to step S601.

[0090] The following processing is performed in the information processing apparatus 100 according to the first embodiment.

[0091] Image processing unit 140 acquires an image from camera platform device 200 obtained by photographing a subject with camera 201. Image processing unit 140, configured to perform processing to acquire the image obtained by photographing the subject, constitutes an image acquisition unit. Tracking target detection unit 150 detects a subject region from the image acquired by image processing unit 140, which is a region of the tracking target subject. Tracking target detection unit 150, configured to perform processing to detect the subject region of the tracking target subject, constitutes a detection unit. If at least a portion of the subject in the image acquired by image processing unit 140 is occluded by obstacle 540, and multiple subject regions of the subject are detected by tracking target detection unit 150, camera platform control unit 170 performs the following processing. In this case, camera platform control unit 170 sets the subject region of the tracking target subject based on the likelihood of the subject. Camera platform control unit 170, configured to perform processing to set the subject region of the tracking target subject, constitutes a setting unit.

[0092] The above configuration reduces the loss of the tracked target subject even when at least a portion of the target subject is obscured by the obstacle 540 and multiple subject regions are detected. This allows for continued automatic subject tracking with high accuracy.

[0093] Furthermore, according to the first embodiment, the camera platform control unit 170 can be configured to set the subject region with the highest likelihood among multiple subject regions detected by the tracking target detection unit 150 as the subject region of the tracking target subject.

[0094] Furthermore, according to the first embodiment, when the tracking target detection unit 150 detects multiple subject regions, and the likelihood of a first subject region included in the multiple subject regions and being tracked becomes less than or equal to a threshold, the camera platform control unit 170 performs the following processing. In this case, the camera platform control unit 170 can be configured to switch the subject region of the tracking target subject from the first subject region to a second subject region among the multiple subject regions whose likelihood is greater than or equal to the threshold. Furthermore, in this case, the camera platform control unit 170 can switch the subject region of the tracking target subject from the first subject region to the second subject region based on the subject's likelihood and at least one of the degree of overlap and adjacency with the first subject region.

[0095] Furthermore, according to the first embodiment, the tracking target detection unit 150 can detect the subject region of the tracking target subject in the image acquired from the image processing unit 140, while using AI to perform object detection, wherein the AI ​​uses the input image and outputs a trained model 420 of the subject region.

[0096] Second Embodiment The second embodiment will now be described. It should be noted that in the following description of the second embodiment, the points common to the first embodiment described above are omitted, and only the differences from the first embodiment are described.

[0097] Schematic configuration of the automatic tracking and photography system 10 according to the second embodiment and Figure 1 The schematic configuration of the automatic tracking and imaging system 10 according to the first embodiment is similar. Furthermore, the hardware configuration of the device for the automatic tracking and imaging system 10 according to the second embodiment is similar to... Figure 2 The hardware configuration of the device for the automatic tracking and photography system 10 according to the first embodiment is similar.

[0098] Furthermore, the software configuration of the device for the automatic tracking and photography system 10 according to the second embodiment is... Figure 3 The software configuration of the apparatus of the automatic tracking and photography system 10 according to the first embodiment is similar.

[0099] Figure 7 This diagram illustrates an example of how the information processing apparatus 100 according to the second embodiment performs automatic tracking of an aircraft as a tracking target. Figure 7 In, with Figure 5 The components shown are similar, each given the same reference numerals, and their detailed descriptions are omitted. Furthermore, Figure 7 An example is shown in which the windshield wiper 203 of the camera platform device 200 is designated as an obstacle 540, which exists between the camera 201 and the aircraft, which is the subject being tracked, and obscures at least a portion of the aircraft in the image captured by the camera 201. Furthermore, the windshield wiper 203, acting as obstacle 540, moves left and right on the screen of the camera 201.

[0100] Furthermore, according to the second embodiment, part tracking is performed, wherein the camera 201 is driven and controlled as needed to detect a part region of the subject, and the center position of the part region corresponds to a specified position within the frame of the camera 201.

[0101] exist Figure 7 In, with Figure 5Similarly, the view from camera 201 transitions from image 510 to image 520, and then to image 530. When image 520 is acquired, the windshield wiper 203 appears as an obstacle 540. Furthermore, the detection of regions of the subject's location being tracked using automatic tracking is continuously performed. Similar to AI object detection of the subject, location detection is performed by sending image data from camera 201 to input unit 104 and executing processing from image processing unit 140 to camera platform control unit 170. Additionally, during location tracking, processing from image processing unit 140 of information processing device 100 to camera platform control unit 170 is performed, and operation information is sent to pan / tilt control unit 210 via communication unit 240 of camera platform device 200. At this time, pre-training is also required on the data used by training unit 110 for tracking the subject's location.

[0102] exist Figure 7 In the image 510, the image was captured by camera 201 in a state before the windshield wiper 203 obscured the aircraft, which was the subject, as an obstacle 540 (i.e., before the windshield wiper 203 was operated). In acquiring... Figure 7 When the image 510 is displayed, for example, the camera platform control unit 170 of the information processing device 100 executes control to set the object detection result area 501 (first subject area) as the subject area for tracking the target subject, and performs automatic tracking. According to this embodiment, when acquiring... Figure 7 When the image 510 is shown, the tracking target detection unit 150 of the information processing device 100 pre-detects the object part detection result areas 701 to 704, which are subject part areas that are at least partially included in the object detection result area 501.

[0103] exist Figure 7 In the image 520, at least a portion of the aircraft, which is the subject, is obstructed by the windshield wiper 203, which acts as an obstacle 540. Figure 7 In the example shown, the image was captured by camera 201 in a partially obscured state (i.e., during windshield wiper 203 operation). The tracking target detection unit 150 of the information processing device 100 detects multiple object detection result regions 501 and 502 from the image 520 containing the obstacle 540 as subject regions, which are the areas of the tracking target subject (aircraft). Furthermore, in acquiring... Figure 7 When the image 520 is displayed, the likelihood of the subject (airplane) in object detection result region 501 decreases from high to low, while the likelihood of the subject (airplane) in object detection result region 502 increases from low to high. In other words, when acquiring... Figure 7When the image 520 is shown, the likelihood of the subject (aircraft) in the object detection result region 502 is higher than the likelihood of the subject (aircraft) in the object detection result region 501. Furthermore, for example, if the likelihood of the subject (aircraft) in the object detection result region 501, which is the first subject region from which automatic tracking is being applied from image 510, becomes less than or equal to a predetermined threshold (first threshold), the camera platform control unit 170 of the information processing device 100 can perform the following processing. In this case, for example, the camera platform control unit 170 switches from automatic tracking of the object detection result region 501 to tracking of the subject in the image being acquired. Figure 7 The image 510 shown illustrates automatic part tracking of the subject part regions detected in object part detection result regions 701 to 704. After switching, the subject part regions for which automatic part tracking is applied can be, for example, a single part, such as object part detection result region 701, or multiple parts, such as object part detection result regions 701 and 702. In this case, if multiple object part detection result regions are detected for the target subject, multiple parts can be tracked. For example, a method can be employed where the center of gravity position of the entire subject within the frame of camera 201 is estimated based on the position of each of the multiple target parts, and multiple target parts are tracked.

[0104] Subsequently, if the likelihood of the subject (aircraft) in the object detection result area 501 further decreases and becomes less than or equal to the second threshold (the second threshold is less than a predetermined threshold (the first threshold)), the camera platform control unit 170 may perform the following processing, for example. In this case, the camera platform control unit 170 switches from automatic part tracking of the object part detection result area to automatic tracking of the object detection result area 502 (the second subject area), which is detected by the tracking target detection unit 150 and is different from the object detection result area 501, and sets the object detection result area 502.

[0105] exist Figure 7 In the image 530, the image is captured by the camera 201 in a state where the windshield wiper 203, acting as an obstacle 540, temporarily obstructs the aircraft (i.e., the state after the windshield wiper 203 has been activated). For example, in acquiring... Figure 7 When the image 530 is displayed, the camera platform control unit 170 of the information processing device 100 performs control to set the object detection result area 502 as the subject area for tracking the target subject and performs automatic tracking.

[0106] Furthermore, according to this embodiment, in obtaining Figure 7When the image 530 is shown, the detection result areas 705 to 708 of the object part are detected in preparation for the next obstruction of the subject (aircraft) by the windshield wiper 203, which is an obstacle 540. In other words, the tracking target detection unit 150 of the information processing device 100 pre-detects the object part detection result areas 705 to 708, which are subject part areas that are at least partially included in the object detection result area 502.

[0107] It should be noted that Figure 7 The object part detection result areas 701 to 708 shown may have a subject likelihood parameter during detection, and like the object detection result areas, the object part detection result areas in the object part detection result areas 701 to 708 with a likelihood greater than or equal to a predetermined threshold can be output as detection result areas.

[0108] Figure 8 This is a flowchart illustrating an example of a processing procedure in a method for controlling an information processing apparatus 100 according to a second embodiment. Specifically, Figure 8 This is a flowchart illustrating an example of a processing procedure where the operating mode of the information processing device 100 is automatic tracking mode and automatic tracking is applied to a detection frame in the subject area (including the subject part area) of the target subject.

[0109] exist Figure 8 In step S800, when automatic tracking is being applied, the tracking target detection unit 150 also detects the subject part region that is at least partially included in the subject region of the tracking target subject.

[0110] Then, in Figure 8 In step S801, for example, the camera platform control unit 170 (CPU 103) determines whether the likelihood of the subject in the detection frame in the first subject region where tracking is being applied is less than or equal to a predetermined threshold (first threshold). When Figure 7 When the windshield wiper 203, which is shown as an obstacle 540, appears in image 520, execution is performed. Figure 8 The process in step S801. Furthermore, according to this embodiment, in... Figure 8 In step S801, when determining the likelihood of the subject, the detection box in the tracked subject region is, for example... Figure 7 The object detection result region 501 in image 520.

[0111] exist Figure 8In step S801, for example, if the camera platform control unit 170 determines that the likelihood of the subject in the detection frame in the first subject region where tracking is being applied is not less than or equal to a predetermined threshold (first threshold) ("No" in step S801), the process remains in step S801.

[0112] On the other hand, Figure 8 In step S801, for example, if the camera platform control unit 170 determines that the likelihood of the subject in the detection frame in the first subject area where tracking is being applied is less than or equal to a predetermined threshold (first threshold) ("Yes" in step S801), the process proceeds to step S802.

[0113] exist Figure 8 In step S802, for example, the camera platform control unit 170 (CPU 103) switches the subject region of the target subject from a first subject region to a subject part region that is at least partially included in the first subject region, and sets the subject part region. In other words, the processing in step S802 initiates automatic part tracking of the subject part region.

[0114] Next, in Figure 8 In step S803, for example, the camera platform control unit 170 (CPU 103) determines whether the likelihood of the subject in the detection frame in the first subject region where part tracking is being applied is less than or equal to a second threshold (the second threshold is less than the first threshold).

[0115] exist Figure 8 In step S803, for example, if the camera platform control unit 170 determines that the likelihood of the subject in the detection frame in the first subject region where part tracking is being applied is not less than or equal to the second threshold ("No" in step S803), the process returns to step S801.

[0116] On the other hand, Figure 8 In step S803, for example, if the camera platform control unit 170 determines that the likelihood of the subject in the detection frame in the first subject region where part tracking is being applied is less than or equal to the second threshold ("Yes" in step S803), the process proceeds to step S804.

[0117] exist Figure 8In step S804, for example, the camera platform control unit 170 (CPU 103) calculates a region encompassing the entire subject based on the detection position of the part of the subject being tracked. Next, for example, the camera platform control unit 170 (CPU 103) extracts a detection frame from the calculated region encompassing the entire subject that is different from the detection frame included in the subject region detected by the tracking target detection unit 150 and currently being tracked. According to this embodiment, in Figure 8 The detection box extracted in step S804 is, for example, Figure 7 The object detection result region 502 in image 520.

[0118] Next, in Figure 8 In step S805, for example, the camera platform control unit 170 (CPU 103) switches the subject area of ​​the target subject from the subject area currently being tracked to the subject area in the detection frame extracted in step S804, and sets the subject area.

[0119] exist Figure 8 When the processing in step S805 terminates, the processing returns to step S801.

[0120] The following processing is performed in the information processing apparatus 100 according to the second embodiment.

[0121] The tracking target detection unit 150 also detects a subject region from the image acquired by the image processing unit 140. This subject region is a region that is at least partially included in a subject region that is a region of the tracking target subject. When the tracking target detection unit 150 detects multiple subject regions, and the likelihood of a first subject region included in the multiple subject regions and for which tracking is being applied becomes less than or equal to a first threshold, the camera platform control unit 170 performs the following processing. In this case, the camera platform control unit 170 switches the subject region of the tracking target subject from the first subject region to a subject region that is at least partially included in the first subject region, and sets the subject region. Furthermore, when tracking is applied to a subject region, and the likelihood of the first subject region becomes less than or equal to a second threshold (the second threshold is less than the first threshold), the camera platform control unit 170 performs the following processing. In this case, the camera platform control unit 170 switches the subject area of ​​the tracked target subject from the first subject area (the subject part area of ​​the first subject area) to a second subject area that is included in the multiple subject areas detected by the tracking target detection unit 150 and is different from the first subject area, and sets the second subject area.

[0122] The above configuration reduces the loss of the tracked target subject even when at least a portion of the target subject is obscured by the obstacle 540 and multiple subject regions are detected. This allows for continued automatic subject tracking with high accuracy.

[0123] Third Embodiment Next, a third embodiment will be described. It should be noted that in the following description of the third embodiment, the commonalities with the first and second embodiments described above are omitted, and only the differences from the first and second embodiments are described.

[0124] Schematic configuration of the automatic tracking and photography system 10 according to the third embodiment and Figure 1 The schematic configuration of the automatic tracking and imaging system according to the first embodiment is similar. Furthermore, the hardware configuration of the device for the automatic tracking and imaging system 10 according to the third embodiment is similar. Figure 2 The hardware configuration of the apparatus for the automatic tracking and photography system according to the first embodiment is similar.

[0125] Furthermore, the software configuration of the device for the automatic tracking and photography system 10 according to the third embodiment is... Figure 3 The software configuration of the apparatus of the automatic tracking and photography system 10 according to the first embodiment is similar.

[0126] While, according to the first and second embodiments described above, the windshield wiper 203 of the camera platform device 200 is designated as an obstacle 540 that obstructs at least a portion of the target subject (aircraft) being tracked, the obstacle is not limited to the windshield wiper 203 of this disclosure. Therefore, a third embodiment will be described in which another obstacle besides the windshield wiper 203 (e.g., a street lamp, traffic control tower, antenna, tree, utility pole, part of another parked aircraft, or equivalent) is designated as an obstacle that obstructs at least a portion of the target subject (aircraft) being tracked.

[0127] According to the third embodiment, for obstacles other than the windshield wiper 203, the positions of obstacles along the path captured by the camera 201 can be pre-registered in the data storage unit 120 of the information processing device 100 based on data from the pan / tilt control unit 210 of the camera platform device 200. This processing enables the information processing device 100 or the camera platform device 200 to identify the time when an obstacle passes through the frame during automatic tracking and photography by the camera 201, thereby enabling an automatic tracking and photography system similar to that according to the first and second embodiments.

[0128] Figure 9This is a diagram illustrating an example of image 920 when obstacle 940 is within the frame of camera 201 according to the third embodiment. As described above, it is assumed that obstacle 940 is an obstacle other than windshield wiper 203. Furthermore, before performing automatic subject tracking, information processing device 100 pre-acquires position information about the obstacle 940 that may appear in image 920 based on pan, tilt, and zoom information obtainable from camera platform device 200. Then, before performing automatic subject tracking, information processing device 100 pre-records information indicating when obstacle 940 will appear within the frame of camera 201 in RAM 101 based on the acquired position information about obstacle 940.

[0129] Figure 10 This diagram illustrates a first example of automatic tracking processing performed by the information processing apparatus 100 according to the third embodiment on an aircraft as a tracking target. Figure 10 In, with Figure 5 The components shown are similar, each given the same reference numerals, and their detailed descriptions are omitted. Furthermore, Figure 10 It also shows Figure 9 An obstacle 940 exists between the camera 201 and the aircraft, which is the target being tracked, and obscures at least a portion of the aircraft in the image captured by the camera 201. Furthermore, the obstacle 940 moves left and right within the view of the camera 201 tracking the aircraft.

[0130] exist Figure 10 In, with Figure 5 Similarly, the view from camera 201 changes from image 510 to image 920, and then to image 530. When image 920 is acquired, obstacle 940 appears.

[0131] exist Figure 10 In, with Figure 5 Similarly, image 510 is an image captured by camera 201 in the state before obstacle 940 obstructs the aircraft, which is the subject (i.e., before obstacle 940 passes). The tracking target detection unit 150 of the information processing device 100 detects the object detection result region 501 in image 510 as the subject region, which is the region of the tracking target subject (aircraft). Furthermore, in acquiring... Figure 5 When the image 510 is shown, for example, the camera platform control unit 170 of the information processing device 100 performs control to set the object detection result area 501 as the subject area for tracking the target subject and performs automatic tracking.

[0132] exist Figure 10 In the image 920, at least a portion of the aircraft, which is the subject, is obscured by an obstacle 940 (in... Figure 10In the example shown, the image is captured by camera 201 in a partially obscured state (i.e., the obstacle 940 is within the frame of camera 201). The tracking target detection unit 150 of the information processing device 100 detects multiple object detection result regions 501 and 502 containing the obstacle 940 from the image 920 as subject regions, which are the subject regions of the tracking target subject (aircraft). Furthermore, in acquiring... Figure 10 When image 920 is shown, the likelihood of the subject (airplane) in object detection result region 501 changes from high to low, while the likelihood of the subject (airplane) in object detection result region 502 changes from low to high. In other words, when acquiring... Figure 10 In the image 920 shown, the likelihood of the subject (aircraft) in object detection result region 502 is higher than the likelihood of the subject (aircraft) in object detection result region 501. In this case, for example, the camera platform control unit 170 of the information processing device 100 performs control to set the subject region of the tracking target subject based on the likelihood of the subject (aircraft) and performs automatic tracking. Specifically, according to this embodiment, for example, the camera platform control unit 170 of the information processing device 100 performs control to set the object detection result region 502 with the highest likelihood of the subject (aircraft) as the subject region of the tracking target subject and performs automatic tracking. In this case, in addition to the likelihood of the subject (aircraft), the camera platform control unit 170 may also set the subject region of the tracking target subject (aircraft) based on at least one of the degree of overlap and the degree of adjacency with the object detection result region 501.

[0133] exist Figure 10 In the image 530, the image is captured by camera 201 in a state where the obstacle 940 temporarily obscures the aircraft (i.e., the obstacle 940 is no longer in the frame of camera 201 after it has passed). For example, in acquiring... Figure 10 When the image 530 is displayed, the camera platform control unit 170 of the information processing device 100 performs control to set the object detection result area 502 as the subject area for tracking the target subject and performs automatic tracking.

[0134] Figure 11 This diagram illustrates a second example of automatic tracking processing performed by the information processing apparatus 100 according to the third embodiment on an aircraft as a tracking target. Figure 11 In, with Figure 7 and Figure 10 Each component shown is similar to the one in the attached figure and is given the same reference numerals, with its detailed description omitted.

[0135] exist Figure 11 In, with Figure 10Similarly, the view from camera 201 transitions from image 510 to image 920, and then to image 530. When image 920 is acquired, obstacle 940 appears. Furthermore, the detection of regions of the subject being tracked using auto-tracking is continuously performed. Similar to AI object detection of the subject, region detection is performed by sending image data from camera 201 to input unit 104 and executing processing from image processing unit 140 to camera platform control unit 170. Additionally, during region tracking, processing from image processing unit 140 of information processing device 100 to camera platform control unit 170 is performed, and operation information is sent to pan / tilt control unit 210 via communication unit 240 of camera platform device 200. At this time, pre-training is also required for the regions of the subject being tracked using data from training unit 110.

[0136] exist Figure 11 In, with Figure 10 Similarly, image 510 is an image captured by camera 201 in a state before obstacle 940 obstructs the aircraft, which is the subject (i.e., before obstacle 940 passes through). For example, in acquiring... Figure 11 When the image 510 is displayed, the camera platform control unit 170 of the information processing device 100 executes control to set the object detection result area 501 (first subject area) as the subject area for tracking the target subject, and performs automatic tracking. According to this embodiment, when acquiring... Figure 11 When the image 510 is shown, the tracking target detection unit 150 of the information processing device 100 pre-detects the object part detection result areas 701 to 704, which are subject part areas that are at least partially included in the object detection result area 501.

[0137] exist Figure 11 In, with Figure 10 Similarly, image 920 is where at least a portion of the aircraft, which is the subject, is obscured by obstacle 940 (in... Figure 11 In the example shown, the image is captured by camera 201 in a partially obscured state (i.e., the obstacle 940 is within the frame of camera 201). The tracking target detection unit 150 of the information processing device 100 detects multiple object detection result regions 501 and 502 containing the obstacle 940 from the image 920 as the subject region, which is the region of the tracking target subject (aircraft). Furthermore, in acquiring... Figure 11 When image 920 is shown, the likelihood of the subject (airplane) in object detection result region 501 changes from high to low, while the likelihood of the subject (airplane) in object detection result region 502 changes from low to high. In other words, when acquiring... Figure 11When the image 920 is shown, the likelihood of the subject (aircraft) in object detection result region 502 is higher than the likelihood of the subject (aircraft) in object detection result region 501. Then, for example, if the likelihood of the subject (aircraft) in object detection result region 501, which is the first subject region from which automatic tracking is being applied from image 510, becomes less than or equal to a predetermined threshold (first threshold), the camera platform control unit 170 of the information processing device 100 can perform the following processing. In this case, for example, the camera platform control unit 170 switches from automatic tracking of object detection result region 501 to tracking of the subject in the image being acquired. Figure 11 The image 510 shown illustrates automatic part tracking of the subject part regions detected in object part detection result areas 701 to 704. After switching, the subject part regions for which automatic part tracking is applied can be, for example, a single part, such as object part detection result area 701, or multiple parts, such as target part detection result areas 701 and 702. In this case, when multiple object part detection result areas are detected for the target subject, multiple parts can be tracked. For example, a method can be employed where the center of gravity position of the entire subject within the frame of camera 201 is estimated based on the position of each of the multiple tracked target parts, and multiple tracked target parts are tracked.

[0138] Subsequently, if the likelihood of the subject (aircraft) in the object detection result area 501 further decreases and becomes less than or equal to the second threshold (the second threshold is less than a predetermined threshold (the first threshold)), the camera platform control unit 170 may perform the following processing, for example. In this case, the camera platform control unit 170 switches from automatic part tracking of the object part detection result area to automatic tracking of the object detection result area 502 (the second subject area), which is detected by the tracking target detection unit 150 and is different from the object detection result area 501, and sets the object detection result area 502.

[0139] exist Figure 11 In, with Figure 10 Similarly, image 530 is an image captured by camera 201 in a state where obstacle 940, after temporarily obstructing the aircraft (i.e., the obstacle 940 is no longer in the frame of camera 201 after passing through), ... For example, in acquiring... Figure 11 When the image 530 is displayed, the camera platform control unit 170 of the information processing device 100 performs control to set the object detection result area 502 as the subject area for tracking the target subject and performs automatic tracking. Furthermore, according to this embodiment, when acquiring... Figure 11The image 530 shown detects the object part detection result areas 705 to 708 in preparation for the next new obstacle to obstruct the subject (aircraft). In other words, the tracking target detection unit 150 of the information processing device 100 pre-detects the object part detection result areas 705 to 708, which are subject part areas that are at least partially included in the object detection result area 502.

[0140] It should be noted that Figure 11 The object part detection result areas 701 to 708 shown may have a subject likelihood parameter during detection, and the object part detection result areas in the object part detection result areas 701 to 708 with a likelihood greater than or equal to a predetermined threshold can be output as detection result areas just like object detection result areas.

[0141] Similar to the first and second embodiments, the third embodiment can reduce the loss of the tracked target subject even when at least a portion of the tracked target subject is occluded by the obstacle 940 and multiple subject regions of the subject are detected. This allows for continued automatic subject tracking with high accuracy.

[0142] Other implementation methods This disclosure can also be implemented in a manner in which a program configured to perform one or more functions of the above embodiments is provided to a system or device via a network or storage medium, and one or more processors of the computer of the system or device read and execute the program. Furthermore, this disclosure can also be implemented using circuitry (e.g., an application-specific integrated circuit (ASIC)) configured to perform one or more functions.

[0143] This disclosure includes the program and a computer-readable storage medium storing the program.

[0144] It should be noted that the above embodiments of this disclosure are merely examples of implementation methods of this disclosure, and the technical scope of the present invention should not be construed as being limited solely to these examples. In other words, this disclosure can be implemented in various forms without departing from its spirit or essential characteristics.

[0145] Other implementation methods The embodiments of this disclosure can also be implemented by a computer in a system or apparatus that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform one or more functions of the above embodiments, and / or includes circuitry (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions of the above embodiments, and by methods executed by the computer in the system or apparatus, such as reading and executing computer-executable instructions from the storage medium to perform one or more functions of the above embodiments and / or controlling one or more circuitry to perform one or more functions of the above embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include separate computers or networks of separate processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), the memory of a distributed computing system, an optical disk (such as an optical disc (CD), a digital versatile optical disc (DVD), or a Blu-ray disc (BD)). TM One or more of the following: flash memory devices, memory cards, etc.

[0146] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the following claims should be given the broadest interpretation to cover all such variations and equivalent structures and functions.

[0147] Other embodiments Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU), microprocessor unit (MPU)) of the system or device reads and executes the computer program.

Claims

1. An information processing apparatus, comprising: The image acquisition unit is configured to acquire an image including the subject obtained by a camera; The detection unit is configured to detect one or more subject regions from the image, each subject region being a region of a subject that serves as a tracking target; as well as The setting unit is configured to set the subject region of the subject as the tracking target based on the likelihood of the subject in the subject region. Wherein, when at least a portion of the subject is obscured by an obstacle in the image and multiple subject regions of the subject are detected by the detection unit, the setting unit sets the subject region among the multiple subject regions based on the likelihood that is higher than that of other subject regions among the multiple subject regions.

2. The information processing apparatus according to claim 1, wherein, The setting unit is configured to set the subject region with the highest likelihood among the plurality of subject regions detected by the detection unit as the subject region of the subject that is the tracking target.

3. The information processing apparatus according to claim 1, wherein, When the likelihood of a first subject region included in the plurality of subject regions and being tracked by the detection unit becomes less than or equal to a threshold, the setting unit is configured to switch the subject region of the subject that is the tracking target from the first subject region to a second subject region included in the plurality of subject regions and having a likelihood greater than or equal to the threshold, and is configured to set the second subject region.

4. The information processing apparatus according to claim 3, wherein, When the detection unit is configured to detect the plurality of subject regions, the setting unit switches the subject region of the subject that is the tracking target from the first subject region to the second subject region based on the likelihood and at least one of the degree of overlap with the first subject region and the degree of adjacency with the first subject region, and is configured to set the second subject region.

5. The information processing device according to claim 1, in, The detection unit is configured to further detect a subject region, the subject region being a region at least partially included in the subject region, and Wherein, when the likelihood of the first subject region, which is included in the plurality of subject regions and is being tracked, being detected by the detection unit becomes less than or equal to a threshold, the setting unit is configured to switch the subject region of the subject that is the tracking target from the first subject region to the subject part region that is at least partially included in the first subject region, and is configured to set the subject part region.

6. The information processing apparatus according to claim 5, wherein, When the likelihood of the first subject region becomes less than or equal to a second threshold (the second threshold is less than the threshold) and tracking is being applied to the subject region, the setting unit is configured to switch the subject region of the subject that is the tracking target from the first subject region to a second subject region that is included in the plurality of subject regions and is different from the first subject region, and is configured to set the second subject region.

7. The information processing apparatus according to claim 1, wherein, The detection unit is configured to perform object detection using artificial intelligence (AI), which uses a trained model that takes the image as input and outputs the subject region.

8. A method for controlling an information processing device, the method comprising: To acquire images of the subject by photographing it; Detect one or more subject regions from the image, each subject region being the region of the subject that serves as the tracking target; as well as The subject region of the subject, which is the tracking target, is set based on the likelihood of the subject in the subject region. Wherein, when at least a portion of the subject is occluded by an obstacle in the image, and multiple subject regions of the subject are detected by the detection, the subject region among the multiple subject regions is set based on the likelihood that is higher than that of other subject regions among the multiple subject regions.

9. A storage medium comprising a program that enables a computer to function as each unit of the information processing apparatus according to claim 1.

Images