Tracking device and tracking method

Abstract

To provide a tracking device capable of suppressing a calculation load even when detecting and tracking a plurality of targets simultaneously. [Solution] A tracking device (10A) comprises an imaging processing unit (1) that captures an image of a target that is a candidate for tracking, a target detection processing unit (4) that detects from the image a bounding box, which is the area where the target exists, and a target position, which is the position of the target corresponding to the center point of the bounding box, and calculates a target probability value that indicates the degree of accuracy that the bounding box is the target based on the image, a probability determination processing unit (5A) that determines the tracking target by determining whether the target is a tracking target based on the target probability value and a probability density distribution, and a tracking processing unit (6A) that calculates an estimated position of the tracking target for the determined tracking target by state estimation processing based on the target position and a target position that precedes the target position in the chronological order, and outputs the calculated estimated position as the estimated position of the tracking target.

Description

[Technical Field]

[0001] The present disclosure relates to a tracking device and a tracking method for tracking a target. [Background technology]

[0002] 2. Description of the Related Art Techniques have been proposed for tracking targets such as ships and vehicles based on images captured by a camera.

[0003] The image processing device described in Patent Document 1 tracks a target using a template that serves as a reference image for tracking the target. This image processing device performs template matching by moving the template within a search range, and tracks the target by estimating the position of the part with the highest correlation value as the tracking position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5772572 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 has a problem in that the calculation load increases when multiple targets are detected and tracked simultaneously.

[0006] The present disclosure has been made in consideration of the above, and aims to provide a tracking device that can reduce the calculation load even when detecting and tracking multiple targets simultaneously. [Means for solving the problem]

[0007] To solve the above-mentioned problems and achieve the object, a tracking device disclosed herein includes an imaging processing unit that captures an image of a target that is a candidate for tracking, and a target detection processing unit that detects from the image a bounding box that is an area where the target exists and a target position that is the position of the target corresponding to the center point of the bounding box, and calculates a target probability value that indicates the degree of accuracy that the bounding box is the target based on the image.The tracking device disclosed herein also includes a probability determination processing unit that determines the tracking target by determining whether the target is the tracking target based on the target probability value and a probability density distribution, and a tracking processing unit that calculates an estimated position of the tracking target for the determined tracking target by performing a state estimation process based on the target position and a previous target position that is a target position that precedes the target position in the chronological order, and outputs the calculated estimated position as the estimated position of the tracking target. [Effects of the Invention]

[0008] The tracking device according to the present disclosure has the advantage of being able to reduce the calculation load even when detecting and tracking multiple targets simultaneously. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a tracking device according to a first embodiment; [Figure 2] 1 is a flowchart showing a processing procedure of a process executed by a tracking device according to a first embodiment; [Figure 3] FIG. 1 is a diagram for explaining a process executed by an image scale conversion unit of a tracking device according to a first embodiment; [Figure 4] FIG. 2 is a diagram illustrating a process executed by an image scale conversion unit of the tracking device according to the first embodiment; [Figure 5] FIG. 1 is a diagram for explaining a process executed by a target detection processing unit of a tracking device according to a first embodiment; [Figure 6] FIG. 2 is a flowchart for explaining a process executed by a target detection processing unit of a tracking device according to a first embodiment; [Figure 7]FIG. 3 is a flowchart for explaining a process executed by a target detection processing unit of a tracking device according to a first embodiment; [Figure 8] FIG. 10 is a diagram illustrating a configuration of a tracking device according to a second embodiment. [Figure 9] 10 is a flowchart showing a processing procedure of a process executed by a tracking device according to a second embodiment. [Figure 10] FIG. 10 is a diagram illustrating a configuration example of a processing circuit provided in the tracking device according to the first and second embodiments when the processing circuit is realized by a processor and a memory. [Figure 11] FIG. 10 is a diagram illustrating a configuration example of a processing circuit included in a tracking device according to first and second embodiments, in the case where the processing circuit is configured with dedicated hardware. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, a tracking device and a tracking method according to an embodiment of the present disclosure will be described in detail with reference to the accompanying drawings.

[0011] Embodiment 1 1 is a diagram illustrating the configuration of a tracking device according to a first embodiment. Tracking device 10A is a device that tracks a tracking target such as a ship, a vehicle, etc. In the following, a case where tracking device 10A tracks a ship on the water will be described, but tracking device 10A may also track a tracking target other than a ship.

[0012] Tracking device 10A includes an imaging processing unit 1, a boundary detection unit 2, an image scale conversion unit 3, a target detection processing unit 4, a probability determination processing unit 5A, and a tracking processing unit 6A. Note that tracking device 10A does not necessarily have to include boundary detection unit 2 and image scale conversion unit 3.

[0013] The imaging processing unit 1 captures a time series of images including one or more targets that are candidates for tracking targets (candidates for tracking targets). For example, the imaging processing unit 1 collects visible light or infrared light from the targets using an imaging lens, converts the collected visible light or infrared light into an electrical signal using an imaging element, and converts the electrical signal into a digital signal using an analog-to-digital converter, thereby obtaining an image including the targets. Note that the imaging lens and the imaging element may be external devices separate from the tracking device 10A. In this case, the imaging processing unit 1 receives the electrical signal from the imaging element and obtains an image including the targets from the electrical signal. The following describes a case where the imaging processing unit 1 captures an image including multiple targets.

[0014] The image including the target captured by the image capturing processor 1 is a two-dimensional image. Hereinafter, an "image that may include the target" will be referred to as a "frame." The image capturing processor 1 transmits the captured frame to the boundary detection unit 2.

[0015] The boundary detection unit 2 detects the coordinates of a boundary line (for example, a horizontal line) that indicates whether or not a target exists in the frame received from the image capture processing unit 1. The boundary line detected by the boundary detection unit 2 is the boundary line between an area in the frame where a target may exist and an area where a target does not exist. Hereinafter, the coordinates of the boundary line detected by the boundary detection unit 2 will be referred to as boundary line coordinates.

[0016] If the target is a ship on the water, the area above the horizon within the frame may not contain any ships, but the area below the horizon may contain ships. The boundary detection unit 2 transmits the frame and the detected boundary coordinates to the image scale conversion unit 3 and the target detection processing unit 4.

[0017] The image scale conversion unit 3 performs image scale conversion using the frame captured by the imaging processing unit 1 and the boundary coordinates detected by the boundary detection unit 2 so that a target within a specific size (pixel size) range is present in the frame after image scale conversion. That is, the image scale conversion unit 3 performs image scale conversion so that a target within a predetermined size range (a target of a size to be detected) is included in the frame after image scale conversion. In other words, the image scale conversion unit 3 performs image scale conversion to match an area on the frame where a target of the specific size may be present. For example, the image scale conversion unit 3 performs image scale conversion so that a target larger than the specific size (a target of a desired minimum size) is included in the frame after image scale conversion. Note that the image scale conversion unit 3 may also perform image scale conversion so that a target smaller than the specific size is included in the frame after image scale conversion. The size range preset in the image scale conversion unit 3 is the size range of targets to be detected by the tracking device 10A.

[0018] The image scale conversion unit 3 calculates a conversion ratio for image scale conversion based on a preset maximum detection size, the size of a cutout area for detection, etc. The maximum detection size is the maximum size of a target to be detected. In other words, the maximum detection size is the maximum size of a target to be detected, and the tracking device 10A detects targets that are equal to or smaller than the maximum detection size. The cutout area is an area that the target detection processing unit 4 cuts out from a frame. The tracking device 10A cuts out multiple cutout areas from a frame, and detects a target for each cutout area. Details of the maximum detection size and the cutout areas will be described later. The image scale conversion unit 3 performs image scale conversion using the calculated conversion ratio. Note that the image scale conversion unit 3 may also perform image scale conversion using a preset conversion ratio.

[0019] The image scale conversion unit 3 performs image scale conversion on the frame including the boundary line by also performing image scale conversion on the boundary line coordinates detected by the boundary line detection unit 2. The image scale conversion unit 3 transmits the frame after image scale conversion and the boundary line coordinates to the target detection processing unit 4.

[0020] The target detection processing unit 4 detects targets using either the boundary line coordinates and frame (frame without image scale conversion) received from the boundary line detection unit 2 or the boundary line coordinates and frame (frame after image scale conversion) received from the image scale conversion unit 3. Based on the boundary line coordinates, the target detection processing unit 4 sets the area below the horizon as the target detection range (target detection range), and detects targets from the target detection range. The target detection processing unit 4 is preset with a setting for the direction of the area located in the target detection range. For example, if a ship is detected, the target detection processing unit 4 is set to set the area below the horizon (boundary line) as the target detection range, and if an airplane is detected, the target detection processing unit 4 is set to set the area above the horizon or skyline as the target detection range. The target detection processing unit 4 cuts out clipping areas from the target detection range of the frame and detects targets for each clipping area. The targets detected by the target detection processing unit 4 are candidates for tracking targets (hereinafter referred to as tracking target candidates).

[0021] The target detection processing unit 4 cuts out multiple cut-out regions from the frame so that adjacent cut-out regions partially overlap each other. The size of the cut-out region is the size of the region that the target detection processing unit 4 cuts out from the frame at specific intervals. The size of the cut-out region used by the target detection processing unit 4 is set in advance in the target detection processing unit 4. The size (vertical dimension and horizontal dimension) of the cut-out region is larger than the expected size (vertical dimension and horizontal dimension) of the target.

[0022] The target detection processing unit 4 detects a target area (bounding box) where a tracking target candidate (target) exists and the center point of the bounding box for each clipped area, and calculates a probability value (hereinafter referred to as a target probability value) that indicates the degree of accuracy that the bounding box is the target. That is, the target detection processing unit 4 detects the bounding box and the center point of the bounding box for each clipped area and calculates the target probability value based on either a frame that has not been image scaled or a frame that has been image scaled. In this way, the target detection processing unit 4 detects a target for each clipped area according to the size of the target, and detects the target's bounding box and center point. Note that the target probability value may be calculated using a general object recognition technology such as YOLO (You Only Look Once) v (version) 2.

[0023] A bounding box is an area (e.g., a rectangular area) that includes a target that is a candidate for tracking. The target detection processing unit 4 detects a bounding box and a center point for each candidate for tracking. The bounding box detected by the target detection processing unit 4 may overlap with other bounding boxes within the frame.

[0024] The target position is the center point of the bounding box detected by the target detection processing unit 4. The target detection processing unit 4 transmits a frame including a combination of the center point (target position) of the bounding box and the target probability value of the target corresponding to this center point to the probability determination processing unit 5A.

[0025] The probability determination processing unit 5A determines, for each bounding box (each target), whether or not the target within the bounding box is a tracking target. The probability determination processing unit 5A determines the presence or absence of a tracking target for each target position based on the center point of the bounding box received from the target detection processing unit 4 and the target probability value corresponding to this bounding box, and determines the tracking target. Of the targets detected by the target detection processing unit 4, the probability determination processing unit 5A determines the target to be tracked if it satisfies the condition of the target probability value. The probability determination processing unit 5A determines the tracking target by determining whether or not the target within the bounding box is a tracking target based on, for example, the transition of the target probability value at the center point of the bounding box. The probability determination processing unit 5A removes targets that do not satisfy the condition of the target probability value.

[0026] For example, the probability determination processing unit 5A specifies a specific bounding box within a frame and determines whether a target within this bounding box is a tracking target. The probability determination processing unit 5A excludes targets exhibiting anomalous behavior from the tracking targets and sets targets not exhibiting anomalous behavior as tracking targets. An example of anomalous behavior is when the target probability value or target position shows a fluctuation greater than a specific value. The probability determination processing unit 5A may also exclude targets with low target probability values ​​from the tracking targets and set targets with high target probability values ​​as tracking targets. The probability determination processing unit 5A may also calculate a probability density distribution based on the target probability values ​​detected by the target detection processing unit 4 and determine whether a tracking target exists according to the probability density distribution (probability distribution) of the target probability values. In this case, the probability determination processing unit 5A stores the probability density distribution in advance.

[0027] A bounding box with a lower target probability value is more likely to be a target, and a bounding box with a higher target probability value is more likely to be a target. Therefore, the probability determination processing unit 5A determines that a target within a specified bounding box is a target to be tracked when, for example, the proportion of all target probability values ​​of the specified bounding box over a specific period of time that are higher than a reference value is higher than a specific proportion.

[0028] For example, the probability determination processing unit 5A designates a first bounding box and determines whether or not a target within the first bounding box is a tracking target based on a target probability value during a preset determination period (tracking target determination period). The probability determination processing unit 5A determines that a target within the designated first bounding box is a tracking target when the ratio between the number of frames during the determination period and the number of frames during the determination period in which the target probability value of the first bounding box is higher than a reference value is higher than a specific ratio.

[0029] For example, if the number of frames captured within the determination period is M (M is a natural number), and there are no missed detections of the first bounding boxes, M first bounding boxes will exist within the determination period, and M target probability values ​​will be calculated. If the number of frames within the determination period whose target probability values ​​of the first bounding boxes are higher than the reference value is m (m is a natural number), probability determination processing unit 5A calculates m / M, and if m / M is higher than a specific ratio, determines that the target within the specified first bounding box is a tracking target.

[0030] Even if there is a detection miss in the first bounding box, the probability determination processing unit 5A calculates m / M, and if m / M is higher than a specific ratio, it determines that the target within the specified first bounding box is the target to be tracked.

[0031] The probability determination processing unit 5A determines for each bounding box in the frame, such as the second bounding box and the third bounding box, whether or not the target in the bounding box is a tracking target. The probability determination processing unit 5A transmits the determined tracking target to the tracking processing unit 6A.

[0032] The probability determination processing unit 5A may notify the target determined not to be a tracking target to the target detection processing unit 4. This allows the target detection processing unit 4 to detect the bounding box and the center point while excluding the target determined not to be a tracking target.

[0033] The tracking processing unit 6A calculates an estimate of the position of the tracking target (estimated position of the tracking target) for the tracking target received from the probability determination processing unit 5A by performing a state estimation process based on the target position received from the target detection processing unit 4 and the target position (previous target position) of the previous frame, which is the frame previous to the frame (current frame) containing this target position (for example, the frame immediately before).

[0034] Tracking processing unit 6A calculates the target position of the first bounding box in the (X+2)th frame (estimated position of the tracking target) based on the target position of the first bounding box in the Xth (X is a natural number) frame and the target position of the first bounding box in the (X+1)th frame. Here, the first bounding box in the Xth frame, the first bounding box in the (X+1)th frame, and the first bounding box in the (X+2)th frame are all bounding boxes of the same target.

[0035] Tracking processing unit 6A transmits the estimated position of the tracking target to an external device such as a display device. In this way, tracking device 10A of the first embodiment determines a tracking target from among the targets based on the target probability value, and calculates an estimated position of the determined tracking target.

[0036] Next, an operation example of the tracking device 10A will be described. Fig. 2 is a flowchart showing a processing procedure of processing executed by the tracking device according to the first embodiment. The imaging processing unit 1 of the tracking device 10A captures frames (step S10). As a result, the imaging processing unit 1 obtains frames in time series.

[0037] The boundary detection unit 2 detects the boundary coordinates of a boundary line indicating whether or not a target may exist in the frame captured by the image capture processing unit 1 (step S20). When detecting the boundary coordinates, the boundary detection unit 2 may use information such as the position where the image capture processing unit 1 is attached.

[0038] The image scale conversion unit 3 performs image scale conversion using the frame captured by the image capture processing unit 1 and the boundary line coordinates detected by the boundary line detection unit 2 so that a target within a specific size range is present on the frame after the image scale conversion (step S30). The image scale conversion unit 3 may use a preset maximum detection size used for target detection when performing this image scale conversion. Note that the tracking device 10A does not need to perform image scale conversion of the frame. In this case, the tracking device 10A may not need to include the image scale conversion unit 3. Furthermore, the tracking device 10A may not need to detect boundary line coordinates. In this case, the tracking device 10A may not need to include the boundary line detection unit 2.

[0039] The target detection processing unit 4 cuts out a plurality of cut-out regions from the frame and detects a target in each cut-out region.

[0040] 3 is a first diagram for explaining the processing executed by the image scale conversion unit of the tracking device according to the first embodiment. FIG. 3 shows one frame F1 that includes multiple ships 21 and a horizon 30, which is a boundary line detected by the boundary detection unit 2. For ships 21 located below the horizon 30 in this frame F1, the closer they are to the horizon 30, the larger they tend to appear. The image scale conversion unit 3 performs image scale conversion to make these large targets larger so that they can be detected.

[0041] Fig. 4 is a second diagram for explaining the processing executed by the image scale conversion unit of the tracking device according to the first embodiment. Fig. 4 shows a state in which the image scale conversion unit 3 performs image scale conversion to produce a frame F2 that is smaller than the frame F1 shown in Fig. 3.

[0042] The image scale conversion unit 3 calculates a conversion ratio for image scale conversion based on a preset maximum detection size, the size of a cutout region for detection, and the like.

[0043] The maximum detection size is the maximum size of the bounding box when detecting the bounding box of the target ship 21. The tracking device 10A detects targets up to the size set in the maximum detection size. The maximum detection size is set in advance in the image scale conversion unit 3 according to the size of the target expected after image scale conversion. In other words, the maximum detection size (vertical and horizontal dimensions) is set in advance in the image scale conversion unit 3 so that it is larger than the size (vertical and horizontal dimensions) of all ships 21 expected after image scale conversion. The image scale conversion unit 3 calculates the conversion ratio for image scale conversion based on the preset maximum detection size and the size of the cut-out area larger than the maximum detection size.

[0044] For example, if the size of the cutout area is small, the size of the ship 21 located at the bottom of the frame may be larger than the maximum detection size and the size of the cutout area. The image scale conversion unit 3 calculates a conversion ratio for image scale conversion so that the size of the ship 21 is smaller than the maximum detection size and the size of the cutout area, even if the ship 21 is located at the bottom of the frame. The image scale conversion unit 3 performs image scale conversion based on the conversion ratio for image scale conversion. As a result, the target included in the frame after image scale conversion becomes smaller than the maximum detection size and the size of the cutout area.

[0045] The image scale conversion unit 3 may perform image scale conversion using a preset conversion ratio for image scale conversion. In this case, the conversion ratio for image scale conversion is preset in the image scale conversion unit 3 based on the size of the ships 21 expected before the image scale conversion so that the size of all ships 21 after the image scale conversion is smaller than the maximum detection size and the cutout area.

[0046] Also, here, the image scale conversion unit 3 reduces the size of the frame to detect the area below the horizon 30, which is the boundary line, but the image scale conversion unit 3 may also reduce the size of the frame to detect the area above the boundary line. For example, if the target is a flying object such as an airplane, the image scale conversion unit 3 may reduce the size of the frame to detect the area above the horizon 30 or the horizon line.

[0047] The image scale conversion unit 3 may also enlarge the frame to detect an area above or below the boundary line. That is, the image scale conversion unit 3 may enlarge or reduce the frame to detect one of the areas divided by the boundary line, depending on the target to be detected.

[0048] The target detection processing unit 4 uses either the boundary line coordinates and frame received from the boundary line detection unit 2 or the boundary line coordinates and frame received from the image scale conversion unit 3 to detect the target, the bounding box, and the center point of the bounding box, and calculates the target probability value of this center point.

[0049] The target detection processing unit 4 extracts crop regions from a frame, detects targets from each crop region, and sets bounding boxes for the detected targets. The target detection processing unit 4 uses, for example, common object recognition technologies such as SSD (Single Shot Multibox Detector) or YOLOv2 to detect targets and the center points of bounding boxes. The following describes an example of the operation when the target detection processing unit 4 detects targets using YOLOv2 and calculates target probability values.

[0050] The target detection processing unit 4 detects the center point of the bounding box and calculates the target probability value of this bounding box (step S40) based on the frame including the boundary line coordinates that have been image scaled by the image scale conversion unit 3. Alternatively, the target detection processing unit 4 may detect the center point of the bounding box and calculate the target probability value of this center point based on the boundary line coordinates and frame (a frame that has not been image scaled) received from the boundary line detection unit 2.

[0051] Fig. 5 is a first diagram for explaining the processing executed by the target detection processing unit of the tracking device according to the first embodiment. Fig. 5 explains the processing executed by the target detection processing unit 4 to detect a target using a frame F1 on which image scale conversion has not been performed.

[0052] FIG. 5 shows an example of processing in which the target detection processing unit 4 detects targets near the boundary line using the boundary line coordinates detected by the boundary line detection unit 2 and a frame F1 including these boundary line coordinates.

[0053] The target detection processing unit 4 cuts out cutout regions at regular intervals along the horizontal line 30, which is a boundary line, for example. The target detection processing unit 4 may also cut out cutout regions at regular intervals along the edge of the frame (the lower side of the frame F1), for example. The target detection processing unit 4 may also perform both the process of cutting out cutout regions at regular intervals along the horizontal line 30 and the process of cutting out cutout regions at regular intervals along the edge of the frame F1.

[0054] The target detection processing unit 4 determines, for example, the range where adjacent clipping regions overlap (hereinafter referred to as clipping overlap region) based on a preset maximum detection size. The target detection processing unit 4 determines the size of the clipping overlap region so that the size of the clipping overlap region is larger than the maximum detection size of the target. In other words, the target detection processing unit 4 determines the size of the clipping overlap region so that the clipping overlap region can encompass the maximum size target.

[0055] Note that Fig. 5 shows a case where the size of the cut-out overlapping region is smaller than the maximum detection size of the target. Fig. 5 also shows a case where the target detection processing unit 4 cuts out cut-out regions 51A to 51D, indicated by dashed lines, from frame F1.

[0056] The target detection processing unit 4 detects a target for each of the cutout regions 51A to 51D. The target detection processing unit 4 detects a bounding box and the center point of the bounding box for the detected target, and calculates a target probability value for each bounding box.

[0057] Fig. 6 is a second diagram for explaining the processing executed by the target detection processing unit of the tracking device according to the first embodiment. Fig. 6 explains the processing executed by the target detection processing unit 4 to detect a target using the frame F2 on which image scale conversion has been performed.

[0058] 6 shows an example of processing in which the target detection processing unit 4 detects a target near the edge of frame F2 (below frame F2) using frame F2 that has been image scale converted by the image scale conversion unit 3. The target detection processing unit 4 determines the size of the cut-out overlap region by processing similar to the processing described in FIG.

[0059] The target detection processing unit 4 cuts out cutout regions at regular intervals along the edge of the frame (the lower side of frame F2), for example. The target detection processing unit 4 may also cut out cutout regions at regular intervals along the horizontal line 30, which is a boundary line, for example. The target detection processing unit 4 may also perform both the process of cutting out cutout regions at regular intervals along the horizontal line 30 and the process of cutting out cutout regions at regular intervals along the edge of frame F2.

[0060] FIG. 6 shows a case where the target detection processing unit 4 cuts out cut-out regions 51E to 51G indicated by dashed lines from the frame F2.

[0061] The target detection processing unit 4 detects targets for each of the cutout regions 51E to 51G. The target detection processing unit 4 detects bounding boxes and center points of the bounding boxes for the detected targets, and calculates a target probability value for each bounding box.

[0062] Fig. 7 is a third diagram for explaining the processing executed by the target detection processing unit of the tracking device according to the first embodiment. Fig. 7 shows a bounding box 71 set by the target detection processing unit 4 for frame F1 in Fig. 5 for which image scale conversion has not been performed, and a center point 81 of the bounding box 71.

[0063] The target detection processing unit 4 may set a bounding box and a center point of the bounding box for frame F2 that has undergone the image scale conversion shown in Fig. 6. In this case, the target detection processing unit 4 may calculate the bounding box 71 and center point 81 of the bounding box 71 shown in Fig. 7 by dividing the set bounding box and center point of the bounding box by the conversion ratio of the image scale conversion (by multiplying by the reciprocal of the conversion ratio).

[0064] The probability determination processing unit 5A determines whether or not there is a target to be tracked for each target position based on the bounding box and the target probability value, and determines the target to be tracked (step S50).

[0065] Specifically, the probability determination processing unit 5A determines whether or not there is a tracking target in accordance with the target probability value detected by the target detection processing unit 4 and the probability density distribution for the target probability value, and determines the tracking target. That is, the probability determination processing unit 5A determines whether or not there is a tracking target in accordance with the detected target probability value and the probability density distribution stored in advance, and determines the tracking target. Note that the probability determination processing unit 5A may use, for example, a technique such as the SPRT (Sequential Probability Ratio Test) method to determine the tracking target.

[0066] The tracking processing unit 6A calculates an estimated position of the tracking target determined by the probability determination processing unit 5A by performing a state estimation process based on the transition of the target position for the tracking target (step S60). Specifically, the tracking processing unit 6A calculates an estimated position of the tracking target (the target position in the frame immediately following the current frame) by performing a state estimation process for the tracking target determined by the probability determination processing unit 5A among the targets in the frames detected by the target detection processing unit 4, based on the target position (center point of the bounding box) for the current frame and the target position for the previous frame (for example, the frame immediately preceding the current frame).

[0067] The tracking processing unit 6A may use the center point of the bounding box and an estimated value of the speed of the center point in calculating the estimated position of the tracking target. In this case, the tracking processing unit 6A estimates the speed of the center point of the bounding box based on the distance traveled by the center point of the bounding box and the time required for the movement. The tracking processing unit 6A estimates the estimated position of the tracking target (next target position) based on the center point of the bounding box and the estimated value of the speed of the center point.

[0068] The tracking processing unit 6A may also use a technique such as GNN (Global Nearest Neighbor), JPDA (Joint Probabilistic Data Association), or MHT (Multiple Hypothesis Tracking) in calculating the estimated position of the tracking target. The tracking processing unit 6A also performs delay processing on the output result of the target detection processing unit 4 in order to match the output from the target detection processing unit 4 with the output from the probability determination processing unit 5A.

[0069] As described above, in the first embodiment, the tracking device 10A calculates a target probability value that indicates the degree of accuracy that the target is a target, and determines the tracking target based on the target probability value and the probability density distribution, thereby enabling stable tracking of the target while reducing the calculation load.

[0070] Furthermore, since the tracking device 10A can detect a target without the user having to specify the target, the user does not have to specify the target in advance. Furthermore, since the tracking device 10A does not perform template matching, the calculation load can be reduced even when detecting multiple targets.

[0071] Furthermore, since tracking device 10A cuts out cutout areas from a frame and detects a target from each cutout area, the number of processes to be performed until a target is detected is determined once the frame size and the cutout area size are determined. Therefore, tracking device 10A can easily estimate the processing load until a target is detected, regardless of the number of targets included in the frame, which is the imaging result. Furthermore, since tracking device 10A does not vary the processing load when detecting a target, it is possible to prevent a decrease in the processing rate for target detection and to prevent some targets from becoming undetectable.

[0072] Furthermore, since tracking device 10A can detect targets regardless of the number of targets and the arrangement of targets in an imaged frame, it is possible to reduce the delay time from image capture to tracking, and prevent a decrease in processing rate and a decrease in tracking performance.

[0073] Furthermore, tracking device 10A determines the tracking target based on the target probability value and the probability density distribution, and therefore can accurately determine whether the target has disappeared.

[0074] According to the first embodiment, tracking device 10A determines whether a target is a tracking target based on the target probability value and the probability density distribution, which makes it easy to determine whether a target is a tracking target. As a result, tracking device 10A can detect and track targets with reduced calculation load, even when detecting and tracking multiple targets simultaneously.

[0075] Furthermore, since tracking device 10A extracts extraction regions from a frame and detects a target for each extraction region, fluctuations in the calculation load and the process of detecting overlapping targets can be reduced, and determination accuracy can be improved. Therefore, tracking device 10A can stably track a tracking target while keeping the calculation load when detecting a target small and constant.

[0076] Embodiment 2 Next, a second embodiment will be described with reference to Fig. 8 and Fig. 9. In the second embodiment, all targets are set as candidates for tracking targets (tracking target candidates), and among the tracking targets, tracking target candidates that do not satisfy the condition of the target probability value are excluded from the tracking targets. That is, in the first embodiment, among the targets, targets that satisfy the condition of the target probability value are tracked as tracking targets, but in the second embodiment, tracking is started with all targets as tracking target candidates, and only among the tracking target candidates, tracking target candidates that satisfy the condition of the target probability value are tracked as the final tracking targets.

[0077] Fig. 8 is a diagram showing the configuration of a tracking device according to embodiment 2. Among the components in Fig. 8, components that achieve the same functions as those of tracking device 10A according to embodiment 1 shown in Fig. 1 are assigned the same reference numerals, and redundant explanations will be omitted.

[0078] Similar to tracking device 10A, tracking device 10B of embodiment 2 tracks targets such as ships, vehicles, etc. Tracking device 10B includes an image capture processing unit 1, a boundary detection unit 2, an image scale conversion unit 3, a target detection processing unit 4, a probability determination processing unit 5B, and a tracking processing unit 6B.

[0079] In the tracking device 10B, the target detection processing unit 4 transmits a frame including a combination of the center point of a bounding box (target position) and a target probability value corresponding to the target in the bounding box for each target to the probability determination processing unit 5B. The target detection processing unit 4 also transmits the target position (center point of the bounding box) to the tracking processing unit 6B.

[0080] The tracking processing unit 6B calculates an estimated value of the target position as an estimated position of the tracking target candidate based on the target position in the frame detected by the target detection processing unit 4. Specifically, the tracking processing unit 6B calculates an estimated value of the target position (estimated position of the tracking target candidate) by state estimation processing using the target position received from the target detection processing unit 4 and the target position of a previous frame (e.g., the frame immediately before) that is a frame before the current frame that includes this target position. The tracking processing unit 6B transmits the calculated estimated value of the target position to the probability determination processing unit 5B.

[0081] The probability determination processing unit 5B makes a final decision on the tracking target based on the target position (center point of the bounding box) and target probability value detected by the target detection processing unit 4 and the estimated value of the target position calculated by the tracking processing unit 6B. The probability determination processing unit 5B decides, as the final tracking target, a tracking target candidate that satisfies the condition of the target probability value from among the tracking target candidates (targets) that the tracking processing unit 6B is tracking by calculating the estimated value of the target position. The estimated value of the target position corresponding to the tracking target candidate decided as the final tracking target is the estimated position of the final tracking target. The probability determination processing unit 5B deletes tracking target candidates that do not satisfy the condition of the target probability value.

[0082] Like the probability determination processing unit 5A, the probability determination processing unit 5B determines whether a tracking target candidate within the bounding box is the final tracking target, for example, based on the transition of the target probability value of the bounding box. That is, like the probability determination processing unit 5A, the probability determination processing unit 5B excludes tracking target candidates exhibiting anomalous behavior from the final tracking target and sets tracking target candidates not exhibiting anomalous behavior as the final tracking target. Alternatively, the probability determination processing unit 5B may exclude tracking target candidates with low target probability values ​​from the final tracking target and set tracking target candidates with high target probability values ​​as the final tracking target. Alternatively, the probability determination processing unit 5B may determine whether a final tracking target is present based on the target probability value detected by the target detection processing unit 4 and the probability density distribution for the target probability value. In this case, the probability determination processing unit 5B stores the probability density distribution in advance.

[0083] The probability determination processing unit 5B determines whether to exclude each tracking target candidate from the list of tracking targets or to select it as the final tracking target by the same process as the probability determination processing unit 5A. That is, the probability determination processing unit 5B excludes tracking target candidates that perform anomalous behavior from the list of final tracking targets, and determines tracking target candidates that do not perform anomalous behavior as the final tracking target. The probability determination processing unit 5B also excludes tracking target candidates with low target probability values ​​from the list of final tracking targets, and determines tracking target candidates with high target probability values ​​as the final tracking target.

[0084] The probability determination processing unit 5B may make a final decision on the tracking target based on the position and speed of the tracking target candidate. In this case, the probability determination processing unit 5B estimates the speed of the center point of the bounding box based on the distance traveled by the center point of the bounding box and the time required for the movement. The probability determination processing unit 5B makes a final decision on the tracking target based on the center point of the bounding box (position of the tracking target candidate) and an estimated value of the speed of the center point (speed of the tracking target candidate). For example, if the speed of the center point shows a fluctuation greater than a specific value, the probability determination processing unit 5B excludes the tracking target candidate from the final tracking target.

[0085] Probability determination processing unit 5B transmits the finally determined estimated position of the tracking target to an external device such as a display device. In this way, tracking device 10B of the second embodiment calculates estimated positions of all targets as tracking target candidates, and determines the final tracking target from among the tracking target candidates based on the target probability value.

[0086] The probability determination processing unit 5B may notify the target determined not to be the final tracking target to the target detection processing unit 4. This allows the target detection processing unit 4 to detect the bounding box and the center point while excluding the target determined not to be the final tracking target.

[0087] Next, an example of the operation of the tracking device 10B will be described. Fig. 9 is a flowchart showing the procedure of the processing executed by the tracking device according to the second embodiment. The tracking device 10B executes the processing of steps S10 to S40 in the same manner as the tracking device 10A.

[0088] The tracking processing unit 6B of the tracking device 10B calculates an estimated position of the target (tracking target candidate) by a state estimation process using the target position in the frame detected by the target detection processing unit 4 and the target position in the previous frame (step S51).

[0089] The tracking processing unit 6B uses, for example, an estimated value of the position and velocity of the center point of the bounding box in calculating the estimated position of the tracking target candidate. The tracking processing unit 6B may also use the GNN, JPDA, or MHT technique in calculating the estimated position of the tracking target candidate.

[0090] The probability determination processing unit 5B determines a final tracking target based on the target probability value detected by the target detection processing unit 4 for the estimated position of the target calculated by the tracking processing unit 6B (step S61). That is, the probability determination processing unit 5B determines a final tracking target for the target at the target position in accordance with the target probability value and probability density distribution. Specifically, the probability determination processing unit 5B determines whether a tracking target exists in accordance with the probability density distribution for the target probability value detected by the target detection processing unit 4, and determines the tracking target determined to exist as the final tracking target. Like the probability determination processing unit 5A, the probability determination processing unit 5B uses, for example, the SPRT technique to determine the tracking target.

[0091] The probability determination processing unit 5B may make a final determination of the tracking target based on the speed of the tracking target. That is, the probability determination processing unit 5B may determine the final tracking target in accordance with a probability density distribution corresponding to a target probability value for the position and speed of the tracking target.

[0092] Next, the hardware configuration of the tracking devices 10A and 10B will be described. The tracking devices 10A and 10B are realized by a processing circuit. The processing circuit may be a processor and memory that executes a program stored in a memory, or may be dedicated hardware.

[0093] 10 is a diagram illustrating an example of the configuration of a processing circuit when the processing circuit included in the tracking device according to the first and second embodiments is realized by a processor and a memory. Since the tracking devices 10A and 10B have similar hardware configurations, the hardware configuration of the tracking device 10A will be described here.

[0094] The processing circuit 90 shown in FIG. 10 includes a processor 91 and a memory 92. When the processing circuit 90 is configured with the processor 91 and the memory 92, each function of the processing circuit 90 is realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a tracking program and stored in the memory 92. In the processing circuit 90, each function is realized by the processor 91 reading and executing the tracking program stored in the memory 92. That is, the processing circuit 90 includes the memory 92 for storing the tracking program that results in the processing of the tracking device 10A being executed. This tracking program can also be said to be a program that causes the tracking device 10A to execute each function realized by the processing circuit 90. This tracking program may be provided by a computer-readable recording medium on which the tracking program is recorded, or by other means such as a communication medium.

[0095] The tracking program can also be said to be a program that causes tracking device 10A to execute the processes of steps S10 to S60 in Fig. 2. Here, processor 91 is, for example, a CPU (Central Processing Unit), a processing device, an arithmetic device, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor). Also, memory 92 is, for example, a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), or EEPROM (Electrically EPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a minidisk, or a DVD (Digital Versatile Disc).

[0096] FIG. 11 is a diagram illustrating a configuration example of a processing circuit included in a tracking device according to the first and second embodiments, configured with dedicated hardware. The processing circuit 93 illustrated in FIG. 11 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The processing circuit 93 may be partially implemented with dedicated hardware and partially implemented with software or firmware. In this way, the processing circuit 93 can realize each of the above-described functions by dedicated hardware, software, firmware, or a combination thereof.

[0097] According to the second embodiment, tracking device 10B determines whether a target is a tracking target based on the target probability value and the probability density distribution, similar to tracking device 10A, and therefore can easily determine whether a target is a tracking target. This allows tracking device 10B to track targets with reduced calculation load, even when multiple targets are detected simultaneously.

[0098] Furthermore, tracking device 10B cuts out cut-out regions from a frame and detects a target for each cut-out region, thereby reducing the calculation load.

[0099] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0100] Various aspects of the present disclosure are summarized below as appendices.

[0101] (Appendix 1) an imaging processing unit that captures an image of a target that is a candidate for a tracking target; a target detection processing unit that detects a bounding box, which is an area where the target exists, from the image and a target position, which is a position of the target corresponding to a center point of the bounding box, and calculates a target probability value, which indicates the degree of accuracy that the bounding box is the target, based on the image; a probability determination processing unit that determines whether the target is the tracking target based on the target probability value and the probability density distribution, thereby determining the tracking target; a tracking processing unit that calculates an estimated position of the tracking target by a state estimation process based on the target position and a previous target position that is a target position that precedes the target position in the time series, and outputs the calculated estimated position as the estimated position of the tracking target; Equipped with A tracking device characterized by: (Appendix 2) an imaging processing unit that captures an image of a target that is a candidate for a tracking target; a target detection processing unit that detects a bounding box, which is an area where the target exists, from the image and a target position, which is a position of the target corresponding to a center point of the bounding box, and calculates a target probability value, which indicates the degree of accuracy that the bounding box is the target, based on the image; a tracking processing unit that calculates an estimated position of the target by a state estimation process based on the target position and a previous target position that is a target position that precedes the target position in the time series; a probability determination processing unit that determines whether the target is the tracking target based on the target probability value and the probability density distribution, thereby determining the tracking target, and outputs an estimated position of the target corresponding to the determined tracking target as an estimated position of the tracking target; Equipped with A tracking device characterized by: (Appendix 3) a boundary detection unit that detects boundary coordinates that are coordinates of a boundary between an area in the image where the target may exist and an area where the target does not exist, the target detection processing unit detects the bounding box and the target position based on the boundary line coordinates and the image; 3. The tracking device according to claim 1 or 2. (Appendix 4) an image scale conversion unit that performs image scale conversion of the image based on the boundary line and the image so that a target within a predetermined size range is included in a frame after image scale conversion; the target detection processing unit detects the bounding box and the target position based on the boundary line coordinates after image scale conversion and the image; 4. The tracking device according to claim 3, (Appendix 5) the target detection processing unit cuts out a plurality of cutout regions from the image, detects the bounding box and the target position for each cutout region, and calculates the target probability value; 5. A tracking device according to any one of claims 1 to 4. (Appendix 6) the target detection processing unit determines an overlapping range between adjacent clipping regions based on a maximum detection size when detecting the bounding box. 6. The tracking device according to claim 5, (Appendix 7) the target detection processing unit sets a target detection range, which is a detection range of the target, based on the boundary line coordinates and the image, and detects the bounding box and the target position from an area within the target detection range. 5. The tracking device according to claim 3 or 4. (Appendix 8) the target detection processing unit detects the target position using a general object recognition technology; 8. A tracking device according to any one of claims 1 to 7. (Appendix 9) an imaging processing step in which the tracking device captures an image of a target that is a candidate for a tracking target; a target detection processing step in which the tracking device detects a bounding box, which is an area where the target exists, from the image and a target position, which is the position of the target corresponding to the center point of the bounding box, and calculates a target probability value, which represents the degree of accuracy that the bounding box is the target, based on the image; a probability determination processing step in which the tracking device determines whether the target is the tracking target based on the target probability value and the probability density distribution, thereby determining the tracking target; a tracking processing step in which the tracking device calculates an estimated position of the tracking target by performing a state estimation process based on the target position and a previous target position that is a target position that precedes the target position in the time series, and outputs the calculated estimated position as the estimated position of the tracking target; Including, A tracking method characterized by: (Appendix 10) an imaging processing step in which the tracking device captures an image of a target that is a candidate for a tracking target; a target detection processing step in which the tracking device detects a bounding box, which is an area where the target exists, from the image and a target position, which is the position of the target corresponding to the center point of the bounding box, and calculates a target probability value, which represents the degree of accuracy that the bounding box is the target, based on the image; a tracking processing step in which the tracking device calculates an estimated position of the target by a state estimation process based on the target position and a previous target position that is a target position that precedes the target position in the time series; a probability determination processing step in which the tracking device determines whether the target is the tracking target based on the target probability value and the probability density distribution, thereby determining the tracking target, and outputting an estimated position of the target corresponding to the determined tracking target as the estimated position of the tracking target; Including, A tracking method characterized by: [Explanation of symbols]

[0102] 1 imaging processing unit, 2 boundary detection unit, 3 image scale conversion unit, 4 target detection processing unit, 5A, 5B probability judgment processing unit, 6A, 6B tracking processing unit, 10A, 10B tracking device, 21 ship, 30 horizon, 51A to 51G cut-out area, 71 bounding box, 90, 93 processing circuit, 91 processor, 92 memory, F1, F2 frames.

Claims

1. an imaging processing unit that captures an image of a target that is a candidate for a tracking target; a target detection processing unit that detects a bounding box, which is an area where the target exists, from the image and a target position, which is a position of the target corresponding to a center point of the bounding box, and calculates a target probability value, which indicates the degree of accuracy that the bounding box is the target, based on the image; a probability determination processing unit that determines whether the target is the tracking target based on the target probability value and the probability density distribution, thereby determining the tracking target; a tracking processing unit that calculates an estimated position of the tracking target by a state estimation process based on the target position and a previous target position that is a target position that precedes the target position in the time series, and outputs the calculated estimated position as the estimated position of the tracking target; Equipped with A tracking device characterized by:

2. an imaging processing unit that captures an image of a target that is a candidate for a tracking target; a target detection processing unit that detects a bounding box, which is an area where the target exists, from the image and a target position, which is a position of the target corresponding to a center point of the bounding box, and calculates a target probability value, which indicates the degree of accuracy that the bounding box is the target, based on the image; a tracking processing unit that calculates an estimated position of the target by a state estimation process based on the target position and a previous target position that is a target position that precedes the target position in the time series; a probability determination processing unit that determines whether the target is the tracking target based on the target probability value and the probability density distribution, thereby determining the tracking target, and outputs an estimated position of the target corresponding to the determined tracking target as an estimated position of the tracking target; Equipped with A tracking device characterized by:

3. a boundary detection unit that detects boundary coordinates that are coordinates of a boundary between an area in the image where the target may exist and an area where the target does not exist, the target detection processing unit detects the bounding box and the target position based on the boundary line coordinates and the image; 3. The tracking device according to claim 1 or 2.

4. an image scale conversion unit that performs image scale conversion of the image based on the boundary line and the image so that a target within a predetermined size range is included in a frame after image scale conversion; the target detection processing unit detects the bounding box and the target position based on the boundary line coordinates after image scale conversion and the image; 4. The tracking device according to claim 3.

5. the target detection processing unit cuts out a plurality of cutout regions from the image, detects the bounding box and the target position for each cutout region, and calculates the target probability value; 2. The tracking device according to claim 1.

6. the target detection processing unit determines an overlapping range between adjacent clipping regions based on a maximum detection size when detecting the bounding box.

6. The tracking device according to claim 5.

7. the target detection processing unit sets a target detection range, which is a detection range of the target, based on the boundary line coordinates and the image, and detects the bounding box and the target position from an area within the target detection range.

4. The tracking device according to claim 3.

8. the target detection processing unit detects the target position using a general object recognition technology; 2. The tracking device according to claim 1.

9. an imaging processing step in which the tracking device captures an image of a target that is a candidate for a tracking target; a target detection processing step in which the tracking device detects a bounding box, which is an area where the target exists, from the image and a target position, which is the position of the target corresponding to the center point of the bounding box, and calculates a target probability value, which represents the degree of accuracy that the bounding box is the target, based on the image; a probability determination processing step in which the tracking device determines whether the target is the tracking target based on the target probability value and the probability density distribution, thereby determining the tracking target; a tracking processing step in which the tracking device calculates an estimated position of the tracking target by performing a state estimation process based on the target position and a previous target position that is a target position that precedes the target position in the time series, and outputs the calculated estimated position as the estimated position of the tracking target; Including, A tracking method characterized by:

10. an imaging processing step in which the tracking device captures an image of a target that is a candidate for a tracking target; a target detection processing step in which the tracking device detects a bounding box, which is an area where the target exists, from the image and a target position, which is the position of the target corresponding to the center point of the bounding box, and calculates a target probability value, which represents the degree of accuracy that the bounding box is the target, based on the image; a tracking processing step in which the tracking device calculates an estimated position of the target by a state estimation process based on the target position and a previous target position that is a target position that precedes the target position in the time series; a probability determination processing step in which the tracking device determines whether the target is the tracking target based on the target probability value and the probability density distribution, thereby determining the tracking target, and outputting an estimated position of the target corresponding to the determined tracking target as the estimated position of the tracking target; Including, A tracking method characterized by:

Images