Tracking device and tracking method

Abstract

Provided is a tracking device comprising: a division processing unit that, for each frame image of a frame image group including a plurality of frame images, on the basis of a tracking result obtained in any other frame image followed by or following the frame image, combines a result of tracking an object in an inference image selected from among a plurality of divided images obtained by dividing the frame image with a result of tracking an object in a downsampled image other than the inference image among the plurality of divided images; an overall processing unit that, for each of the frame images, tracks an object in a reduced image obtained by reducing the frame image; and a synthesis unit that, for each of the frame images, outputs, as a tracking result of the frame image, overlapping tracking results among tracking results obtained by synthesizing a tracking result from the division processing unit and a tracking result from the overall processing unit with an excessive tracking result removed.

Description

Tracking device and tracking method

[0001] The disclosed technologies relate to tracking devices and tracking methods.

[0002] Object tracking devices are devices that simultaneously track the positions of multiple specific objects from a video or a series of images. In recent years, object detection methods, such as You Only Look Once (YOLO) (see Non-Patent Document 1), have attracted attention. Object detection methods are applied to various AI (Artificial Intelligence) applications, including facial recognition and surveillance systems. Object tracking technology, an extension of object detection, is an essential element technology for realizing AI applications such as pedestrian flow analysis and traffic flow analysis, and demand is increasing. Simple Online and Realtime Tracking (SORT) (see Non-Patent Document 2) has been proposed as a method specifically for real-time object tracking. In this method, each video frame is given as input to an object detection model like YOLO, and the degree of overlap between the detection frame position predicted from the previous frame and the detection frame actually obtained in the current frame is calculated for all detection frames. By assigning the ID (Identification) to each detection frame obtained in the previous frame to the frame detected in the next frame, it becomes possible to continue tracking the same object. SORT addresses the assignment of IDs to frames by treating it as a maximum-weighted matching problem of a bipartite graph with repetition as the weight, and optimizing it using the Hungarian algorithm.

[0003] Joseph Redmon et al., “YOLOv3: An Incremental Improvement”. https: / / arxiv.org / abs / 1804.02767A. Bewley et al., “SIMPLE ONLINE AND REALTIME TRACKING”. https: / / arxiv.org / abs / 1602.00763

[0004] Incidentally, the Hungarian algorithm used to pass IDs between frames during object tracking is known to have a computational complexity that increases exponentially on the order of the cube of the number of objects being tracked. Therefore, with conventional object tracking techniques, the computational complexity can increase exponentially when trying to track objects in a set of frame images containing multiple frame images, especially a large number of objects in high-definition video.

[0005] The technology disclosed herein has been made in view of the above points and provides a tracking device and a tracking method that can reduce the computational cost when tracking an object across multiple frame images.

[0006] The tracking device of this disclosure includes, for each frame image in a group of frame images including a plurality of frame images in a time series, a division processing unit that combines the result of tracking an object in an inference image selected from a plurality of divided images obtained by dividing the frame image with the result of tracking an object in a reduced image other than the inference image among the plurality of divided images, based on tracking results obtained from other frame images either before or after the frame image; an overall processing unit that tracks an object in a reduced image obtained by reducing the frame image for each frame image; and a compositing unit that, for each frame image, removes any redundant tracking results from the tracking result obtained by combining the tracking result of the division processing unit and the tracking result of the overall processing unit and outputs the result as the tracking result of the frame image.

[0007] Furthermore, the tracking method of this disclosure includes, for each frame image in a group of frame images including a plurality of consecutive frame images in time series, a division process that combines the result of tracking an object in an inference image selected from a plurality of divided images obtained by dividing the frame image with the result of tracking an object in a downsampled image other than the inference image among the plurality of divided images, based on the tracking result obtained from another frame image either before or after the frame image; for each frame image, an overall process that tracks an object in a reduced image obtained by reducing the frame image; and for each frame image, a process that combines the tracking result of the division process and the tracking result of the overall process, removes any redundant tracking results, and outputs the result as the tracking result for the frame image.

[0008] The technology disclosed herein can reduce the computational complexity required when tracking an object across multiple frame images.

[0009] This is a functional block diagram showing an example of the configuration of the tracking device of the embodiment. This is a diagram illustrating an example of the operation of the tracking device. This is a diagram illustrating an example of the operation of the tracking device. This is a diagram illustrating an example of the operation of the overlap determination unit of the synthesis section of the tracking device. This is a diagram illustrating another example of the operation of the overlap determination unit of the synthesis section of the tracking device. This is a diagram illustrating an example of the operation of the identification information determination unit of the synthesis section of the tracking device. This is a diagram illustrating another example of the operation of the identification information determination unit of the synthesis section of the tracking device. This is a diagram illustrating an example of the operation of the tracking device of the second embodiment. This is a functional block diagram showing the division processing section and the overall processing section of the second embodiment. This is a diagram illustrating an example of the operation of the tracking device of the second embodiment. This is a configuration diagram showing an example of the hardware configuration of the tracking device of the embodiment. This is a flowchart illustrating an example of the flow of the tracking process performed by the tracking device of the embodiment.

[0010] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals. Furthermore, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0011] [First Embodiment] The tracking device of this embodiment is a device that detects objects in each frame image of a group of frame images that include multiple frame images in a time series and tracks them across consecutive frames. Furthermore, the tracking device of this embodiment is capable of tracking multiple objects contained in a single frame image. Specifically, the tracking device of this embodiment assigns identification information (hereinafter referred to as ID) to the detection frame surrounding the object detected in the frame image, and tracks the detection frame to which the same identification information is assigned in each frame image, thereby enabling object tracking. Therefore, the device performs the process of detecting an object from each frame image, generating a detection frame for the object, and assigning an ID to the detection frame. Examples of frame image groups used for object tracking in the tracking device include high-definition video (images).

[0012] First, the configuration of the tracking device of this embodiment will be described. Figure 1 is a functional block diagram showing an example of the configuration of the tracking device 10 of this embodiment. As shown in Figure 1, the tracking device 10 of this embodiment includes a division processing unit 30, an overall processing unit 32, and a synthesis unit 34.

[0013] The following section describes the case where the processing target is a group of frame images F containing x frame images f (f_1 to f_x).

[0014] In the tracking device 10, for each frame image f in the frame image group F, the division processing unit 30 divides the frame image f into multiple partial images and performs division processing to detect and track objects. The overall processing unit 32 performs overall processing to detect and track objects in the reduced images obtained by reducing the size of the frame image f. The synthesis unit 34 combines the tracking results of the division processing unit 30 and the tracking results of the overall processing unit 32 for each frame image f to obtain the object tracking results for the frame image group F.

[0015] The operation of each part of the tracking device 10 will be explained in detail with reference to Figure 2. Figure 2 shows an outline of an example of the flow of processing performed by the tracking device 10. In the example shown in Figure 2, the case where object tracking is performed in real time for the Nth (x≧N>1)th frame image f_N included in the frame image group F is illustrated. When the tracking device 10 performs object tracking in real time in response to the acquisition of frame image f, frame image f_N will be the image at the current time or the latest image.

[0016] The division processing unit 30 divides the frame image f_N into multiple divided images 50. A certain overlapping region is provided between the partial images 50 for the purpose of determining identical objects in the compositing unit 34.

[0017] Figure 2 shows an embodiment in which the division processing unit 30 divides one frame image f into four divided images 50 (50_1 to 50_4). The number of divisions the division processing unit 30 makes to the frame image f is not limited to this embodiment. For example, it may be determined according to the size of the frame image f, the processing capacity of the tracking device 10, and the desired accuracy.

[0018] The segmentation processing unit 30 predicts the position of the detection frames in all partial images 50 of frame image f_N based on the positions of the ID-assigned detection frames 80 (80_1 to 80_7) obtained in the frame image f_N-1, which is the frame image immediately preceding frame image f_N. The method by which the segmentation processing unit 30 predicts the position of the detection frames 80 in frame image f_N based on the detection frames 80 in frame image f_N-1 is not limited, and examples include using motion vectors obtained by block matching between frames or a Kalman filter.

[0019] The segmentation processing unit 30 selects a specific face from among the segmented partial images 50_1 to 50_4 as the inference image 60. Figure 2 shows the case where partial image 50_1 is selected as the inference image 60. The segmentation processing unit 30 performs object detection processing on the inference image 60 and generates new detection frames 82 (82_1 to 82_3) along with class probabilities (confidence levels), regardless of the detection frames 80 detected in the frame image f_N-1.

[0020] As shown in Figure 3, the division processing unit 30 calculates the overlap IoU (Intersection over Union) of the detection frame 82 generated from frame image f_N and the detection frame 80 predicted from frame image f_N-1, and the area A of the detection frame 82 generated from frame image f_N. ｄｅｔｅｃｔ And the area A of the detection frame 80 predicted from frame image f_N-1. ｐｒｅｄｉｃｔ The area of ​​the overlapping region of both detection frames (hereinafter referred to as the overlapping area) A ｄｕｐｌｉｃａｔｅ Therefore, the calculation is performed using the following equation (1). The calculation of the redundancy IoU using the above equation (1) is performed for all detection frames 82 in the inference image 60.

[0021] ... (1)

[0022] As shown in Figure 3, the division processing unit 30 creates an IoU weight table using the overlap degree IoU as the weight, and determines how to assign IDs to the detection results (detection frames) using the Hungarian algorithm, treating it as a problem of maximizing the sum of IoUs. Note that the closer the overlap degree IoU is to 1, the higher the overlap degree and the more likely the ID is to be carried over. In the example shown in Figure 3, ID=1 is assigned to detection result I, ID=2 is assigned to detection result II, and ID=3 is assigned to detection result III.

[0023] As shown in Figure 2, the division processing unit 30 selects all but the partial image 50_1 to 50_4 obtained by dividing the frame image f_N as inference image 60, and uses them as downsampled images 52_1 to 52_3. That is, partial images 50_2 to 50_4 are used as downsampled images 52_1 to 52_3. In the downsampled image 52, the position of the object detection frame 80 with an ID obtained in the frame image f_N is predicted using motion vectors, and this is output to the synthesis unit 34 as the tracking result.

[0024] The segmentation processing unit 30 combines the tracking results obtained from the inference image 60 and the tracking results obtained from the downsampled image 52 to obtain a provisional object tracking result for the entire frame image f_N.

[0025] On the other hand, the overall processing unit 32 scales the frame image f_N, reduces it to a pixel size that can be processed by the object detection model, and generates a reduced image 70. The overall processing unit 32 performs the same processing on the reduced image 70 as the segmentation processing unit 30 performs on the partial image 50. The overall processing unit 32 predicts the position of the detection frame 80 based on the motion vector for the entire surface of the frame image f_N from the object tracking result of the reduced image 70 in the frame image f_N-1. Next, the overall processing unit 32 determines whether to thin out the object detection for the entire surface of the frame image f_N. If not thinned out, it executes object detection and ID assignment processing to obtain an object tracking result for the entire surface. If thinned out, similar to the processing of the segmentation processing unit 30, it outputs the prediction result based on the motion vector for each of the detection frames 80 in the reduced image 70 of the frame image f_N-1 to the synthesis unit 34 as a tracking result.

[0026] The synthesis unit 34 synthesizes the tracking result obtained by the segmentation processing unit 30 and the tracking result obtained by the overall processing unit 32. The synthesis unit 34 includes a duplication determination unit 40 and an identification information determination unit 42.

[0027] In the tracking result obtained by the segmentation processing unit 30, excessive detection frames 80 are generated for objects detected across the segmentation boundary. In other words, for an object existing across the segmentation boundary, multiple detection frames 80 are generated for one object. Therefore, the duplication determination unit 40 performs synthesis with the detection result obtained by the overall processing unit 32 to remove the excessive detection frames 80.

[0028] The duplication determination unit 40 determines whether the detection frames 80 are for the same object (same object determination) based on the overlapping area between the detection frames 80 obtained by the segmentation processing unit 30 and the detection frames 80 obtained by the overall processing unit 32. In the overall processing unit 32, as described above, the detection frame 80 is obtained in the reduced image 70. However, the area of the detection frame 80 obtained by the overall processing unit 32 is not the area of the detection frame 80 in the reduced image 70, but the area of the detection frame 80 when the reduced image 70 is restored to the size of the original frame image f_N. Hereinafter, the area of the detection frame 80 obtained by the overall processing unit 32 is referred to as the area of the detection frame obtained in the overall image.

[0029] First, a case where a relatively large object with a size detectable by the overall processing unit 32 is located across the image segmentation boundary will be described. An example of this case is shown in FIG. 4A as a case where a large person is segmented by image segmentation. The overlap determination unit 40 calculates the ratio A ｄｉｖ of the area A ｄｕｐｌｉｃａｔｅ of the detection frame obtained in the partial image 50 ｄｕｐｌｉｃａｔｅ to the overlap area A ｄｉｖ of the detection frame obtained in the overall image. When this ratio A

[0030] exceeds a certain threshold α, they are determined as detection frames 80 for the same object. The overlap determination unit 40 outputs only the detection frame 80 obtained in the overall image as the synthesis result. ｄｕｐｌｉｃａｔｅ Next, a case where a small object that cannot be detected by the overall processing unit 32 exists across the segmentation boundary will be described. An example of this case is shown in FIG. 4B as a case where a small person is segmented by image segmentation. In this case, detection frames are generated in a plurality of partial images 50, and the same object determination is performed based on the overlap index in the overlap region between the partial images. In the example shown in FIG. 4, the same object determination is performed based on the overlap index in the overlap region between partial image a and partial image b. The overlap determination unit 40 calculates the ratio A ｏｖｅｒｌａｐ of the overlap area A ｄｕｐｌｉｃａｔｅ to the detection frame area A ｏｖｅｒｌａｐ in the overlap region. When this ratio A

[0031] exceeds a certain threshold β, they are determined as detection frames 80 for the same object, the detection frames with low reliability are removed, and the detection frames with high reliability are output as the synthesis result. In the example shown in FIG. 4, the overlap determination unit 40 outputs the detection frame 80 in partial image a with a reliability of 80% as the synthesis result.Furthermore, the identification information determination unit 42 performs a synthesis process of the IDs assigned to the detection frames 80. As shown in Figures 5A and 5B, when performing the above-mentioned identical object determination, the number of consecutive decimations up to frame image f_N is compared in the overall image, which is the entire frame image f, and in each of the partial images 50. The ID of the detection frame 80 belonging to the image with fewer consecutive decimations is inherited into the synthesized detection frame 80, and the other IDs are discarded. This is because the accuracy of predicting the position of the detection frame 80 by motion vectors decreases as the number of consecutive decimations increases, and the object tracking accuracy decreases. If the number of consecutive decimations is the same, it is considered that there is almost no difference in prediction accuracy, so one of the predetermined IDs is retained. The identification information determination unit 42 overwrites the detection frames obtained from the overall image and the detection frames obtained from the partial images 50 with the ID inherited during synthesis, and uses it for object tracking processing of the next frame image (frame image f_N+1) as the object tracking result in the previous frame. In the example shown in Figure 5A, the number of consecutive decimation steps in the overall image is 3, in partial image 50_1 it is 5, and in partial image 50_3 it is 6. Therefore, the identification information determination unit 42 inherits the ID for the detection frame 80 in the overall image, which has the fewest consecutive decimation steps, for both the overall image and the partial image 50. In the example shown in Figure 5B, no object is detected in the overall image and no detection frame is generated, and the number of consecutive decimation steps in partial image 50_1 is 3, and in partial image 50_3 it is 4. Therefore, the identification information determination unit 42 inherits the ID for the detection frame 80 in partial image 50_1, which has the fewest consecutive decimation steps, for the partial image 50.

[0032] [Second Embodiment] The tracking device 10 of this embodiment differs from the tracking device 10 of the first embodiment in that, in addition to the degree of overlap IoU between detection frames, the coordinate distance is used as an indicator when assigning IDs. In conventional SORT, the position of the ID-assigned detection frame obtained in the previous or next frame image f is predicted in the current frame image, and the degree of overlap IoU with the new detection frame obtained through object detection in the current frame image is calculated and the assignment of IDs is determined based on this. Therefore, if the degree of overlap IoU is "0", it is considered to be a different object, and the ID that was assigned to the object in the previous frame image is not inherited, and a new ID is assigned. On the other hand, in the tracking device 10 of the first embodiment, there are partial images 50 or whole images in which object detection processing is continuously thinned out over a long period of frames. Therefore, in such images, the accuracy of predicting the detection frame position by motion vector deteriorates, and as a result, there are cases in which the predicted position of the detection frame and the actual position of the object deviate greatly. In the example shown in Figure 6, the partial image corresponding to inference image 60_1 was not selected as the continuous inference image 60, but was a downsampled image. As a result, when it was selected as inference image 60_1, there was a large discrepancy between the predicted position of the detection frame and the actual detection area (position of the object).

[0033] When an ID is assigned to this event, the degree of overlap (IoU) between the predicted detection frame position during the continuous thinning process and the newly generated detection frame position is calculated as "0". In other words, they are considered not to overlap. Therefore, they are judged as non-identical objects, resulting in an ID swap.

[0034] Therefore, in this embodiment, the division processing unit 30 and the overall processing unit 32 of the tracking device 10 each include a distance calculation unit 46 and an assignment unit 48, as shown in Figure 7. Similar to the tracking device 10 of the first embodiment, the predicted position of the detection frame j and the degree of overlap IoU of the detection frame i are first calculated, and the ID is assigned by solving the maximum weighted matching problem of a bipartite graph using the Hungarian algorithm with these as weights. Subsequently, the distance calculation unit 46 calculates the coordinate distance d between the detection frame i and the predicted position of the detection frame j for detection frames i where the degree of overlap IoU is calculated to be "0" and no ID is assigned. ｉｊThe allocation unit 48 also calculates the set threshold D. ｔｈ If the distance is less than the specified value, a new ID will be assigned with that distance as the cost.

[0035] For all combinations of detection frames in which the overlap IoU was calculated to be "0", the coordinate distance d ｉｊ Figure 8 shows two examples, Example 1 and Example 2, in which the threshold D is calculated and used as a cost table to assign IDs using the Hungarian algorithm. Example 1 is calculated as the threshold D ｔｈ This is the case where the same value is applied to all detection frames. When creating the cost table, the coordinate distance d ｉｊ For those that exceed the threshold, a pre-set value d ｍａｘ This imposes a high cost to prevent the ID from being incorrectly inherited between the detection frames.

[0036] The above implementation example is effective for a series of frames F where the apparent movement speed of each object in the image varies relatively little, such as in aerial footage taken from directly above. On the other hand, tracking accuracy may deteriorate in a series of frames F where the apparent movement speed variation is large. For example, if the position coordinates in the depth direction differ for each object, such as in footage taken from an oblique angle, the apparent movement speed of the objects in the image will differ, and objects closer to the viewer will have a longer coordinate distance d. ｉｊ Because the increase is particularly noticeable, a uniformly set threshold D ｔｈ If D is too small, the accuracy of assigning IDs to objects in the foreground will deteriorate. Conversely, D ｔｈ If the threshold is too large, incorrect ID assignments will occur between objects that appear small in the background. Therefore, in Example 2, the object position in the depth direction is estimated from the size of the detection frame and the moving average distance, and the threshold D ｔｈ （ｉ） The aforementioned problem is addressed by dynamically controlling the detection frame i for each detection frame.

[0037] The tracking device 10 of each of the above embodiments can be realized with the hardware shown in Figure 9. A block diagram showing an example of the hardware configuration of the tracking device 10 of this embodiment is shown. As shown in Figure 9, the tracking device 10 includes a CPU (Central Processing Unit) 12, a ROM (Read Only Memory) 14, a RAM (Random Access Memory) 16, a storage 18, an input unit 20, a display unit 22, and a communication I / F (Interface) 24. The CPU 12, ROM 14, RAM 16, storage 18, input unit 20, display unit 22, and communication I / F 24 are connected to each other so as to be able to communicate with each other via a bus 29.

[0038] The CPU 12 is a central processing unit that executes various programs such as program 19 and controls each part. Specifically, the CPU 12 reads a program from ROM 14 or storage 18 and executes the program using RAM 16 as a working area. The CPU 12 controls each of the above components and performs various calculations according to the program stored in ROM 14 or storage 18. In this embodiment, program 19, which will be described in detail later, is stored in storage 18. Program 19 may also be stored in ROM 14. By executing program 19, the CPU 12 functions as a partitioning processing unit 30, an overall processing unit 32, and a synthesis unit 34.

[0039] ROM 14 stores various programs and data. RAM 16 temporarily stores programs or data as a working area. Storage 18 consists of storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive) and stores various programs and data, including the operating system.

[0040] The input unit 20 includes a pointing device such as a mouse, and a keyboard, and is used for various types of input. The display unit 22 is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and displays various types of information. Alternatively, the input unit 20 and the display unit 22 may be integrated into a so-called touch panel display.

[0041] The communication interface 24 is an interface for communicating with devices outside the tracking device 10. For this communication, wireless communication standards such as 4G, 5G, and Wi-Fi (registered trademark) are used.

[0042] Next, the operation of the tracking device 10 in each of the above embodiments will be described. The tracking process shown in Figure 10 corresponds to the processes of both the first and second embodiments described above. Figure 10 shows a flowchart representing an example of the tracking process. The selection process shown in Figure 10 is performed by the CPU 12 executing the program 19 stored in the storage 18.

[0043] In step S100 of Figure 10, the division processing unit 30 and the overall processing unit 32 each acquire the current frame image f and the previous frame image f, as described above. In the specific example described above, frame image f_N and frame image f_N-1 are acquired.

[0044] The next steps, S102 to S116, are division processes performed by the division processing unit 30. In step S102, the division processing unit 30 divides the two acquired frame images f into a plurality of partial images 50, as described above.

[0045] In the next step S104, the segmentation processing unit 30 predicts the position of the ID-labeled detection frame, as described above. In the next step S106, the segmentation processing unit 30 determines the inference image 60 and the downsampled image 52 from among the multiple partial images 50, as described above.

[0046] In the next step S108, loop 1 is started, which repeats the processing in steps S108 to S114 for all of the partial images 50.

[0047] For the selected partial image 50, in step S110, the division processing unit 30 determines whether or not it is an inference image 60. If it is not an inference image 60, in other words, if it is a decimated image 52, the determination in step S110 is negative, and the process proceeds to step S116. On the other hand, if the partial image 50 is an inference image 60, the determination in step S110 is positive, and the process proceeds to step S112. In step S112, the division processing unit 30 detects an object and generates a detection frame for the detected object. In the next step S114, the division processing unit 30 executes an ID assignment process to assign an ID to the generated detection frame.

[0048] In the ID assignment process, as described above, the duplicate determination unit 40 calculates the degree of overlap IoU between all generated detection frames and the predicted position of the predicted detection frame j based on equation (1) above. If the degree of overlap IoU is not "0", as described in the tracking device 10 of the first embodiment, the degree of overlap IoU is added to the weight table, and an ID is assigned to each detection frame by the Hungarian algorithm, treating it as a problem of maximizing the sum of the degree of overlap IoU.

[0049] On the other hand, when the degree of overlap IoU is "0", the threshold D is as described in the tracking device 10 of the second embodiment. ｔｈ Alternatively, threshold D ｔｈ （ｉ） The distance calculation unit 46 then determines the coordinate distance d between the generated detection frame i and the predicted detection frame j. ｉｊ Calculate.

[0050] The allocation unit 48 is the coordinate distance d ｉｊ Threshold D ｔｈ Alternatively, threshold D ｔｈ （ｉ） If it exceeds the predetermined value d, ｍａｘ Add this to the cost table. Meanwhile, coordinate distance d ｉｊ Threshold D ｔｈ Alternatively, threshold D ｔｈ （ｉ） In the following case, the coordinate distance d ｉｊ The value is added to the cost table. The allocation unit 48 treats this as a cost maximization problem and assigns an ID to each detection frame using the Hungarian algorithm.

[0051] Once processing is complete for all partial images 50, in the next step S116, loop 1 ends and the process moves to step S118.

[0052] In step S118, the division processing unit 30 combines the results of all the partial images 50 as described above. This completes the division process by the division processing unit 30.

[0053] On the other hand, steps S120 to S126 are overall processes executed by the overall processing unit 32.

[0054] In step S120, the overall processing unit 32 reduces the size of the two acquired frame images f to generate reduced images 70. In the example above, reduced images 70 are generated for each of the frame image f_N and frame image f_N-1.

[0055] In the next step S122, it is decided whether or not to perform inference on the reduced image 70. That is, it is decided whether or not to downsample the reduced image 70.

[0056] In the next step S124, the overall processing unit 32 predicts the position of the ID-labeled detection frame in frame image f_N based on frame image f_N-1, as described above.

[0057] In the next step, S126, the overall processing unit 32 determines whether or not there is inference, that is, whether or not there is any thinning. If there is no inference, in other words, if thinning is performed, the determination in step S126 is negative. In this case, the overall processing ends and the process moves to step S130. On the other hand, if there is inference, in other words, if thinning is not performed, the determination in step S126 is positive, and the process moves to step S128. In step S128, the overall processing unit 32 executes the ID assignment process, similar to the process in step S114 of the division process described above. Once the process in step S128 is completed, the overall processing ends and the process moves to step S130.

[0058] In step S130, the synthesis unit 34 synthesizes the detection results (tracking results) from the division process and the detection results (tracking results) from the overall process, as described above. The synthesis unit 34 also determines whether the overlapping detection frames are the same object, as described above.

[0059] In the next step S132, the synthesis unit 34 determines the ID to be assigned to the detection frame based on the number of consecutive thinning cycles, as described above.

[0060] In the next step S134, the synthesis unit 34 removes the excess detection frames as described above, and then draws the remaining detection frames on the output image. When the processing in step S134 is completed, the tracking process shown in Figure 10 is completed.

[0061] As described above, the tracking device 10 of each embodiment comprises a division processing unit 30, an overall processing unit 32, and a synthesis unit 34. The division processing unit 30 combines, for each frame image in a group of frame images including a plurality of frame images in a time series, the result of tracking an object in an inference image selected from among a plurality of divided images obtained by dividing the frame image, and the result of tracking an object in a reduced image other than the inference image among the plurality of divided images, based on the tracking result obtained from another frame image either before or after the frame image. The overall processing unit 32 tracks an object in a reduced image obtained by reducing the frame image for each frame image. The synthesis unit 34, for each frame image, combines the tracking result from the division processing unit and the tracking result from the overall processing unit, and outputs the frame image tracking result after removing any overlapping tracking results. Specifically, the synthesis unit 34 removes excessive tracking results by removing excessive detection frames and IDs based on the overlapping area between the detection frame 80 obtained by the division processing unit 30 and the detection frame 80 obtained by the overall processing unit 32, and the number of consecutive reductions.

[0062] According to the tracking device 10 of the above embodiment, assigning IDs to all the numerous small objects contained in each frame image of the frame image group would result in an enormous amount of computation, reducing the frame rate that can be processed in real time. By assigning IDs to only a portion of the image or the entire image in the tracking device 10, the number of objects to be calculated for each frame image can be greatly reduced. Therefore, according to the tracking device 10 of each embodiment, the amount of computation required when tracking an object across multiple frame images can be reduced. As a result, an improvement in the frame rate due to reduced computation is expected. Furthermore, according to the tracking device 10, it is possible to perform tracking processing on high-resolution moving images in real time.

[0063] Furthermore, the tracking device 10 of the second embodiment can also address the issue of ID swapping that occurs when the predicted detection frame position and the actual object position deviate significantly due to the continuous thinning of object detection processing. Therefore, it is expected that object tracking accuracy will be improved by suppressing ID swapping.

[0064] In the above embodiment, as a specific example, the current frame image f_N and the frame image f_N-1 from the previous frame were described. However, it is also possible to use the frame image f to be processed and any of the frame images f before or after it.

[0065] In addition, the selection process that the CPU reads and executes in each of the above embodiments may be executed by various processors other than the CPU. Examples of such processors include dedicated electrical circuits, which are processors with circuit configurations specifically designed to execute particular processes, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices) whose circuit configuration can be changed after manufacturing, and ASICs (Application Specific Integrated Circuits). Furthermore, the selection process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0066] Furthermore, although the above embodiments describe a configuration in which the program 19 is pre-stored (installed) in the storage 18, the invention is not limited to this. The program 19 may be provided in a form stored on a non-transitor storage medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), or USB (Universal Serial Bus) memory. The program 19 may also be provided in a form downloaded from an external device via a network. In addition, although the above embodiments describe a tracking device 10 as a single device, the tracking device 10 may be configured as a multi-device setup. For example, the functions of the tracking device 10 may be provided through a cloud service.

[0067] Furthermore, all documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually incorporated by reference.

[0068] The following additional information is disclosed regarding the embodiments described above.

[0069] (Note 1) A tracking device comprising: a division processing unit that, for each frame image in a group of frame images including multiple consecutive frame images in time series, combines the result of tracking an object in an inference image selected from multiple divided images obtained by dividing the frame image, and the result of tracking an object in a reduced image other than the inference image among the multiple divided images, based on tracking results obtained from other frame images either before or after the frame image; an overall processing unit that, for each frame image, tracks an object in a reduced image obtained by reducing the frame image; and a compositing unit that, for each frame image, removes any redundant tracking results from the tracking result obtained by combining the tracking result of the division processing unit and the tracking result of the overall processing unit, and outputs the result as the tracking result of the frame image.

[0070] (Note 2) The tracking device according to Note 1, wherein the overall processing unit detects objects in a reduced image obtained by reducing the frame image for each frame image when no object detection is performed on the frame image, the tracking result includes a detection frame for an object and identification information attached to the detection frame, and the combining unit includes an overlap determination unit that determines whether the object detection frame, which is the tracking result of the division processing unit, is the same object as the object detection frame, which is the tracking result of the overall processing unit, based on the area of ​​the overlapping region between the object detection frame, which is the tracking result of the division processing unit, and the object detection frame, which is the tracking result of the overall processing unit, and, when it is determined that they are the same object, an identification information determination unit that determines which of the identification information attached to each of the multiple detection frames for the same object should be the tracking result, based on the number of consecutive decimations in the division image including the detection frame for the same object and the number of consecutive decimations in the frame image.

[0071] (Note 3) The tracking device according to Note 1 or Note 2, wherein each of the division processing unit and the overall processing unit includes: a distance calculation unit that calculates the coordinate distance between the position of the generated detection frame and the position of the predicted detection frame when the detection frame for an object generated from the frame image does not overlap with the detection frame predicted from the other frame image; and an assignment unit that assigns identification information to be attached to the detection frame included in the tracking result using the coordinate distance that is less than a threshold among the calculated coordinate distances.

[0072] (Appendix 4) An object tracking method that includes, for each frame image in a group of frame images containing multiple consecutive frame images in time series, a division process that combines the result of tracking an object in an inference image selected from among multiple divided images obtained by dividing the frame image, and the result of tracking an object in a downsampled image other than the inference image among the multiple divided images, based on the tracking result obtained in another frame image either before or after the frame image; for each frame image, an overall process that tracks an object in a reduced image obtained by reducing the frame image; and for each frame image, a process that combines the tracking result of the division process and the tracking result of the overall process, removes any redundant tracking results, and outputs the result as the tracking result for the frame image.

[0073] (Note 5) A program to cause a computer to function as: a splitting processing unit that, for each frame image in a group of frame images including a plurality of consecutive frame images in time series, combines the result of tracking an object in an inference image selected from a plurality of divided images obtained by splitting the frame image, and the result of tracking an object in a reduced image other than the inference image among the plurality of divided images, based on tracking results obtained in other frame images either before or after the frame image; an overall processing unit that, for each frame image, tracks an object in a reduced image obtained by reducing the frame image; and a compositing unit that, for each frame image, removes any redundant tracking results from the tracking result obtained by combining the tracking result of the splitting processing unit and the tracking result of the overall processing unit, and outputs the result as the tracking result of the frame image.

[0074] (Appendix 6) A tracking device comprising at least one processor, wherein the processor performs a splitting process for each frame image in a group of frame images including a plurality of frame images in a time series, combining the result of tracking an object in an inference image selected from a plurality of split images obtained by splitting the frame image with the result of tracking an object in a reduced image other than the inference image from the plurality of split images, based on tracking results obtained from other frame images either before or after the frame image; performs an overall process for each frame image, tracking an object in a reduced image obtained by reducing the frame image; and for each frame image, performs a process to output the tracking result of the frame image as the tracking result, after removing any redundant tracking results from the tracking result obtained by combining the tracking result of the splitting process and the tracking result of the overall process.

[0075] (Appendix 7) A non-temporary storage medium storing a program executable for a computer to perform a tracking process, wherein the tracking process is configured to perform a splitting process for each frame image of a group of frame images including a plurality of frame images in a time series, which combines the result of tracking an object in an inference image selected from a plurality of split images obtained by dividing the frame image with the result of tracking an object in a reduced image other than the inference image from the plurality of split images, based on the tracking result obtained from another frame image either before or after the frame image; perform an overall process for each frame image to track an object in a reduced image obtained by reducing the frame image; and for each frame image, perform a process to remove any redundant tracking results from the tracking result obtained by combining the tracking result of the splitting process and the tracking result of the overall process, and output the result as the tracking result of the frame image.

[0076] 10 Tracking device 19 Program 30 Division processing unit 32 Overall processing unit 34 Synthesis unit 40 Duplicate determination unit 42 Identification information determination unit 46 Distance calculation unit 48 Assignment unit

Claims

1. A tracking device comprising: a division processing unit that, for each frame image in a group of frame images including multiple consecutive frame images in time series, combines the result of tracking an object in an inference image selected from multiple divided images obtained by dividing the frame image, and the result of tracking an object in a downsampled image other than the inference image among the multiple divided images, based on tracking results obtained from other frame images either before or after the frame image; an overall processing unit that, for each frame image, tracks an object in a reduced image obtained by reducing the frame image; and a compositing unit that, for each frame image, removes any redundant tracking results from the tracking result obtained by combining the tracking result of the division processing unit and the tracking result of the overall processing unit, and outputs the result as the tracking result of the frame image.

2. The tracking device according to claim 1, comprising:

1. The overall processing unit detects objects in a reduced image obtained by reducing the frame image for each frame image, provided that no object detection is performed on the frame image for each frame image; 2. The tracking result includes a detection frame for an object and identification information assigned to the detection frame; 3. The combining unit includes: an overlap determination unit that determines whether the object detection frame, which is the tracking result of the division processing unit, is the same object as the object detection frame, which is the tracking result of the overall processing unit, based on the area of ​​the overlapping region between the object detection frame, which is the tracking result of the division processing unit, and the object detection frame, which is the tracking result of the overall processing unit; and 3. If it is determined that the detection results are for the same object, an identification information determination unit that determines which of the identification information assigned to each of the multiple detection frames for the same object should be the tracking result, based on the number of consecutive decimations in the division image including the detection frame for the same object and the number of consecutive decimations in the frame image; 3. The tracking device according to claim 1, wherein each of the division processing unit and the overall processing unit includes: a distance calculation unit that calculates the coordinate distance between the position of the generated detection frame and the position of the predicted detection frame when the detection frame for an object generated from the frame image does not overlap with the detection frame predicted from the other frame image; and an assignment unit that assigns identification information to be attached to the detection frame included in the tracking result using the coordinate distance that is less than a threshold among the calculated coordinate distances.

4. A method for tracking objects, comprising: for each frame image in a group of frame images including multiple consecutive frame images in time series, a division process is performed to combine the result of tracking an object in an inference image selected from multiple divided images obtained by dividing the frame image with the result of tracking an object in a downsampled image other than the inference image among the multiple divided images, based on tracking results obtained from other frame images either before or after the frame image; for each frame image, an overall process is performed to track an object in a reduced image obtained by reducing the frame image; and for each frame image, a process is performed to remove any redundant tracking results from the tracking result obtained by combining the tracking result of the division process and the tracking result of the overall process, and output the result as the tracking result of the frame image.

Images