Object tracking method, object tracking system, and computer readable storage medium

Abstract

In step S1, at least one object in a dynamic image is recognized by a learning recognition method at a first frame rate. In step S2, a ROI position is initialized based on the object recognized by the learning recognition method. In step S3, based on the ROI position, the object in the dynamic image is tracked by template matching using a template at a second frame rate higher than the first frame rate. In step S4, the difference between the ROI of the frame at which the object is recognized by the learning recognition method and the ROI of the frame where template matching is continued at the second frame rate to the present is corrected.

Description

Technical Field

[0001] The present invention relates to an object tracking method, an object tracking system and a computer-readable storage medium. Background Art

[0002] Object tracking technology is a technology for identifying and tracking a specific object in a dynamic image captured by a camera. Representative object tracking technologies in recent years include a method of locally tracking an object using template matching (hereinafter referred to as a "local object tracking method"), a method of identifying an object by processing based on a learning theory such as deep learning (hereinafter referred to as a "learning recognition method"), and the like (for example, Non-Patent Document 1).

[0003] Prior Art Literature

[0004] Non-patent literature

[0005] Non-patent document 1: Kang Mingjun, Takagi Ken, and Ishii Bao, "High-speed object tracking using deep learning," Proceedings of the Robotics and Mechatronics Symposium 2018 of the Japan Society of Mechanical Engineers, 2A1-K17 (2018) Summary of the invention

[0006] Problems to be solved by the invention

[0007] The local object tracking method only performs a local search near the object, so it has the advantage of being able to execute at a high speed, but on the other hand, it requires manual specification of the initial template at the beginning of tracking. Therefore, there is a problem that if tracking fails many times due to the complexity of the background image, occlusion of the object, or changes in posture, tracking becomes difficult.

[0008] On the other hand, in recent years, the performance of deep learning has significantly improved in learning recognition methods based on convolutional neural networks (CNNs). Combined with parallel implementation on high-performance GPUs, learning recognition methods have achieved real-time object recognition at a frame rate of tens of fps (frames per second).

[0009] The learning recognition method recognizes objects in an independent manner for each frame, so there is an advantage that even if the object is lost once, it can be recognized again instantly. However, recognition is not necessarily stable in all frames, so there is a problem of intermittent recognition results. In addition, in mechanical feedback such as active vision, high-speed real-time object detection of several hundred fps is required. However, this is difficult with existing computer technology. In this way, there is a problem of limitations on execution speed in the learning recognition method.

[0010] In contrast, in recent years, a hybrid object tracking method that utilizes the advantages of both the local object tracking method and the learning recognition method has been proposed (e.g., Non-Patent Document 1). In the technology described in Non-Patent Document 1, object recognition based on the learning recognition method performed at a certain frame rate and object tracking based on the local object tracking method performed at a higher frame rate operate in parallel.

[0011] In this technology, as long as the tracking is working properly, the ROI (Region Of Interest) position and template are updated through the correlation filter, and the tracking is continued. On the other hand, if the tracking deviates and the object is lost, the ROI position and template are initialized based on the object recognized by the learning recognition method, and new tracking is performed based on this. This allows robust object tracking even for changes in the appearance of the object such as occlusion.

[0012] However, in reality, object recognition based on learning recognition methods requires a certain amount of computing time. Therefore, the above-mentioned computing time will pass from the time when the object recognition process starts based on a certain frame to the time when the process is completed and the object is recognized. The frame that becomes the object of ROI position and template initialization is the frame that continues to perform template matching until now (hereinafter referred to as the "current frame"). Therefore, the frame that starts the object recognition process is delayed relative to the current frame. For example, when the sampling rate of the dynamic image is 500fps, and the frame that starts the object recognition process is the frame 30 frames ago relative to the current frame, the delay is 60ms.

[0013] If the object moves within the frame during this delay, the ROI position shifts between the frame where the object recognition process is started and the current frame. In particular, in the case of dynamic images captured by a high-speed camera or dynamic images in which the object in the frame moves greatly, the shift caused by panning the camera left and right during shooting is large. In this case, it is difficult to perform continuous and stable tracking even with the hybrid object tracking method.

[0014] The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technology that can realize continuous and stable object tracking even for high-speed moving images.

[0015] Solutions to Solve Problems

[0016] In order to solve the above-mentioned problems, an object tracking method of a certain scheme of the present invention includes: a step of identifying at least one object in a dynamic image by a learning recognition method at a first frame rate; a step of tracking the object in the dynamic image by template matching using a template at a second frame rate higher than the first frame rate; a step of correcting the difference between the ROI of the frame where the recognition of the object based on the learning recognition method is started and the ROI of the current frame where the template matching is continued at the second frame rate; and a step of initializing the ROI position of the current frame based on the corrected ROI.

[0017] The step of correcting the difference may include the step of reproducing actual images from a frame at which recognition of the object is started to a current frame.

[0018] In the step of reproducing the actual image, an actual image obtained by extracting a part of the actual image from the frame where the recognition of the object is started to the current frame may be reproduced.

[0019] Alternatively, the ROI position of the current frame is initialized at the first frame rate.

[0020] The embodiment may also include a step of initializing the template at a first frame rate based on the object recognized by the learning recognition method.

[0021] The implementation method may also include a step of comparing the object identified by the learning recognition method with the object tracked by template matching. When the comparison result in the comparison step is below a specified value, the ROI position is initialized based on the object tracked by template matching. When the comparison result in the comparison step is greater than a specified value, the ROI position is initialized based on the corrected ROI.

[0022] The implementation method may also include a step of comparing an object identified by a learning recognition method with an object tracked by template matching. When the comparison result in the comparison step is below a specified value, the template is initialized based on the object tracked by template matching. When the comparison result in the comparison step is greater than a specified value, the template is initialized based on the object identified by the learning recognition method.

[0023] The implementation method may also include the step of storing frames from the frame where the recognition of the object based on the learning recognition method was started to the current frame, and using the ROI position and template of the frame where the recognition of the object based on the learning recognition method was started, performing tracking based on template matching on the stored frames, thereby performing the step of correcting the difference.

[0024] Another object tracking method of the present invention includes: a step of identifying an object in a dynamic image by a learning recognition method at a first frame rate; a step of tracking the object in the dynamic image by template matching using a template at a second frame rate higher than the first frame rate; a step of predicting an image of a current frame that continues to perform template matching at the second frame rate based on an image of a frame where object recognition based on the learning recognition method is started; a step of correcting the difference between the ROI of the frame where object recognition based on the learning recognition method is started and the predicted ROI of the current frame; and a step of initializing the ROI position of the current frame based on the corrected ROI.

[0025] Alternatively, the learning recognition method includes deep learning.

[0026] Alternatively, the dynamic image may be captured by a high-speed camera.

[0027] The second frame rate may be 8 times or more of the first frame rate.

[0028] Another embodiment of the present invention is an object tracking system. The object tracking system includes: an object recognition unit that recognizes an object in a dynamic image by a learning recognition method at a first frame rate; an object tracking unit that tracks at least one object in the dynamic image by template matching using a template at a second frame rate higher than the first frame rate; a correction unit that corrects the difference between the ROI of the frame where the recognition of the object based on the learning recognition method is started and the ROI of the current frame where the template matching is continued at the second frame rate; and an ROI position initialization unit that initializes the ROI position of the current frame based on the corrected ROI.

[0029] Another embodiment of the present invention is an object tracking program. The object tracking program causes a computer to execute the following steps: a step of identifying at least one object in a dynamic image by a learning recognition method at a first frame rate; a step of tracking the object in the dynamic image by template matching using a template at a second frame rate higher than the first frame rate; a step of correcting the difference between the ROI of the frame where the recognition of the object based on the learning recognition method is started and the ROI of the current frame where the template matching is continued at the second frame rate; and a step of initializing the ROI position of the current frame based on the corrected ROI.

[0030] It should be noted that any combination of the above-mentioned constituent elements and any form of expression of the present invention converted into an apparatus, a method, a system, a storage medium, a computer program, etc. may also be effective as an aspect of the present invention.

[0031] Effects of the Invention

[0032] According to the present invention, continuous and stable object tracking can be achieved even for high-speed dynamic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a diagram showing a frame of a moving image according to an embodiment.

[0034] Figure 2 This is a flowchart of the object tracking method according to the first embodiment.

[0035] Figure 3 This is a flowchart of the object tracking method according to the second embodiment.

[0036] Figure 4 This is a flowchart of the object tracking method according to the third embodiment.

[0037] Figure 5 This is a flowchart of the object tracking method according to the fourth embodiment.

[0038] Figure 6 This is a flowchart of the object tracking method according to the fifth embodiment.

[0039] Figure 7 This is a flowchart of the object tracking method according to the sixth embodiment.

[0040] Figure 8 This is a flowchart of the object tracking method according to the seventh embodiment.

[0041] Fig. 9 It is a block diagram of an object tracking system according to an eighth embodiment.

[0042] Fig.10 It is a block diagram of an object tracking system according to a ninth embodiment.

[0043] Fig.11 : is a figure which shows the result of the object tracking experiment performed based on the method of a comparative example. Fig.11 (a) Fig.11 (b) represents the x-coordinate and y-coordinate of the tracked object respectively.

[0044] Fig.12 It is a diagram showing the results of an object tracking experiment conducted based on the sixth embodiment. Fig.12 (a) Fig.12 (b) represents the x-coordinate and y-coordinate of the tracked object respectively.

[0045] Fig.13 It is a diagram showing the results of an object tracking experiment performed based on the ninth embodiment. DETAILED DESCRIPTION

[0046] Before describing the embodiments in detail, an overview will be described.

[0047] Figure 1Frames of a moving image according to an embodiment of the present invention are shown. Frames F0, F1, F2, F3, ..., F N-1 is a frame of the moving image to be tracked. 1b 、f 1a 、f 2b 、f 2a It is a frame for performing object recognition by learning recognition methods such as deep learning. 1b ,fI 1a ,fI 2b ,fI 2a This is the actual image stored in a buffer, etc.

[0048] exist Figure 1 In the example, in the first period t1, the object in the dynamic image is periodically identified by learning the recognition method. Hereinafter, the reciprocal of the first period t1 is referred to as the first frame rate. When the object is identified by learning the recognition method, the ROI position is initialized based on the identified object. Based on the initialized ROI position, in the second period t2, the object in the dynamic image is tracked by template matching using a template. Hereinafter, the reciprocal of the second period t2 is referred to as the second frame rate. In addition, the calculation time required for object recognition based on the learning method is set to t3. It should be noted that the tracking object in the dynamic image is not limited to one, but can also be multiple.

[0049] The first period t1 is longer than the second period t2. Here, t1=N·t2. Preferably, t1 is 8 times or more of t2, that is, N≥8. In other words, the first frame rate is preferably less than one eighth of the second frame rate.

[0050] f 1b represents the frame at which the object recognition starts by learning the recognition method at a certain moment. Let the moment t be -t3. As mentioned above, since the calculation time required for object recognition is t3, the calculation is completed at t=0, and the frame f is 1a Identify the object.

[0051] When from f 1b When time t1 has passed from the moment t = t1-t3, the frame f 2b The next cycle of object recognition begins. Then, at t=t1, the object recognition is completed. The same applies to the following.

[0052] The actual image from the start of object recognition at t = -t3 to the completion of object recognition at t = 0 is stored in a buffer or the like. In this example, the actual image fI at t = -t3 1b The actual image fI at t = 0 1a Stored in buffer B1.

[0053] In the next object recognition cycle, the actual image fI at t = t1-t3 2b The actual image fI at t = t1 2a Stored in buffer B2. Same as below.

[0054] F0, F1, F2, F3, ... respectively represent frames in which the object is tracked at t = 0, t2, 2 t2 ... As described above, the object is recognized by the learning recognition method at t = 0, so the ROI position of the frame F0 at this time is updated. This frame F0 is called the current frame CF (Current Frame).

[0055] At time t=t1=N·t2, according to the object recognition based on the learning recognition method in the next cycle, frame F N-1 The ROI position is initialized to the new position. N-2 If the object is lost due to tracking deviation in any frame, the frame F N-1 Object recognition in F N Tracking starts again in subsequent frames.

[0056] like Figure 1 As shown in the lower part of FIG, the frame f at which object recognition based on the learning recognition method is started 1b The delay time t3 is relative to the current frame CF. If the object in the dynamic image moves significantly during the period t3, then in frame f 1b There is a large difference between the object position (ROI position) in the image and the object position in the current frame CF.

[0057] The embodiment corrects this difference. The embodiment reproduces the frame fI stored in the buffer B1. 1b To 1a The object is tracked using the actual image and the ROI position is corrected. The reproduction of the actual image may be a high-speed reproduction using all the stored frames or a frame-by-frame reproduction of a subset of frames extracted from the stored frames.

[0058] like Figure 1 As shown in the middle section, at t = -t3, it is located in frame fI 1b ROII in the lower left corner 1b Move to frame fI at t=0 1a The I in the lower right corner 1a On the other hand, Figure 1 As shown in the lower part of FIG, the frame f in which the object is recognized at t=0 1a ROI1 is based on frame f at t = -t3 1b Calculated, so located in frame f 1a The ROI1 is based on the actual image I 1aThen, based on the corrected ROI, the ROI position of the current frame CF is updated.

[0059] In this way, the updated ROI position will not lag behind the object position in the current frame CF, but will always follow the current frame CF, thereby achieving smooth and uninterrupted object tracking.

[0060] Below, refer to Figure 2 to Figure 8 Various embodiments of the present invention will be described. In the embodiments and modifications, the same or equivalent components are denoted by the same reference numerals, and duplicate descriptions are omitted as appropriate. Figure 2 to Figure 8 The flowchart shows a process of recognizing a cycle of an object, that is, one cycle of the first cycle, by a learning method at a first frame rate.

[0061] (First Embodiment)

[0062] Figure 2 This is a flowchart of the object tracking method according to the first embodiment.

[0063] At "start", a first cycle of object tracking based on a learning method begins.

[0064] In step S1, the method recognizes at least one object in a dynamic image by a learning recognition method at a first frame rate. The learning recognition method may use, for example, deep learning based on CNN. The recognition of an object is performed in two stages, for example, (1) detection of recognition candidates and (2) determination of a target. Thus, the image, position, size, etc. of the target ROI are recognized.

[0065] In step S2, the method tracks the object in the dynamic image by template matching using a template at a second frame rate higher than the first frame rate. The tracking of the object in the dynamic image can use local area tracking based on a correlation filter. The local area tracking based on the correlation filter can also include (1) ROI image selection used in the correlation filter, (2) correlation calculation in frequency space, (3) tracking ROI area update based on correlation peak detection and other processes.

[0066] In step S3, the method corrects the difference between the ROI of the frame where the recognition of the object based on the learning recognition method was started in step S1 and the ROI of the current frame where the template matching was continued at the second frame rate in step S2. As described above, for example, the correction can be performed by storing the video images from the frame where the recognition of the object was started by the learning recognition method to the current frame in a buffer, and reproducing the actual image in the buffer to track the object. The reproduction of the actual image can be high-speed reproduction using all the frames stored in the buffer, or frame-by-frame reproduction of a subset of the frames extracted from the buffer.

[0067] In step S4, the method initializes the ROI position of the current frame based on the ROI corrected in step S3.

[0068] At "end", the processing in the process based on the learning method of this cycle is completed. Then, the process shifts to the next cycle, and the tracking of the object based on the learning method of the next first cycle begins. Hereinafter, the same process is repeated for each first cycle.

[0069] According to this embodiment, the ROI position does not lag behind the object position in the current frame, but always follows the current frame, thereby achieving smooth and uninterrupted object tracking.

[0070] (Second Embodiment)

[0071] Figure 3 This is a flowchart of the object tracking method according to the second embodiment.

[0072] The second embodiment includes step S41 instead of step S4 of the first embodiment. The other processing is the same as that of the first embodiment.

[0073] In step S41, the method initializes the ROI position of the current frame at the first frame rate based on the ROI corrected in step S3. That is, in this embodiment, when the object is recognized by the learning recognition method at the first frame rate, the ROI position is initialized accordingly each time.

[0074] According to the present embodiment, the recognition result obtained by the learning recognition method can be used to the maximum extent for updating the ROI position.

[0075] (Third Embodiment)

[0076] Figure 4 This is a flowchart of the object tracking method according to the third embodiment.

[0077] The third embodiment further includes step S5 in addition to the processes of the first embodiment. The other processes are the same as those of the first embodiment.

[0078] In step S5, the method initializes the template and ROI position at the first frame rate based on the object recognized by the learning recognition method in step S1. That is, in this embodiment, each time the object is recognized by the learning recognition method at the first frame rate, the template and ROI position used in the object tracking at the second frame rate are initialized each time.

[0079] According to the present embodiment, the recognition result obtained by the learning recognition method can be used to the maximum extent for updating the template and the ROI position.

[0080] (Fourth Embodiment)

[0081] Figure 5 This is a flowchart of the object tracking method according to the fourth embodiment.

[0082] The fourth embodiment further includes step S42 and step S6 in addition to the processes of the first embodiment. The other processes are the same as those of the first embodiment.

[0083] In step S6, the method compares the object identified by the learning recognition method with the object tracked by template matching in the frame where the recognition of the object is started. The comparison may include the position offset of the ROI of the object, the difference in the pattern of the shape of the template, etc. When the comparison result in step S6 is below the specified value, in step S42, the ROI position is initialized based on the object tracked by template matching. On the other hand, when the comparison result in step S6 is greater than the specified value, in step S4, the ROI position of the current frame is initialized based on the corrected ROI. That is, in this embodiment, as long as the tracking works normally, there is no specified difference between the object identified by the learning recognition method and the object tracked by template matching, and the ROI position is initialized based on the object tracked by template matching. Only when the tracking deviates and the object is lost, the ROI position is initialized based on the object identified by the learning recognition method.

[0084] According to this embodiment, the ROI position can be updated so that the tracking trajectory is continuous.

[0085] (Fifth Embodiment)

[0086] Figure 6 This is a flowchart of the object tracking method according to the fifth embodiment.

[0087] The fifth embodiment further includes step S52 and step S6 in addition to the respective processes of the third embodiment, and includes step S51 instead of step S5. The other processes are the same as those of the third embodiment.

[0088] In step S6, the method compares the object identified by the learning recognition method with the object tracked by template matching. The comparison may include the position offset of the ROI of the object, the difference in the pattern of the shape of the template, etc. When the comparison result in step S6 is below the specified value, in step S52, the template and ROI position are initialized based on the object tracked by template matching. On the other hand, when the comparison result in step S6 is greater than the specified value, in step S51, the template and ROI position are initialized based on the object identified by the learning recognition method. That is, in the present embodiment, in the frame where the recognition of the object is started, as long as the tracking works normally, there is no specified difference between the object identified by the learning recognition method and the object tracked by template matching, and the template and ROI position are initialized based on the object tracked by template matching. Only when the tracking ROI deviates from the recognition result, the template and ROI position are initialized based on the object identified by the learning recognition method.

[0089] According to the present embodiment, the recognition result obtained by the learning recognition method can be used for updating the template only when necessary.

[0090] (Sixth Embodiment)

[0091] Figure 7 This is a flowchart of the object tracking method according to the sixth embodiment.

[0092] The sixth embodiment further includes step S7 in addition to the processes of the first embodiment, and includes step S31 instead of step S3. The other processes are the same as those of the first embodiment.

[0093] In step S7 , the method stores frames from the frame at which the recognition of the object based on the learning recognition method was started to the current frame.

[0094] In step S31, the method uses the ROI position of the frame where the object recognition based on the learning recognition method is started and the template to perform tracking based on the template matching on the stored frame, thereby correcting the difference between the ROI of the frame where the object recognition is started and the ROI of the current frame. At this time, the template matching is performed for a time equivalent to one frame at the second frame rate, for example.

[0095] According to the present embodiment, the ROI position can be corrected by executing tracking based on template matching from the frame where the recognition of the object started by the learning recognition method to the current frame.

[0096] (Seventh Embodiment)

[0097] Figure 8 This is a flowchart of the object tracking method according to the seventh embodiment.

[0098] The seventh embodiment further includes step S8 in addition to the processes of the first embodiment, and includes step S32 instead of step S3. The other processes are the same as those of the first embodiment.

[0099] In step S8, the method predicts the image of the current frame based on the frame in which the object recognition was started by the learning recognition method in step S1. For this prediction, a known method such as linear prediction can be used.

[0100] In step S42 , the method corrects the difference between the ROI of the frame where the object recognition was started in step S1 by the learning recognition method and the ROI of the current frame predicted in step S8 .

[0101] According to the present embodiment, the amount of template matching processing can be greatly reduced by predicting the current frame instead of using the current frame of the actual image.

[0102] (Eighth Embodiment)

[0103] Fig. 9 It is a block diagram of an object tracking system according to an eighth embodiment.

[0104] The object tracking system 1 includes an object recognition unit 10 , an object tracking unit 11 , a correction unit 12 , and an initialization unit 13 .

[0105] The object recognition unit 10 recognizes an object in a dynamic image by a learning recognition method at a first frame rate. The object tracking unit 11 tracks the object in the dynamic image by template matching using a template at a second frame rate higher than the first frame rate. The correction unit 12 corrects the difference between the ROI of the frame in which the object recognition unit 10 starts to recognize the object by the learning recognition method and the ROI of the current frame in which the template matching is continued at the second frame rate in the object tracking unit 11. The initialization unit 13 initializes the template and ROI position of the current frame based on the ROI corrected by the correction unit 12.

[0106] Fig.10 This is a block diagram of an object tracking system 2 according to a ninth embodiment.

[0107] The object tracking system of this embodiment includes an object tracking device 3 including an object recognition unit 10, an object tracking unit 11, a correction unit 12, and an initialization unit 13, and other peripheral devices. The configuration and operation of the object tracking device 3 are the same as those of the object tracking system 1 according to the eighth embodiment, and thus description thereof is omitted.

[0108] The object tracking system 2 may include a high-speed camera 14 . The high-speed camera 14 captures moving images at a frame rate of, for example, 500 frames per second or more, and transmits the captured moving image data to the object tracking device 3 .

[0109] The object tracking system 2 may include an operating PC 15. The operating PC 15 is operated by an operator, and transmits setting data and various commands set by the operator to the object tracking device 3.

[0110] The object tracking device 3 may also include a display device 16, a smartphone 17, or a server 18. The tracking object data tracked by the object tracking device 3 is sent to the display device 16 or the smartphone 17, and displayed as a dynamic image in these devices. Alternatively, the tracking object data tracked by the object tracking device 3 is sent to the server 18 and stored as dynamic image data.

[0111] According to this embodiment, the ROI position will not lag behind the object position in the current frame, but will always follow the current frame, so smooth and uninterrupted object tracking can be achieved.

[0112] (Ninth Embodiment)

[0113] There may be multiple objects to be tracked. In particular, these objects may also move at different speeds. Figure 2 The flow of the object tracking method of the ninth embodiment will be described. At "Start", tracking of an object using a learning method in a first cycle is started.

[0114] In step S1, the method recognizes multiple objects in a dynamic image by a learning recognition method at a first frame rate. The multiple objects may be, for example, multiple objects moving in a dynamic image captured by a single camera. In particular, the movement speeds of these objects may be different. Below, as an example, "people" are identified as the first object and "motor vehicles" are identified as the second object. It should be noted that the learning recognition method in this case is also the same as that in the case of tracking a single object. For example, CNN-based deep learning may also be used.

[0115] In step S2, the method tracks people and vehicles in the dynamic image respectively by using template matching using a template at a second frame rate higher than the first frame rate. Tracking can use local area tracking based on a correlation filter. Local area tracking based on a correlation filter may include (1) ROI image selection used in the correlation filter, (2) correlation calculation in frequency space, (3) tracking ROI area update based on correlation peak detection and other processes.

[0116] In step S3, the method corrects the difference between the ROI of the person and the vehicle in the frame where the recognition based on the learning recognition method was started in step S1 and the ROI of the current frame where the template matching was continued at the second frame rate in step S2. As described above, for example, the video images from the frame where the recognition of the person and the vehicle was started by the learning recognition method to the current frame can be stored in a buffer, and the actual image in the buffer can be reproduced and tracked, thereby performing the correction. The reproduction of the actual image can be a high-speed reproduction using all the frames stored in the buffer, or a frame-by-frame reproduction of a subset of the frames extracted from the buffer.

[0117] In step S4, the method initializes the ROI positions of the person and the vehicle in the current frame based on the ROI corrected in step S3.

[0118] At "End", the processing in the process based on the learning method of this cycle is completed. Then, the process transfers to the next cycle, and the tracking of people and vehicles based on the learning method of the next first cycle begins. Hereinafter, the same process is repeated for each first cycle.

[0119] In the previous object tracking technology, when it is desired to track multiple objects with different moving speeds, the difficulty of tracking varies depending on the object. That is, in general, an object moving at a high speed is more difficult to track than an object moving at a low speed. This causes the objects to be mistaken for each other during tracking, and the object tracking fails due to the occlusion caused by the intersection of two images on the screen. According to the present embodiment, the ROI positions of the multiple objects are not delayed from the current frame, but always follow the current frame, so that smooth and uninterrupted tracking can be achieved, and continuous tracking can be performed.

[0120] The above embodiments can be applied to any dynamic images. In particular, the embodiments can also be applied to dynamic images captured by high-speed cameras, which have significantly improved performance in recent years and are widely used in the industry. In the case of high-speed cameras with a high sampling rate for image capture, the above-mentioned problems in object tracking technology become particularly serious. Therefore, the embodiments of the present invention are particularly effective for dynamic images captured by high-speed cameras.

[0121] As long as the second frame rate is higher than the first frame rate, these frame rates can be any value. As an example, the first frame rate can be less than 100 frames per second, and the second frame rate can be more than 500 frames per second. When learning methods such as deep learning exceed 100 frames per second, the disadvantages in computer cost and calculation time will be large. On the other hand, considering the performance and requirements in the specifications of current and future high-speed cameras, it is preferred that tracking based on template matching be performed at more than 500 frames per second.

[0122] (Verification experiment)

[0123] The present inventors conducted an object tracking experiment according to the embodiment in order to confirm the effects of the present invention.

[0124] Fig.11 : is a figure which shows the result of the object tracking experiment performed based on the method of a comparative example.

[0125] Here, as a comparative example, the technology described in non-patent document 1 was used. The experiment was conducted by tracking a dynamic image acquired at a sampling rate of 500fps at the same frame rate of 500fps using a correlation filter, and object recognition based on deep learning was performed at a frame rate of 50fps. As long as tracking is working properly, the ROI position and template are updated by the correlation filter, and tracking continues. In the event that the tracking deviates and the object is lost, the template ROI is updated based on the object recognized by deep learning, and tracking is restarted based on this.

[0126] Fig.11 (a) Fig.11 (b) represents the x-coordinate and y-coordinate of the object being tracked on the screen, respectively. Here, the x-coordinate refers to the horizontal direction of the image, and the y-coordinate refers to the vertical direction of the image (the same below). As shown in the figure, the graph is interrupted at multiple times and tracking fails. In addition, even in places where tracking is successful, the graph does not depict a smooth curve. It is believed that this is because the frame at the time of object recognition based on deep learning is delayed relative to the current frame and accurate tracking cannot be performed.

[0127] Fig.12 It is a diagram showing the results of an object tracking experiment according to the sixth embodiment of the present invention.

[0128] The difference between this embodiment and the comparative example is that this embodiment includes a step of correcting the difference in ROI between the frame at the time of object recognition based on deep learning and the current frame. The other experimental conditions are the same as those of the comparative example.

[0129] Fig.12 (a) Fig.12 (b) represents the x-coordinate and y-coordinate of the object being tracked on the screen. Fig.11 In contrast, the curve graph is uninterrupted within the entire 5 seconds of the measurement time. In addition, the curve graph as a whole draws a generally smooth curve. It is believed that this indicates that accurate object tracking has been achieved, confirming the effectiveness of the present invention.

[0130] Fig.13 FIG. 5 is a diagram showing the results of an object tracking experiment conducted based on the ninth embodiment of the present invention. Fig.13The x-coordinates of the objects, i.e., "person" and "motor vehicle" on the screen are shown. The graph of the tracked motor vehicle is almost uninterrupted during the entire 5 seconds of measurement time. On the other hand, the tracked person was blocked from the camera from 3 seconds to 3.7 seconds due to the motor vehicle passing in front of the camera, and the graph was interrupted during this period. However, the person was immediately recognized after the motor vehicle passed, and the subsequent continuous tracking was successful. In this way, the effectiveness of the present invention was confirmed for multiple objects with different moving speeds, even if there was an occlusion in the middle. However, there is also a stop period when the moving body flips over, changes direction, etc., so the concept of speed includes speed = 0.

[0131] The present invention has been described above based on the embodiments. It should be understood by those skilled in the art that these embodiments are illustrative only, and that the combination of the above-mentioned components and processing steps may have various modifications, and such modifications are also within the scope of the present invention.

[0132] For example, the second frame rate may be consistent with the frame rate when the dynamic image is taken in. Alternatively, the second frame rate may be related to a portion of the frames of the taken dynamic image. That is, the tracking of the object based on template matching may be performed frame by frame or in a subset of frames.

[0133] The high-speed camera may be an invisible light camera for making invisible light such as infrared rays visible, instead of a visible light camera for capturing images that can be recognized by the naked eye.

[0134] The method of the present invention is not limited to fixed cameras such as surveillance cameras, but can also be applied to dynamic images in a wide range of fields, such as images captured by mobile cameras such as vehicle-mounted cameras or drone cameras, and images captured by cameras with large movements such as translation, and has high industrial applicability.

[0135] Industrial Applicability

[0136] The present invention relates to an object tracking method, an object tracking system and an object tracking program.

[0137] Explanation of symbols:

[0138] S1: a step of recognizing an object in a dynamic image by learning a recognition method at a first frame rate;

[0139] S2: a step of tracking an object in a dynamic image by template matching using a template at a second frame rate;

[0140] S3: a step of correcting a difference between the ROI of the frame where the recognition of the object was started and the ROI of the current frame;

[0141] S31: a step of performing tracking based on template matching on the stored frame using the ROI position of the frame where the recognition of the object based on the learning recognition method is started and the template, thereby correcting the difference between the ROI of the frame where the recognition of the object is started and the ROI of the current frame;

[0142] S32: a step of correcting a difference between the ROI of the frame where the recognition of the object based on the learning recognition method is started and the predicted ROI of the current frame;

[0143] S4: a step of initializing the ROI position of the current frame based on the corrected ROI;

[0144] S41: a step of initializing the ROI position of the current frame at a first frame rate based on the corrected ROI;

[0145] S42 ...initializing the ROI position of the current frame based on the object tracked by template matching;

[0146] S5: a step of initializing a template and a ROI position at a first frame rate based on the object identified by the learning recognition method;

[0147] S51: a step of initializing the template and ROI position of the current frame based on the object identified by the learning recognition method;

[0148] S52: a step of initializing the template and ROI position of the current frame based on the object tracked by template matching;

[0149] S6: a step of comparing the difference between the object recognized by the learning recognition method and the object tracked by template matching in the frame where the recognition of the object is started;

[0150] S7: a step of storing frames from the frame at which the recognition of the object based on the learning recognition method is started to the current frame;

[0151] S8: a step of predicting an image of a current frame based on an image of a frame where recognition of an object based on a learning recognition method is started;

[0152] 1…Object Tracking System;

[0153] 10…object recognition unit;

[0154] 11…Object Tracking Department;

[0155] 12…Revision Department;

[0156] 13…Initialization section.

Claims

1. A method for object tracking, comprising: a step of recognizing at least one object in a dynamic image by learning a recognition method at a first frame rate; a step of tracking the object in the dynamic image by template matching using a template at a second frame rate higher than the first frame rate; a step of correcting a difference between the ROI of the frame where the recognition of the object based on the learning recognition method is started and the ROI of the current frame where the template matching is continued at the second frame rate until now; as well as The step of initializing the ROI position of the current frame based on the corrected ROI, The step of correcting the difference includes the step of reproducing actual images from the frame where recognition of the object is started to the current frame.

2. The object tracking method according to claim 1, wherein: In the step of reproducing the actual image, an actual image obtained by extracting a part of the actual image from the frame where recognition of the object is started to the current frame is reproduced.

3. The object tracking method according to claim 2, wherein: The ROI position of the current frame is initialized at the first frame rate.

4. The object tracking method according to claim 3, wherein: The object tracking method further includes the step of initializing the template at the first frame rate based on the object identified by the learning identification method.

5. The object tracking method according to claim 2, wherein: The object tracking method further includes a step of comparing the object identified by the learning recognition method with the object tracked by the template matching in a frame where the recognition of the object is started. When the comparison result in the step of comparing is equal to or less than a predetermined value, the ROI position is initialized based on the object tracked by the template matching, When the comparison result in the step of comparing is greater than a predetermined value, the ROI position is initialized based on the corrected ROI.

6. The object tracking method according to claim 2, wherein: The object tracking method further includes a step of comparing the object identified by the learning recognition method with the object tracked by the template matching in a frame where the recognition of the object is started. When the comparison result in the step of performing the comparison is equal to or less than a predetermined value, the template and the ROI position are initialized based on the object tracked by the template matching, When the comparison result in the step of comparing is greater than a predetermined value, the template and the ROI position are initialized based on the object recognized by the learning recognition method.

7. The object tracking method according to claim 1, wherein: The object tracking method further includes the step of storing frames from the frame where the object recognition based on the learning recognition method is started to the current frame, The step of correcting the difference is performed by performing tracking based on template matching on the stored frame using the ROI position and the template of the frame where the recognition of the object based on the learning recognition method is started.

8. An object tracking method, comprising: The step of recognizing an object in a dynamic image by learning a recognition method at a first frame rate; a step of tracking the object in the dynamic image by template matching using a template at a second frame rate higher than the first frame rate; A step of predicting an image of a current frame to which template matching is continued at the second frame rate based on an image of a frame in which recognition of an object based on the learning recognition method is started; A step of correcting a difference between the ROI of the frame where the recognition of the object based on the learning recognition method is started and the predicted ROI of the current frame; as well as The step of initializing the ROI position of the current frame based on the corrected ROI, The step of correcting the difference includes the step of reproducing actual images from the frame where recognition of the object is started to the current frame.

9. The object tracking method according to claim 8, wherein: The learning recognition method includes deep learning.

10. The object tracking method according to claim 9, wherein: The dynamic image is a dynamic image captured by a high-speed camera.

11. The object tracking method according to claim 10, wherein: The second frame rate is greater than 8 times the first frame rate.

12. An object tracking system comprising: an object recognition unit for recognizing at least one object in a dynamic image by learning a recognition method at a first frame rate; an object tracking unit that tracks the object in the dynamic image by template matching using a template at a second frame rate higher than the first frame rate; a correction unit that corrects a difference between the ROI of the frame where the recognition of the object based on the learning recognition method is started and the ROI of the current frame where the template matching is continued at the second frame rate until now; as well as an ROI position initialization unit, which initializes the ROI position of the current frame based on the corrected ROI, Here, correcting the difference includes reproducing actual images from a frame where recognition of the object is started to the current frame.

13. A computer-readable storage medium storing an object tracking program, wherein the object tracking program causes a computer to execute the following steps: a step of recognizing at least one object in a dynamic image by learning a recognition method at a first frame rate; a step of tracking the object in the dynamic image by template matching using a template at a second frame rate higher than the first frame rate; a step of correcting a difference between the ROI of the frame where the recognition of the object based on the learning recognition method is started and the ROI of the current frame where the template matching is continued at the second frame rate until now; as well as The step of initializing the ROI position of the current frame based on the corrected ROI, The step of correcting the difference includes the step of reproducing actual images from the frame where recognition of the object is started to the current frame.

Images