Siamese network long-term target tracking method based on template matching
Through the long-term target tracking method of twin network based on template matching, NCC template matching search method and dynamic matching template images are used to solve the problem of target disappearance and reappearance in long-term target tracking, and efficient real-time target tracking performance and speed are achieved .
Patent Information
- Application Number
- CN202210630014.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-06
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-06-06
AI Technical Summary
Long-term target tracking In video sequence, due to the problem of target deformation, disappearance and reappearance, the existing short-term target tracking algorithm cannot effectively respond, resulting in a degradation in tracking performance and the inability to achieve real-time target tracking.
The long-term target tracking method of twin network based on template matching is adopted, and the tracking confidence is determined through the SiamFC++ target tracking method, the target position update and re-detection are used to dynamically match the template image to track the target, reducing the re-detection time.
It significantly improves the long-term target tracking performance, realizes real-time target tracking, and tracking speed reaches 45fps, meeting real-time requirements, and improves detection accuracy through accuracy compensation and template update strategies.
Smart Images

Figure CN114998628B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of long-term target tracking, and in particular relates to a twin network long-term target tracking method based on template matching. Background Art
[0002] Target tracking plays an important role in intelligent monitoring, intelligent transportation, autonomous driving, military guidance, and unmanned aerial vehicles. As the performance of short-term target tracking using deep learning continues to improve, people have begun to pay attention to long-term target tracking applications that are closer to actual scenarios in recent years.
[0003] Compared with short-term tracking, long-term tracking tasks require the tracker to have the ability to capture the tracked object in long-term videos and handle the frequent disappearance and reappearance of the target. This poses more challenges from two aspects.
[0004] Since the video sequence length of long-term target tracking is much longer than that of short-term scenes, and the problems of target deformation, disappearance and reappearance are particularly prominent, the direct application of short-term target tracking algorithms cannot cope with these difficulties, and the tracking performance is greatly reduced. Therefore, many long-term target trackers use target loss judgment mechanism and re-detection mechanism on the short-term tracking algorithm to solve the problem of easy target loss.
[0005] The target re-detection mechanism in long-term target trackers often adopts the method of sliding window global detection or expanding the search area. However, due to the large detection area of these methods, these methods are very time-consuming, and even many deep learning long-term target tracking algorithms cannot achieve real-time target tracking. Summary of the invention
[0006] The purpose of the present invention is to provide a twin network long-term target tracking method based on template matching, which can reduce the time consumption of target re-detection in the long-term target tracking process to meet the real-time target tracking needs.
[0007] The present invention adopts the following technical solution: a twin network long-term target tracking method based on template matching, comprising the following steps:
[0008] The SiamFC++ target tracking method is used to determine the first target tracking result image and the tracking confidence score of the current frame;
[0009] Determine whether the target is lost based on the tracking confidence score;
[0010] When the target is lost, the NCC template matching search method is used to update the target position in the current frame;
[0011] Taking the target position as the center, the dynamic matching template image and SiamFC++ target tracking method are used to track the target in the current frame to obtain the second target tracking result image;
[0012] The second target tracking result image is used as the target tracking result image of the current frame.
[0013] Further, updating the target position in the current frame using the NCC template matching search method includes:
[0014] Taking the dynamic matching template image as the matching template, calculating the first similarity between the matching template and all sub-images in the current frame;
[0015] Selecting a similar sub-image corresponding to the first maximum similarity value;
[0016] Determine the target location based on similar sub-images.
[0017] Furthermore, before determining the target position according to the similar sub-images, the method further includes:
[0018] Calculating a second similarity between the similar sub-image and the dynamic matching template image;
[0019] When the second similarity is greater than or equal to the similarity threshold, the position corresponding to the similar sub-image is obtained, and the position is used as the target position.
[0020] Further, when the second similarity value is less than the similarity threshold, the target position of the previous frame is used as the target position of the current frame.
[0021] Furthermore, the second similarity is calculated as follows:
[0022]
[0023] Among them, SSIM(x,y) is the similarity between image x and image y, l(x,y) is the brightness similarity function, c(x,y) is the contrast similarity function, s(x,y) is the structural similarity function, α, β and γ are all constants, μ x is the grayscale average of image x, μ y is the grayscale average of image y, σ x is the variance of image x, σ y is the variance of image y, σ xy is the covariance of image x and image y, c1=(k1L) 2 , c2=(k2L) 2 , k1, k2 and L are all constants.
[0024] Furthermore, after determining the tracking confidence score, the following steps are also included:
[0025] comparing the tracking confidence score to the template update threshold;
[0026] When the tracking confidence score is greater than or equal to the template update threshold, the first target tracking result image corresponding to the tracking confidence score is used as a new dynamic matching template image.
[0027] Furthermore, the template update threshold is calculated according to the tracking confidence score of the second frame of the target tracking video.
[0028] Furthermore, when the tracking confidence score is less than the template update threshold, the dynamic matching template image is not updated.
[0029] Furthermore, judging whether the target is lost according to the tracking confidence score includes:
[0030] Compare the tracking confidence score to the loss threshold;
[0031] When the tracking confidence score ≥ the loss threshold, it is determined that the target is not lost;
[0032] When the tracking confidence score is less than the loss threshold, the target is determined to be lost.
[0033] Another technical solution of the present invention: a twin network long-term target tracking device based on template matching, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned twin network long-term target tracking method based on template matching is implemented.
[0034] The beneficial effects of the present invention are as follows: based on the SiamFC++ target tracking method, the present invention uses the NCC template matching search method to predict coarse positioning, finds the target position in the current frame, and tracks the target with the target position as the center in combination with a dynamic matching template image, which can effectively reduce the re-detection time in the long-term target tracking process, thereby meeting the needs of real-time target tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is a flow chart of a twin network long-term target tracking method based on template matching according to an embodiment of the present invention;
[0036] Figure 2 It is a principle block diagram of a twin network long-term target tracking method based on template matching according to an embodiment of the present invention;
[0037] Figure 3 This is a schematic diagram of updating the dynamic matching template of the “skiing” video in the VOT2019_LT dataset in an embodiment of the present invention;
[0038] Figure 4 This is a flow chart of global search and re-detection based on dynamic template matching in an embodiment of the present invention;
[0039] Figure 5This is a flow chart of long-term target tracking based on SiamFC++ algorithm in an embodiment of the present invention;
[0040] Figure 6 Comparison of visual results of four long-term target tracking methods for the LaSOT dataset “elephant-16” video;
[0041] Figure 7 Comparison of visual results of four long-term target tracking methods for the “warmup” video of the VOT2019_LT dataset;
[0042] Figure 8 Comparison of visual results of four long-term target tracking methods in the TLP dataset "CarChase1" video;
[0043] Fig. 9 It is a schematic diagram of the structure of a twin network long-term target tracking device based on template matching according to an embodiment of the present invention. DETAILED DESCRIPTION
[0044] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0045] In the process of long-term target tracking, in order to re-track the lost target after the target is lost, the re-detection mechanism of many current long-term target tracking algorithms is relatively time-consuming, and the algorithms cannot meet the real-time tracking performance. For example, the MBMD algorithm designs a regression network and matching network framework for long-term target tracking, and can effectively retrieve the target through sliding window detection, but it still has certain defects in real-time performance. The target tracking rate on the VOT2018_LT dataset is only 2.5fps. When the DaSiamRPN_LT and SiamRPN++_LT algorithms determine that the target is lost, they will expand the target search area from 255x255 to 831x831 for target re-detection. Their tracking rates on the VOT2018_LT dataset are 20.8fps and 21.1fps respectively.
[0046] Therefore, the present invention proposes a Siamese Networks Long-Term Tracking Based on Dynamic Template Matching (SiamTM_LT) method, which can not only significantly improve the long-term tracking performance, but also achieve a tracking speed of about 45fps, which can meet the needs of implementing target tracking.
[0047] Specifically, Figure 1As shown, an embodiment of the present invention discloses a twin network long-term target tracking method based on template matching, comprising the following steps: step S110, using the SiamFC++ target tracking method to determine the first target tracking result image and the tracking confidence score of the current frame; step S120, judging whether the target is lost according to the tracking confidence score; step S130, when the target is lost, using the NCC template matching search method to update the target position in the current frame; step S140, with the target position as the center, using the dynamic matching template image and the SiamFC++ target tracking method to track the target in the current frame to obtain a second target tracking result image; step S150, using the second target tracking result image as the target tracking result image of the current frame.
[0048] like Figure 2 As shown, based on the SiamFC++ target tracking method, the present invention uses the NCC template matching search method to predict coarse positioning, finds the target position in the current frame, and tracks the target with the target position as the center in combination with the dynamic matching template image, which can effectively reduce the re-detection time in the long-term target tracking process, thereby meeting the needs of real-time target tracking.
[0049] The key to long-term target tracking methods is whether they can track the target stably and continuously, determine whether the target is lost and retrieve it again. At present, many long-term tracking methods often use target re-detection mechanism methods such as sliding window global detection and expanding search area to solve the problem of target loss. However, since these methods all have the need to extract feature maps first, the process is very time-consuming, resulting in many deep learning long-term target tracking methods being unable to achieve real-time target tracking. The present invention introduces a template matching method for re-detection through unconventional applications. The template matching method does not need to extract the feature map of the image, and only requires simple image elements (such as grayscale, texture and other features) to complete the rough positioning of the target. Although the accuracy is relatively reduced, the problem of reduced accuracy can be compensated by two SiamFC++ target tracking before and after, and even the detection accuracy can be improved, thereby effectively reducing the re-detection time, and proposing a re-detection strategy and a dynamic update strategy for dynamic matching templates, which can more quickly and effectively locate and reproduce the target or correct the lost target, thereby effectively improving the long-term target tracking performance. In addition, since the method of the present invention tracks more accurately and reduces the number of video frames that need to be re-detected, the tracking time can be further reduced, and the tracking frame rate can be increased to 40fps, meeting the real-time target tracking performance.
[0050] In this embodiment, judging whether the target is lost according to the tracking confidence score includes: comparing the tracking confidence score with a loss threshold; when the tracking confidence score ≥ the loss threshold, determining that the target is not lost; when the tracking confidence score is less than the loss threshold, determining that the target is lost.
[0051] Specifically, in an embodiment of the present invention, the NCC template matching search method is used to update the target position in the current frame, including: taking a dynamic matching template image as a matching template, calculating a first similarity between the matching template and all sub-images in the current frame; selecting a similar sub-image corresponding to the maximum value of the first similarity; and determining the target position based on the similar sub-image.
[0052] More specifically, first input the search image x, the initial target template z, the dynamic matching template T = z, and the target initial position pos ini Next, SiamFC++ target tracking is performed using the initial target template z as the tracking model to obtain the target tracking result image x of the current frame. t (pos opt ) and track confidence scores max (z,x). Then, according to the tracking confidence score score max (z,x) determines whether the target is lost. If score max (z,x) is less than the re-detection threshold Th tl , then the target tracking is considered lost and the global search and re-detection mechanism of dynamic template matching needs to be started. max (z,x) is greater than or equal to Th tl , the target is considered to be lost without tracking.
[0053] When the target tracking is considered lost, the global search re-detection mechanism of dynamic template matching is started, and the target initial position pos is updated according to the SSIM similarity judgment result. ini Then take the target initial position pos ini In the local area centered on , SiamFC++ target tracking is performed using the dynamic matching template T as the tracking model to obtain the target tracking result x t_T (pos opt ), and finally update the target tracking result x t (pos opt )=x t_T (pos opt ).
[0054] As a specific implementation method, when determining whether to update the target initial position, first input the image to be searched X, the dynamic matching template T, and the target initial position pos ini , target sub-image X(x ini ,y ini Then, the dynamic matching template T is used to perform normalized cross correlation (NCC) global template matching on the image to be searched X to obtain the rough predicted position pos of the re-detection. tm .
[0055] In one embodiment, before determining the target position according to the similar sub-image, the method further includes: calculating the second similarity between the similar sub-image and the dynamic matching template image; when the second similarity is greater than or equal to the similarity threshold, obtaining the position corresponding to the similar sub-image and using the position as the target position. When the second similarity value is less than the similarity threshold, using the target position of the previous frame as the target position of the current frame.
[0056] The second similarity is calculated as follows:
[0057]
[0058] Among them, SSIM(x,y) is the similarity between image x and image y, l(x,y) is the brightness similarity function, c(x,y) is the contrast similarity function, s(x,y) is the structural similarity function, α, β and γ are all constants, μ x is the grayscale average of image x, μ y is the grayscale average of image y, σ x is the variance of image x, σ y is the variance of image y, σ xy is the covariance of image x and image y, c1=(k1L) 2 , c2=(k2L) 2 , k1, k2 and L are all constants.
[0059] That is, the template matching image is used as the calculation re-detection prediction target image X tm , and then calculate the structural similarity ssim between it and the dynamic matching template T tm =SSIM(X tm ,T). If ssim tm Greater than or equal to the similarity threshold Th tm , retain the rough predicted position pos of the re-detection tm If ssim tm Less than the threshold Th tm , eliminating the retest prediction results.
[0060] If ssim tm Greater than or equal to the similarity threshold Th tm , update the target initial position to the re-detected rough predicted position pos ini =pos tm , ending the global search and re-detection mechanism of dynamic template matching. The SiamFC++ tracker can be used to further obtain the accurate position of the target. The specific flow chart of this method is as follows Figure 5 shown.
[0061] If ssim tm Less than the threshold Th tm, do not update the target initial position pos ini , ending the global search and re-detection mechanism of dynamic template matching.
[0062] In the embodiment of the present invention, in order to realize the effective combination of the SiamFC++ target tracking method and the NCC template matching search method, after determining the tracking confidence score, the following steps are further included:
[0063] Compare the tracking confidence score with the template update threshold; when the tracking confidence score ≥ the template update threshold, use the first target tracking result image corresponding to the tracking confidence score as the new dynamic matching template image. The template update threshold is calculated based on the tracking confidence score of the second frame of the target tracking video. When the tracking confidence score < the template update threshold, the dynamic matching template image is not updated.
[0064] Specifically, the NCC template matching global search algorithm is widely used as an indicator for measuring the similarity between images due to its high matching accuracy and strong robustness. Figure 4 As shown in the figure, for the image X to be retrieved of size M×N and the matching template T of size W×H, the template T is translated on the image to be retrieved X and sequentially matched with the sub-image X intercepted at (xy, c (x,y)={X(x+i,y+j)|i∈[1,…,W],j∈[1,…,H]} uses formula (2) to perform similarity comparison.
[0065]
[0066] Among them, the position corresponding to the maximum value in {NCC(x,y)|x∈[1,…,M],y∈[1,…,N]} is the template matching position pos tm =(x tm_opt ,y tm_opt ), corresponding to the sub-image X tm =X c (x tm_opt ,y tm_opt ) is the dynamic template matching image. In this formula, x and y both represent the coordinate values of the image. This method uses the template T dynamically updated frame by frame to perform global template matching, further improving the anti-interference performance.
[0067] In addition, in order to avoid inaccurate template matching and accumulated errors, this embodiment proposes a template similarity discrimination analysis and positioning optimization strategy to ensure that a relatively accurate prediction initial position is provided for the subsequent target tracking algorithm. SSIM structural similarity is an image evaluation index that is more in line with human vision. It comprehensively measures the similarity of images from three features: brightness, contrast, and structure. For details, please refer to formula (1).
[0068] The closer the template matching result is to the real position of the target, the more accurate the tracking performance of the subsequent target tracking algorithm will be. Therefore, the improved algorithm uses the template matching similarity judgment criterion of formula (3) to classify the matching results.
[0069]
[0070] Dynamic matching template T and output matching image X tm Get SSIM tm =SSIM(X tm ,T) similarity index to measure the template matching accuracy. tm Greater than the set matching similarity threshold Th tm , it indicates that the template matching is more accurate and the matching position is closer to the tracking target position, so the initial position pos of the subsequent target tracking algorithm ini Use template matching position pos tm Update instead; otherwise ssim tm Less than the matching similarity threshold Th tm , the template matching is considered not accurate enough. At this time, the initial position of the subsequent target tracking algorithm is not updated and is still the target tracking position of the previous frame. Through a large number of experiments on multiple long-term standard data sets, Th tm =0.15 is the optimal matching threshold.
[0071] In addition, since the maximum confidence score of the classification branch of the SiamFC++ algorithm reflects the similarity between the predicted target and the template, the confidence peak is generally low when tracking is lost. Therefore, the present invention introduces the confidence peak to judge the target's lost state and proposes the re-detection mechanism startup strategy of formula (4). The confidence peak is used to analyze the target tracking state and determine whether re-detection processing is required.
[0072]
[0073] As shown in formula (4), if the confidence peak score max (z,x) is greater than or equal to the target loss judgment threshold Th tl , the target tracking is considered relatively accurate, and the target re-detection mechanism is not started, that is, the target re-detection flag is set to 0. On the contrary, if the score max (z,x) is less than Th tl , if the target tracking is considered inaccurate or the target is lost, the target re-detection mechanism is started, the target re-detection flag is updated to 1, and the dynamic template matching global search method is used to perform target re-detection.
[0074] Through a large number of experiments on multiple long-term standard datasets, the target loss discrimination threshold Th is obtained. tlThe best choice is 0.3.
[0075] In long-term video sequences, target appearance deformation will seriously affect the accuracy of template matching target re-detection and positioning, resulting in the inability of the improved template matching strategy to play a good role. Therefore, the embodiment of the present invention introduces a dynamic matching template update strategy to further improve the anti-interference performance. The confidence peak is used to analyze the target tracking state to determine whether the current frame tracking result can be used as a matching template.
[0076] If the confidence peak score max (z,x) is greater than or equal to the template update judgment threshold Th td , then the tracking target is considered accurate, and the current frame is used to track the target x(pos opt ) Update and replace the dynamic matching template; otherwise, if the score max (z,x) is less than Th td , if the target tracking is considered inaccurate or the target is lost, no matching template update will be performed.
[0077] This embodiment introduces a dynamic matching template update strategy, and performs a template update as shown in formula (5) after each frame of SiamFC++ target tracking. After a large number of experimental analyses, after the first frame specifies the tracking target, the second frame can usually track the target position more accurately. Therefore, the confidence peak score of the second frame is used. max (z, x2) as the benchmark, and obtain the template update discrimination threshold Th by conducting a large number of experiments on multiple long-term standard datasets td =score max (z,x2)-0.15 is the best choice.
[0078]
[0079] According to formula (5), the embodiment of the present invention determines and updates the dynamic matching template frame by frame, such as Figure 3 The schematic diagram of the dynamic matching template update of the "skiing" video in the VOT2019_LT dataset is given. The first frame image target is used as the initial template. The moving targets in each subsequent video frame change continuously in shape and scale, and have significant appearance differences from the initial target template. After the target is tracked, the algorithm in this paper continuously updates the dynamic matching template. Figure 3 It can be found that the updated dynamic matching template is more similar to the target shape of the current video frame, thus providing a reliable prerequisite for tracking the target in the next frame.
[0080] In order to further prove the effectiveness of the embodiments of the present invention, the following comparative verification experiments were also conducted. This experiment was implemented using Python's PyTorch programming on Intel i7-6700 3.40GHz, GeForce GTX TITAN X 32GB GPU. The method of this embodiment is compared with a variety of advanced target tracking algorithms on three video databases, LaSOT, VOT2019_LT and TLP, respectively. Among them, the LaSOT dataset uses three evaluation indicators: success rate (Success), normalized precision (Normalized Precision) and precision (Precision). The VOT2019_LT dataset uses tracking precision (Precision), recall rate (Recall), F-score (F-score), and tracking speed (Frames PerSecond, FPS) as long-term tracking performance evaluation indicators. The TLP dataset uses two evaluation indicators: success rate (Success) and precision (Precision). The backbone network of the SiamTM_LT method in the embodiment of the present invention uses the AlexNet network to extract feature information.
[0081] (1) Quantitative results and analysis:
[0082] Tables 1, 2 and 3 respectively give the comparison results of the method of the present invention and multiple current advanced long-term target tracking algorithms on the LaSOT, VOT2019_LT and TLP datasets.
[0083] The test set of LaSOT has a total of 280 long-term video sequences, with an average frame number of about 2512 frames. Compared with LTMU, DIMP50, DIMP18, Global Track, SiamCAR and ATOM algorithms, the success rate of the method of the present invention is improved by 0.2%, 0.6%, 4.0%, 5.7%, 5.8% and 5.9% respectively, the normalized precision is improved by 0.4%, 2.1%, 5.6%, 7.2%, 5.9% and 9.3% respectively, and the precision is improved by 1.4%, 2.2%, 5.3%, 5.8%, 6.2% and 8.1% respectively. Compared with the other algorithms in the table, the method of the present invention has obvious advantages in success rate, normalized precision and precision.
[0084] Table 1 Comparison of target tracking performance based on LaSOT dataset
[0085]
[0086] The VOT2019_LT dataset contains 50 long-term video sequences with an average frame number of about 4296 frames. Compared with the SiamDW_LT algorithm, the proposed method has a lower tracking performance, but the tracking speed is 16.16 times that of the SiamDW_LT algorithm. Compared with the SiamRPNs_LT algorithm, the proposed method has a lower precision, but the recall rate is increased by 14.2% and the F value is increased by 7.8%.
[0087] As shown in Table 2, compared with SiamRPN++_LT, mbdet, Siamfcos_LT, and FuCoLoT algorithms, the precision of the method of the present invention is improved by 4.9%, 7.8%, 19.8%, and 18.4%, respectively, the recall rate is improved by 0.5%, 5.2%, 3.6%, and 23.9%, respectively, and the F value is improved by 2.5%, 6.4%, 11.4%, and 22.3%, respectively. Compared with the other algorithms in the table, the precision, recall, and F value of the method of the present invention also have obvious advantages. From the perspective of tracking speed, the tracking speed of the method of the present invention is 2.03 times, 20.2 times, 1.77 times, 26.93 times, and 5.77 times that of SiamRPN++_LT, mbdet, SiamRPNs_LT, Siamfcos_LT, and FuCoLoT algorithms, respectively. It can be seen that the method of the present invention not only significantly improves the tracking performance, but also achieves a tracking frame rate of 40.4fps, which has good real-time performance.
[0088] Table 2 Comparison of target tracking performance based on the VOT2019_LT dataset
[0089]
[0090] As shown in Table 3, the TLP dataset has 50 long-duration video sequences, with an average frame number of about 13529 frames. The tracking performance of the method of the present invention is similar to that of the Global Track algorithm, with a success rate only slightly higher by 0.5% and an accuracy only lower by 0.4%. Compared with other algorithms in the table, the success rate and accuracy of the method of the present invention have obvious advantages.
[0091] Table 3. Comparison of target tracking performance based on TLP dataset
[0092]
[0093]
[0094] In summary, Figure 6 , Figure 7 and Figure 8 The visual qualitative comparison results of various excellent long-term target tracking methods on the LaSOT, VOT2019_LT and TLP datasets are shown respectively.
[0095] Figure 6 In the LaSOT dataset "elephant-16" video, the target is blocked in the 540th and 1505th frames. The proposed method can quickly detect the target position and maintain stable tracking. Although the LTMU method can also detect the target, the tracking target scale is inaccurate and it is easy to track the wrong similar object.
[0096] Figure 7 In the "warmup" video of the VOT2019_LT dataset, the target is completely blocked in the 3795th frame, and reappears in the 3800th frame, but its appearance is quite different from the initial target template. In the 3805th frame, the method of the present invention can quickly and effectively detect the target through the template matching re-detection mechanism, so the SiamDMT_LT method of the present invention can effectively track the target in subsequent frames of the video. The Mbdet method did not detect the target until the 3815th frame, and the other two methods did not accurately track the target.
[0097] Figure 8 The figure shows the video tracking result of the TLP dataset "CarChase1". The targets in the 2530th and 4800th frames are out of view. The method of the present invention immediately starts the template matching re-detection mechanism. After the targets in the 2560th and 4830th frames are reproduced, the SiamDMT_LT algorithm captures the reproduced targets more accurately. Therefore, the tracking effect of subsequent frames of the video is obviously more accurate.
[0098] In summary, these visual tracking results show that the method of the present invention has good tracking performance for attribute video tracking such as “out of field of view” and “target occlusion” in long-term target tracking applications.
[0099] The present invention also discloses a twin network long-term target tracking device based on template matching, such as Fig. 9 As shown, it includes a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, the above-mentioned twin network long-term target tracking method based on template matching is implemented.
[0100] The above-mentioned device can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The device may include but is not limited to a processor and a memory. Those skilled in the art will appreciate that the device may include more or fewer components, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.
[0101] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0102] In some embodiments, the memory may be an internal storage unit of the device, such as a hard disk or memory of the device. In other embodiments, the memory may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the device. Further, the memory may also include both an internal storage unit of the device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.
[0103] It should be noted that the specific content of the above-mentioned device is based on the same concept as the embodiment of the method of the present invention. Its specific functions and technical effects can be found in the method embodiment part, and will not be repeated here.
[0104] In the above embodiments, the description of each embodiment has its own emphasis. For the parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. Those of ordinary skill in the art can appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
Claims
1. Siamese network long-term target tracking method based on template matching, It is characterized in that The following steps are involved: The SiamFC++ target tracking method is used to determine the first target tracking result image and the tracking confidence score of the current frame; Determining whether the target is lost according to the tracking confidence score; When the target is lost, the target position in the current frame is updated using an NCC template matching search method; Taking the target position as the center, using the dynamic matching template image and the SiamFC++ target tracking method to track the target in the current frame, and obtaining a second target tracking result image; Using the second target tracking result image as the target tracking result image of the current frame; Adopting the NCC template matching search method to update the target position in the current frame includes: Taking the dynamic matching template image as a matching template, calculating a first similarity between the matching template and all sub-images in the current frame; Selecting a similar sub-image corresponding to the first maximum similarity value; Determine the target position according to the similar sub-images; Before determining the target position according to the similar sub-image, the following step is further included: Calculating a second similarity between the similar sub-image and the dynamic matching template image; When the second similarity is greater than or equal to a similarity threshold, obtaining a position corresponding to the similar sub-image and taking the position as the target position; Determining the tracking confidence score also includes: comparing the tracking confidence score to a template update threshold; When the tracking confidence score is greater than or equal to the template update threshold, using the first target tracking result image corresponding to the tracking confidence score as a new dynamic matching template image; Judging whether the target is lost according to the tracking confidence score includes: comparing the tracking confidence score to a loss threshold; When the tracking confidence score is ≥ the loss threshold, determining that the target is not lost; When the tracking confidence score is less than the loss threshold, it is determined that the target is lost.
2. The template matching-based twin network long-term target tracking method according to claim 1, It is characterized in that When the second similarity value is less than the similarity threshold, the target position of the previous frame is used as the target position of the current frame.
3. The template matching-based twin network long-term target tracking method according to claim 2, It is characterized in that The second similarity is calculated as follows: Among them, SSIM(x,y) is the similarity between image x and image y, l(x,y) is the brightness similarity function, c(x,y) is the contrast similarity function, s(x,y) is the structural similarity function, α, β and γ are all constants, μ x is the grayscale average of image x, μ y is the grayscale average of image y, σ x is the variance of image x, σ y is the variance of image y, σ xy is the covariance of image x and image y, c 1 =(k 1 L) 2 , c 2 =(k 2 L) 2 , k 1 , k 2 and L are both constants.
4. The template matching-based twin network long-term target tracking method according to claim 3, It is characterized in that The template update threshold is calculated according to the tracking confidence score of the second frame of the target tracking video.
5. The template matching-based twin network long-term target tracking method according to claim 4, It is characterized in that When the tracking confidence score is less than the template update threshold, the dynamic matching template image is not updated.
6. A twin network long-term target tracking device based on template matching, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, it implements the template matching-based twin network long-term target tracking method as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Twin network target tracking method based on dynamic label distribution and mobile equipment
CN113255611A
Visual target tracking method and apparatus based on deep adversarial training
US20200327680A1
Target tracking method and device
WO2018121286A1
Body movement recognition method and device
WO2020199480A1
