Satellite video target tracking method and device based on twin network, and software product

By introducing timing information module and trajectory prediction module in satellite video target tracking, simplifying occlusion judgment, solving the problems of insufficient discriminant characteristics and complex occlusion processing in the prior art, improving the accuracy and efficiency of satellite video target tracking, and improving the generalization of the model.

CN118918489AActive Publication Date: 2024-11-08WUHAN UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410908043.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-08
Estimated Expiration
2044-07-08

AI Technical Summary

Technical Problem

The existing satellite video target tracking methods have insufficient discriminant characteristics, complex occlusion processing methods and excessive hyperparameters, resulting in poor generalization of the model.

Method used

A method based on twin network is adopted, combined with the timing information module to extract more discriminant target features, and predict the target trajectory through polynomial fitting, simplifying occlusion judgment, reducing hyperparameters, and improving the robustness and efficiency of the model.

Benefits of technology

It improves the accuracy and efficiency of satellite video target tracking, enhances the generalization and robustness of the model, and can more reliably match template images and search images to effectively deal with occlusion problems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918489A_ABST
    Figure CN118918489A_ABST
Patent Text Reader

Abstract

The invention discloses a satellite video target tracking method and device based on a twin network and a software product. A target tracking task is completed based on the twin network, a time sequence information module, a trajectory prediction module and a shielding judgment module. On the basis of the twin network, template images, search images and time sequence information features are extracted. Through introduction of time sequence information and a time sequence information module, template features with higher discrimination are extracted, and the accurate matching capability of the model is improved; besides, by introducing a target size self-adaption-based occlusion judgment module and a simple and effective trajectory prediction module, the problem that the target is frequently occluded is effectively solved, and the efficiency and generalization of the model are further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and relates to a satellite video target tracking method, device and software product, and in particular to a satellite video target tracking method, device and software product based on a twin network combined with timing information and trajectory prediction. Background Art

[0002] Satellite video single target tracking technology selects a target of interest in the initial frame and continuously gives the state of the target in subsequent frames. Target tracking is usually used in many fields such as military strategic decision-making, disaster emergency response, and smart transportation. [1],[2] Since satellite video data has the characteristics of large image range and it is difficult for targets to "escape" the imaging range, satellite video target tracking has broad application prospects, especially in the field of national defense and military.

[0003] Compared with traditional cameras, satellite video faces new challenges, mainly in three aspects: 1) small targets and blurred features; 2) a large number of similar targets and cluttered backgrounds; 3) serious occlusion problems. Existing satellite video target tracking methods are committed to solving the above three problems. Existing work can be divided into two categories: First, based on traditional kernel filtering methods [3],[4] This method extracts target features by training a filter online and filters the search image. The position with the highest response value is the location of the target. However, the manual features extracted by the traditional kernel filtering method are difficult to express the rich information of the target in depth, and the tracking performance is not robust enough. Secondly, it is a method based on deep learning. The mainstream work is based on the twin network. [5],[8],[9] and Transformer's deep learning algorithm [6] The current deep learning algorithm uses convolutional neural networks to extract features between the template image and the search image, and then uses an attention-based method to fuse the information between the two features. However, this method of directly interacting features rich in redundant information will make it difficult for the target features to be highlighted.

[10] In dealing with frequent occlusion issues, the current work focuses on [7]

[11] If the occlusion judgment conditions are set manually and then combined with Kalman filtering or its extension to predict the target trajectory, this method increases the complexity of the model and reduces the generalization of the model. In general, the existing methods still have three shortcomings: 1) insufficient discriminative features; 2) complex methods to solve occlusion; 3) too many hyperparameters and poor model generalization.

[0004] References:

[0005] [1]J.Shao,B.Du,C.Wu,M.Gong,and T.Liu,‘HRSiam:High-Resolution SiameseNetwork,Towards Space-Borne Satellite Video Tracking’,IEEE Trans.ImageProcess.,vol.30,pp.3056–3068,2021.

[0006] [2]G.Kopsiaftis and K.Karantzalos,‘Vehicle Detection and TrafficDensity Monitoring from Very High Resolution Satellite Video Data’,inProc.IEEE Int.Geosci.Remote Sens.Symp.(IGARSS),Jul.2015,pp.1881–1884.

[0007] [3]J.Shao,B.Du,C.Wu,and L.Zhang,‘Tracking Objects from SatelliteVideos:A Velocity Feature Based Correlation Filter’,IEEE Trans.Geosci.RemoteSens.,vol.57,no.10,pp.7860–7871,Oct.2019.

[0008] [4]Y.Chen,Y.Tang,Z.Yin,T.Han,B.Zou,and H.Feng,‘Single Object Trackingin Satellite Videos:A Correlation Filter-Based Dual-Flow Tracker’,IEEEJ.Sel.Topics Appl.Earth Observ.Remote Sens.,vol.15,pp.6687–6698,2022.

[0009] [5]B.Li,J.Yan,W.Wu,Z.Zhu,and X.Hu,‘High Performance Visual Trackingwith Siamese Region Proposal Network’,in Proc.IEEE / CVFConf.Comput.Vis.Pattern Recognit.(CVPR),Jun.2018,pp.8971–8980.

[0010] [6]P.Lai,M.Zhang,G.Cheng,S.Li,X.Huang,and J.Han,‘Target-AwareTransformer for Satellite Video Object Tracking’,IEEE Trans.Geosci.RemoteSens.,vol.62,pp.1–10,2024.

[0011] [7]Y.Li,C.Bian,and H.Chen,‘Object Tracking in Satellite Videos:Correlation Particle Filter Tracking Method with Motion Estimation by KalmanFilter’,IEEE Trans.Geosci.Remote Sens.,vol.60,pp.1–12,2022.

[0012] [8]J.Yang,Z.Pan,Z.Wang,B.Lei,and Y.Hu,‘SiamMDM:An Adaptive FusionNetwork with Dynamic Template for Real-Time Satellite Video Single ObjectTracking’,IEEE Trans.Geosci.Remote Sens.,vol.61,pp.1–19,2023.

[0013] [9] Y.Wu et al., 'CSLT: Contourlet-Based Siamese Learning Tracker for Dim and Small Targets in Satellite Videos', IEEE Trans.Geosci.Remote Sens., vol.61, pp.1–13, 2023.

[0014]

[10] W.Song et al., 'A Joint Siamese Attention-Aware Network for Vehicle Object Tracking in Satellite Videos', IEEE Trans.Geosci.Remote Sens., vol.60, pp.1–17, 2022.

[0015]

[11] S.Xuan, S.Li, M.Han, Summary of the invention

[0016] The purpose of the present invention is to improve the accuracy, reliability and efficiency of satellite video target tracking, to provide strong support for the research and application of satellite video target tracking, and to provide a satellite video target tracking method, device and software product based on a twin network combined with timing information and trajectory prediction.

[0017] The technical solution adopted by the method of the present invention is: a satellite video target tracking method based on a twin network, comprising the following steps:

[0018] Step 1: Template image based on the first frame Search images The timing information is Where i = 0, 1, 2, ... N represents the temporal information sequence, C, H, W are the dimensions, length and width of the image respectively. Using the feature extraction module of the twin network, the template image S, the search image T and the temporal information T are extracted. i Common features of the target The subscript n represents the output of the last three blocks of ResNet50, and the superscript l represents different input images; l∈T,S,T i ; n = 3, 4, 5;

[0019] Step 2: Use the temporal information module of the twin network to interact with the template features and the temporal information features to obtain prominent deep features of the template.

[0020] Step 3: Fuse the two relatively deep features in the search image features to obtain the search image deep fusion feature Then, yes and Perform deep cross-correlation operations respectively to obtain the shallow feature response map R low and deep feature response map R high ; Finally, R low and R high Fusion is performed to obtain the final response graph R final ;

[0021] Step 4: Based on the head regression module, use R final Get the classification map M cls 、Target center map M cen and bounding box regression map M reg ;

[0022] Step 5: M cls and M cen Perform element-by-element multiplication to obtain the positioning map M loc ; Combined with positioning map M loc The maximum response position, the specific position and size of the target are obtained from the corresponding position of the bounding box regression map;

[0023] Step 6: Positioning Map M loc Input to the occlusion judgment module to determine the final output.

[0024] As a preference, in step 1, first use the ResNet50 feature extraction network modified by the last two blocks Extract template image S, search image T and timing information T i Common features of the target Then, a 1×1 2D convolution kernel conv is used to reduce the channel dimension and obtain three multi-scale feature maps of uniform scale.

[0025] Preferably, in step 2, the time series information module first performs a concat operation on the template feature and the latter two relatively deeper features of the time series information feature, and performs a deep information fusion operation using a 1×1 2D convolution kernel conv to obtain a deep fusion feature. l=T,T i ; Then the template features and timing information features are interacted in, They represent the deep fusion features of the template and the deep fusion features of the temporal information respectively. Unit is the basic unit, and Norm is the normalization operation. After the two basic units are stacked, the residual connection constitutes the temporal information module.

[0026] The basic unit unit is composed of an attention mechanism and a multi-layer perceptron FFN. The basic structure is:

[0027]

[0028] X F1 =X CF +FFN(X CF ),

[0029] FFN(X CF )=(max(0,W 1 X CF +b 1 ))W 2 +b 2 ,

[0030] Among them, Pos is the position encoding, Attn is the attention mechanism, and X CF It is the feature after the interaction of the two deep fusion feature information, FFN (X CF ) is to solve the high-dimensional features after interaction and increase nonlinearity, X F1 is the output of the basic unit unit. 1 , b 1 , W 2 , b 2 are the weights and biases of linear layer 1 and linear layer 2 respectively; the deep fusion features in the template image and time series information are input into the time series information module to obtain the prominent deep features of the template

[0031] Preferably, in step 3,

[0032]

[0033] R final =conv(concat(R low ,R high ));

[0034] Among them, * represents the depth cross-correlation operation, concat represents the connection operation, and conv is a 2D convolution kernel of size 1×1.

[0035] Preferably, in step 4, the head regression module is composed of 4 layers of 3×3 two-dimensional convolution kernels, normalization layers and activation functions, and then output by a layer of 3×3 two-dimensional convolution kernels to obtain classification maps M respectively. cls , target center map M cen , bounding box regression map M reg .

[0036] As a preferred method, in step 5, the classification graph M is transformed into cls Probability transformation, target center map M cen After normalization, the two are multiplied element by element to obtain the positioning map M loc =softmax(M cls )*M cen ; Positioning map M loc Represents the probability of the target appearing at a certain position, and the positioning map M is obtained using the argmax function loc The maximum probability position of the target in (i, j) = argmax(M loc ); Finally, regress the bounding box image M reg The specific position and size of the target are obtained from the corresponding position.

[0037] As a preferred embodiment, in step 6, the positioning map M loc Input into the occlusion judgment module to perform the following operations:

[0038]

[0039] otherwise; visible

[0040] Among them, ψ represents the occlusion judgment function, and its subscripts represent different judgment conditions; ψ num is the number of high-value groups in the positioning graph, ψ area It is a measure of the area of ​​high-value groups in the positioning map; Tgt area is the target area; occlusion and visible respectively indicate that the target is blocked and visible.

[0041] If the judgment result is occlusion, enter the trajectory prediction module:

[0042]

[0043] in, represents the polynomial fitting function, x pre ,y preThe trajectory prediction module predicts the center position of the target in the current frame. k means that the trajectory prediction module is called only when the current frame t is greater than k. The final output result is x pre ,y pre And the target size obtained in step 5, otherwise directly output the result obtained in step 5.

[0044] Preferably, the head regression module in step 4 has three branches and a three-part loss function. The loss function used in the training process is:

[0045]

[0046] Among them, λ 1 ,λ 2 is the weight, is the classification loss, is the center loss, is the regression loss; y i Represents M cls The true label of each feature point in is 1 if there is a target, otherwise it is 0, n represents the total number of feature points, Represents the upper, lower, left, and right margins of each feature point in the bounding box regression map and the corresponding true value bounding box, M reg (i, j,:) represents the bounding box coordinates of each feature point in the bounding box regression map, M cen (i,j) represents the score of each feature point in the target center map. It is a binary function, representing whether the pixel is in the real frame. If it is in the real frame, the value is 1, otherwise it is 0; L IOU It is a function to calculate the intersection-over-union ratio between the predicted box and the real box, and C(i,j) represents the center value score of the pixel; Represents the upper, lower, left and right margins of the predicted box and the center point of the real box respectively.

[0047] The technical solution adopted by the device of the present invention is: a satellite video target tracking device based on a twin network, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the satellite video target tracking method based on the twin network when executing the program.

[0048] The technical solution adopted by the software product of the present invention is: a software product, including a computer program, which implements the satellite video target tracking method based on the twin network when executed by a processor.

[0049] The advantages and positive effects of the present invention include:

[0050] (1) In order to extract more discriminative target features, the present invention introduces time series information and enhances the target template features through the proposed time series information module to highlight the discriminative features, thereby achieving a more reliable match between the template image and the search image, thereby improving the model performance.

[0051] (2) In order to solve the problem of occlusion and improve the efficiency of the model, the present invention introduces a simple but effective way to predict the target trajectory. The present invention uses polynomials to fit the target's historical trajectory and predict the target's current position, both of which improve the model's ability to deal with occlusion and the efficiency of real-time tracking.

[0052] (3) In order to reduce model hyperparameters, the present invention abandons the strategies of feature fusion and result fusion, and proposes an occlusion judgment module based on target size adaptation, both of which are aimed at avoiding manual adjustments on different data sets and improving the generalization and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The technical solution of the present invention is further described below using embodiments and specific implementation methods. In addition, in the process of describing the technical solution, some drawings are also used. For those skilled in the art, other drawings and the intention of the present invention can also be obtained based on these drawings without paying creative work.

[0054] Figure 1 is a schematic diagram of the method principle of an embodiment of the present invention;

[0055] Figure 2 is a structural diagram of a timing information module according to an embodiment of the present invention;

[0056] Figure 3 4 is a structural diagram of a head regression module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0058] Object tracking is an important field in computer vision, and has important applications in monitoring targets, motion analysis, etc. Traditional and deep learning-based methods have achieved good performance in the field of satellite video object tracking. However, existing methods still face the problems of insufficient extraction of discriminative features, complex occlusion solutions, and too many hyperparameters. In order to solve these problems, the present invention proposes a simple but effective twin network method.

[0059] In order to solve the problem of insufficient discriminative features, the present invention focuses on the discriminative features of prominent targets. Since the state of the target in the video sequence will change over time, such as texture, deep semantics and other information, the target features will change accordingly. Therefore, the method of the present invention introduces the temporal information of the target and proposes a temporal information module based on the attention mechanism to enhance the discriminative features of the target template. In response to the problem that the existing methods deal with complex occlusion methods, the method of the present invention starts from the characteristics of satellite video imaging and is based on the observation of the target trajectory in the satellite video. It is found that the trajectory of most targets is very simple, linear or curved, so an extremely simple but effective method is proposed to predict the target trajectory using a polynomial function fitting method. In order to alleviate the problem of many hyperparameters and poor generalization of the current model, the method of the present invention abandons the strategies of multi-feature fusion and decision fusion, and proposes a method for adaptively judging occlusion based on target size, without the introduction of hyperparameters, thereby further improving the performance in the field of satellite video target tracking.

[0060] Please see Figure 1 , a satellite video target tracking method based on a twin network provided in this embodiment includes the following steps:

[0061] Step 1: Template image based on the first frame Search images The timing information is Where i = 0, 1, 2, ... N represents the temporal information sequence, C, H, W are the dimensions, length and width of the image respectively. Using the feature extraction module of the twin network, the template image S, the search image T and the temporal information T are extracted. i Common features of the target The subscript n represents the output of the last three blocks of ResNet50, and the superscript l represents different input images; l∈T,S,T i ; n = 3, 4, 5;

[0062] In one embodiment, a ResNet50 feature extraction network with the last two blocks modified is used to extract common features of targets in the template image, search image, and temporal information:

[0063]

[0064] in, is the feature extraction network, is the extracted feature, the subscripts represent the output of the last three layers of ResNet50 blocks, and the superscripts represent different input images. In order to unify the three multi-scale feature maps of different inputs and reduce the amount of calculation, a 1×1 2D convolution kernel is used to reduce the number of channels to 256, and we get

[0065] Step 2: Use the temporal information module of the twin network to interact with the template features and the temporal information features to obtain prominent deep features of the template.

[0066] In one embodiment, in order to aggregate deep information and reduce the amount of calculation, the latter two relatively deeper features of the template feature and the temporal information feature are connected, and the deep information is fused using a 1×1 2D convolution kernel to obtain a deep fusion feature. Considering that the shallow texture and color features of satellite video targets are fuzzy and difficult to enhance, only deep semantic features are used to enhance the features. In order to enhance the discriminative deep features of the target, the present invention proposes a temporal information module based on the attention mechanism. The temporal information module is represented as (see Figure 2 ):

[0067]

[0068] in, They represent the deep fusion features of templates and the deep fusion features of temporal information respectively, unit is the basic unit, and Norm is the normalization operation.

[0069]

[0070] X F1 =X CF +FFN(X CF ),

[0071] FFN(x)=(max(0,W 1 x+b 1 ))W 2 +b 2 ,

[0072] Among them, Pos is the position encoding, Attn is the attention mechanism, and X CF It is the feature after the interaction of the two deep fusion feature information, FFN (X CF ) is to solve the high-dimensional features after interaction and increase nonlinearity, X F1 is the output of the basic unit unit. 1 , b 1 , W 2 , b 2 are the weights and biases of linear layer 1 and linear layer 2 respectively. The deep fusion features in the template image and the time series information are input into the time series information module to obtain the template deep features that highlight the discriminability

[0073] Step 3: Fuse the two relatively deep features in the search image features to obtain the search image deep fusion feature Then, yes and Perform deep cross-correlation operations respectively to obtain the shallow feature response map R low and deep feature response map R high ; Finally, the shallow feature response map R low and deep response map R high Fusion is performed to obtain the final response graph R final ;

[0074] In one embodiment, similar to the template image features and the temporal information features, two relatively deep features in the search image features are fused to obtain the search image deep fusion feature Then, a deep cross-correlation operation is performed on the template image feature map and the search image feature map to obtain the shallow feature response map R low and deep feature response map R high :

[0075]

[0076] Where * represents the deep cross-correlation operation. In order to further obtain a response map rich in target texture, semantics, and other information that is conducive to identifying target positioning, the shallow and deep response maps are fused to obtain the final response map R final :

[0077] R final =conv(concat(R low ,R high )).

[0078] Step 4: Based on the head regression module, use the response map R obtained in step 3 final Get the classification map M cls 、Target center map M cen and bounding box regression map M reg ;

[0079] In one embodiment, the obtained response graph R containing rich target information final Input to the head regression module (see Figure 3 ), the head regression module is composed of 4 layers of 3×3 2D convolution kernel, normalization layer and activation function stacking, and then a 3×3 2D convolution kernel is used for output to obtain the classification map M respectively. cls , target center map M cen , bounding box regression map M reg , which is expressed as follows:

[0080] Convblock=Max(0,Norm(conv 3×3 (x))),

[0081]

[0082] Step 5: The classification map M obtained in step 4 cls and target center map M cen Perform element-by-element multiplication to obtain the positioning map M loc ; Combined with positioning map M loc The maximum response position, the specific position and size of the target are obtained from the corresponding position of the bounding box regression map;

[0083] In one embodiment, the classification map is transformed into a probability map using a softmax function, and the target center map is normalized. The two are multiplied element by element to further refine the classification map to obtain the positioning map M. loc :

[0084] M loc =softmax(M cls )*M cen

[0085] The positioning map now represents the probability of the target appearing at a certain position. The argmax function is used to obtain the position with the maximum probability of the target appearing in the positioning map:

[0086] (i,j)=argmax(M loc )

[0087] Finally, the specific position and size of the target are obtained from the corresponding position of the bounding box regression map.

[0088] Step 6: Positioning Map M loc Input to the occlusion judgment module to determine the final output.

[0089] In one embodiment, the positioning map M loc Input into the occlusion judgment module:

[0090]

[0091] otherwise; visible

[0092] Where ψ represents the occlusion judgment function, and its subscripts represent different judgment conditions. num is the number of high-value groups in the positioning graph, ψ area It is a measure of the area of ​​high value groups in the positioning map. area is the target area. Occlusion and visibility respectively indicate that the target is blocked and visible.

[0093] If the judgment result is occlusion, enter the trajectory prediction module:

[0094]

[0095] in represents the polynomial fitting function, x pre ,y pre The trajectory prediction module predicts the center position of the target in the current frame. k means that the trajectory prediction module is called only when the current frame t is greater than k. The final output result is x pre ,y pre And the target size obtained in step 5, otherwise directly output the result obtained in step 5.

[0096] In one embodiment, the head regression module has three branches and a three-part loss function. The loss function used in the training process is:

[0097]

[0098] Among them, λ 1 ,λ 2 is the weight, is the classification loss, is the center loss, is the regression loss; y i Represents M cls The true label of each feature point in is 1 if there is a target, otherwise it is 0; n represents the total number of feature points, Represents the upper, lower, left, and right margins of each feature point in the bounding box regression map and the corresponding true value bounding box, M reg (i, j,:) represents the bounding box coordinates of each feature point in the bounding box regression map, M cen (i,j) represents the score of each feature point in the target center map. It is a binary function, representing whether the pixel is in the real frame. If it is in the real frame, the value is 1, otherwise it is 0; L IOU It is a function to calculate the intersection-over-union ratio between the predicted box and the real box, and C(i,j) represents the center value score of the pixel; Represents the upper, lower, left and right margins of the predicted box and the center point of the real box respectively.

[0099] This embodiment also provides a satellite video target tracking device based on a twin network, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the satellite video target tracking method based on the twin network is implemented.

[0100] This embodiment also provides a software product, including a computer program, which, when executed by a processor, implements the satellite video target tracking method based on a twin network.

[0101] The beneficial effects of the present invention are verified by comparative experiments as follows.

[0102] This experiment is written in Python and implemented using the classic deep learning framework PYTORCH. Two GTX 3090 graphics cards are used to train the model on large public datasets ImageNet VID, LaSOT, and GOT-10K. The training process consists of 20 rounds, with 600,000 images used in each round.

[0103] In this experiment, two public satellite video target tracking datasets named "SatSOT" and "SV248S" were selected as datasets to verify the effectiveness of the model, in order to verify the accuracy and reliability of the present invention. The SatSOT dataset contains 105 video sequences, including 26 trains, 9 airplanes, 65 cars, and 5 ships. The length of the video sequence ranges from 120 to 750 frames and contains 11 challenging attributes. The SV248S dataset contains 248 video sequences, which include three categories: airplanes, cars, and ships. The SV248S dataset mainly focuses on small object target tracking, so more than 90% of the targets in the videos are cars. Its video sequence length ranges from 500 to 753 frames and contains 10 attribute challenges. In order to comprehensively evaluate the performance of the method of the present invention, a total of 17 methods are used here, which are classic traditional methods and deep learning methods with excellent performance for comparison.

[0104] Experimental setup: All experiments were tested on RTX 3090, following the one-time evaluation protocol. The selected comparison methods used official public codes and training weight parameters, and the hyperparameters were all default settings.

[0105] Evaluation indicators: Area Under Curve (AUC), also known as Success score (S.), Precision score (P.) with a threshold of 20, and Enhanced Intersection over Union score (EIoU). The first two indicators are commonly used performance evaluation indicators for target tracking, and the latter indicator weighs the first two indicators to obtain a more comprehensive evaluation.

[0106] The calculation formulas for these three evaluation indicators are as follows:

[0107]

[0108] EIoU=αO+βD+(1-α-β)OD,

[0109]

[0110] Among them, Bbox preand Bbox gt Represent the predicted box and the real box respectively, N is the total number of frames in a video, K is the total number of videos in a dataset, Δ is the threshold, x pre ,y pre , x gt and gt Represent the predicted center coordinates and the target real center coordinates, respectively, and O represents IoU. For ease of use, the α and β weight parameters of EIoU are generally set to 0. See Table 1 for the comparative test results.

[0111] Table 1 Comparative test results

[0112]

[0113] From the data comparison in Table 1, it can be clearly observed that the method of the present invention shows excellent performance compared with other methods. The present invention has achieved excellent performance in the three key performance indicators of success graph score, accuracy score, and enhanced intersection-over-union score, which is much higher than other methods. In terms of speed, the method of the present invention can still achieve a performance exceeding 30 frames per second, which can meet the needs of real-time tracking. The present invention aims to improve the performance of satellite video target tracking and provide strong support for the research and application of satellite video target tracking.

[0114] The present invention innovatively proposes a new method, a twin network target tracking method combining timing information and trajectory prediction, which completes the target tracking task based on the twin network, timing information module, trajectory prediction module and occlusion judgment module. The present invention is based on the twin network to extract template image, search image and timing information features. Through the introduction of timing information and timing information module, the present invention extracts more discriminative template features and improves the model's ability to accurately match; in addition, the present invention effectively solves the problem of frequent occlusion of the target by introducing an occlusion judgment module based on target size adaptation and a simple but effective trajectory prediction module, further improving the efficiency and generalization of the model. The present invention utilizes the unique timing information in the video and combines it with a simple trajectory prediction method to alleviate the shortcomings of existing methods, thereby further improving the performance of satellite video target tracking.

[0115] It should be understood that the embodiments described above are part of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0116] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A satellite video target tracking method based on twin network, characterized in that: The following steps are involved: Step 1: Template image based on the first frame Search images The timing information is Where i = 0, 1, 2, ... N, represents the sequence of temporal information; C, H, W are the dimensions, length and width of the image respectively. The feature extraction module of the twin network is used to extract the template image S, the search image T and the temporal information T i Common features of the target The subscript n represents the output of the last three blocks of ResNet50, and the superscript l represents different input images; l∈T,S,T i ; n = 3, 4, 5; Step 2: Use the temporal information module of the twin network to interact with the template features and the temporal information features to obtain prominent deep features of the template. Step 3: Fuse the two relatively deep features in the search image features to obtain the search image deep fusion feature Then, yes and Perform deep cross-correlation operations respectively to obtain the shallow feature response map R low and deep feature response map R high ; Finally, R low and R high Fusion is performed to obtain the final response graph R final ; Step 4: Based on the head regression module, use R final Get the classification map M cls 、Target center map M cen and bounding box regression map M reg ; Step 5: M cls and M cen Perform element-by-element multiplication to obtain the positioning map M loc ; Combined with positioning map M loc The maximum response position, the specific position and size of the target are obtained from the corresponding position of the bounding box regression map; Step 6: Positioning Map M loc Input to the occlusion judgment module to determine the final output.

2. The satellite video target tracking method based on twin network according to claim 1 is characterized in that: In step 1, first use the modified ResNet50 feature extraction network of the last two blocks Extract template image S, search image T and timing information T i Common features of the target Then, a 1×1 2D convolution kernel conv is used to reduce the channel dimension and obtain three multi-scale feature maps of uniform scale.

3. The satellite video target tracking method based on twin network according to claim 1, characterized in that: In step 2, the temporal information module first concatenates the template features and the last two relatively deeper features of the temporal information features, and uses a 1×1 2D convolution kernel conv to perform a deep information fusion operation to obtain a deep fusion feature. Then, information interaction is performed between the template features and the timing information features. in, They represent the deep fusion features of the template and the deep fusion features of the temporal information respectively. Unit is the basic unit, and Norm is the normalization operation. After the two basic units are stacked, the residual connection constitutes the temporal information module. The basic unit unit is composed of an attention mechanism and a multi-layer perceptron FFN. The basic structure is: X F1 =X CF +FFN(X CF ), FFN(X CF )=(max(0,W1X CF +b1))W2+b2, Among them, Pos is the position encoding, Attn is the attention mechanism, and X CF It is the feature after the interaction of the two deep fusion feature information, FFN (X CF) It is to solve the high-dimensionality and increase the nonlinearity of the features after interaction. F1 is the output of the basic unit unit; W1, b1, W2, b2 are the weights and biases of linear layer 1 and linear layer 2 respectively; the deep fusion features in the template image and time series information are input into the time series information module to obtain the prominent template deep features 4. The satellite video target tracking method based on twin network according to claim 1 is characterized in that: In step 3, R final =conv(concat(R low ,R high )); Among them, * represents the depth cross-correlation operation, concat represents the connection operation, and conv is a 2D convolution kernel of size 1×1.

5. The satellite video target tracking method based on twin network according to claim 1, characterized in that: In step 4, the head regression module is composed of 4 layers of 3×3 two-dimensional convolution kernels, normalization layers and activation functions, and then output by a layer of 3×3 two-dimensional convolution kernels to obtain classification maps M and M respectively. cls , target center map M cen , bounding box regression map M reg .

6. The satellite video target tracking method based on twin network according to claim 1 is characterized in that: In step 5, the classification graph M is transformed into cls Probability transformation, target center map M cen After normalization, the two are multiplied element by element to obtain the positioning map M loc =softmax(M cls )*M cen ; Positioning map M loc Represents the probability of the target appearing at a certain position, and the positioning map M is obtained using the argmax function loc The maximum probability position of the target in (i, j) = argmax(M loc ); Finally, regress the bounding box image M reg The specific position and size of the target are obtained from the corresponding position.

7. The satellite video target tracking method based on twin network according to claim 1 is characterized in that: In step 6, the positioning map M loc Input into the occlusion judgment module to perform the following operations: Among them, ψ represents the occlusion judgment function, and its subscripts represent different judgment conditions; ψ num is the number of high-value groups in the positioning graph, ψ area It is a measure of the area of ​​high-value groups in the positioning map; Tgt area is the target area; occlusion and visibility respectively indicate that the object is occluded and visible; If the judgment result is occlusion, enter the trajectory prediction module: in, represents the polynomial fitting function, x pre ,y pre The trajectory prediction module predicts the center position of the target in the current frame. k means that the trajectory prediction module is called only when the current frame t is greater than k. The final output result is x pre ,y pre And the target size obtained in step 5, otherwise directly output the result obtained in step 5.

8. The satellite video target tracking method based on twin network according to any one of claims 1 to 7, characterized in that: The head regression module described in step 4 has three branches, and the loss function used in the training process is: Among them, λ1, λ2 are weights, is the classification loss, is the center loss, is the regression loss; y i Represents M cls The true label of each feature point in is 1 if there is a target, otherwise it is 0; n represents the total number of feature points, Represents the upper, lower, left, and right margins of each feature point in the bounding box regression map and the corresponding true value bounding box, M reg (i, j,:) represents the bounding box coordinates of each feature point in the bounding box regression map, M cen (i, j) represents the score of each feature point in the target center graph; It is a binary function, representing whether the pixel is in the real frame. If it is in the real frame, the value is 1, otherwise it is 0; L IOU It is a function to calculate the intersection-over-union ratio between the predicted box and the real box, and C(i,j) represents the center value score of the pixel; Represents the upper, lower, left and right margins of the predicted box and the center point of the real box respectively.

9. A satellite video target tracking device based on a twin network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the satellite video target tracking method based on the twin network as described in any one of claims 1 to 8.

10. A software product comprising a computer program, characterized in that When the computer program is executed by a processor, the satellite video target tracking method based on a twin network as described in any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Satellite video target tracking method based on full convolution twin network and trajectory prediction

    CN110580713A

  • Twin network tracking method based on self-adaptive template updating

    CN112132856A

  • Twin network target tracking method considering sequential relationship

    CN114596338A

  • Remote sensing satellite video target tracking method based on twin neural network

    CN117274636A

  • Single target tracking method based on attention mechanism

    CN117576149A