Satellite video target tracking method, device and software product in combination with priori knowledge and time sequence memory Transform network

Abstract

The invention discloses a satellite video target tracking method combining priori knowledge and a time sequence memory Transform network. A target tracking task is completed based on a Transform framework, a time sequence memory module, a trajectory prediction module and a shielding judgment module. According to the method, on the basis of a Transform architecture, a template image is extracted, and image features are searched. Through introduced time sequence information and a proposed time sequence memory module, rich discriminative features related to targets are extracted, and the accurate matching capability of the model is improved; besides, by introducing a target size self-adaption-based shielding judgment module and a simple and effective trajectory prediction module, the problem that the target is frequently shielded is effectively solved, and the robustness of the model to a shielding scene is further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to a satellite video target tracking method, device and software product, and specifically relates to a satellite video target tracking method, device and software product that combines prior knowledge and a temporal memory Transformer network. Background Art

[0002] Satellite video single-target tracking technology selects a target of interest in an initial frame and continuously displays its status in subsequent frames. Target tracking is commonly used in a variety of fields, including military strategic decision-making, disaster response, and smart transportation. Because satellite video data has a large image area, making it difficult for targets to escape the imaging range, satellite video target tracking has broad application prospects, especially in the defense and military sectors.

[0003] Compared to traditional cameras, satellite video faces new challenges, primarily in three areas: 1) small targets and blurred features; 2) a large number of similar targets and cluttered backgrounds; and 3) severe occlusion. Existing satellite video target tracking methods aim to address these three issues. Existing work can be divided into two main categories. First, there are traditional kernel filtering methods. This approach extracts target features by online training a filter and then filters the search image. The location with the highest response is the target's location. However, the handcrafted features extracted by traditional kernel filtering methods fail to capture the rich, deep-level information of the target, resulting in limited robust tracking performance. Second, there are deep learning methods. The mainstream work is based on deep learning algorithms such as Siamese networks and Transformers. Current deep learning algorithms use convolutional neural networks to extract features between the template image and the search image, respectively, and then fuse the information between these two features using attention-based methods. However, these algorithms overlook the importance of temporal information in the tracking process. To address frequent occlusion, current work primarily relies on manually setting occlusion judgment criteria and then combining them with Kalman filtering or its extensions to predict the target trajectory. This approach increases model complexity and reduces generalization.

[0004] In general, the existing methods still have two shortcomings: 1) insufficient utilization of temporal information; 2) complex methods for solving occlusion. Summary of the Invention

[0005] The purpose of this invention is to improve the accuracy, reliability and efficiency of satellite video target tracking, provide strong support for the research and application of satellite video target tracking, and provide a satellite video target tracking method that combines prior knowledge and temporal memory Transformer network. The target tracking task is completed by introducing Transformer architecture, temporal memory module, trajectory prediction module and occlusion judgment module.

[0006] According to one aspect of the present invention, a method for satellite video target tracking is provided that combines prior knowledge and a temporal memory Transformer network, comprising:

[0007] Step 1: Use the Transformer feature extraction network to extract target features of the template image and the search image respectively;

[0008] Step 2: Introduce the temporal information in the tracking process and use the temporal memory module to store the target's historical discriminative features, guiding the network to focus on the key areas of the search image, and ultimately obtain the search features guided by temporal information;

[0009] Step 3: Input the search features guided by temporal information into the mask prediction head and the bounding box prediction head respectively to obtain the target prediction mask, target prediction center, target prediction offset and target prediction size. According to the maximum probability index of the prediction center, the prediction offset and prediction size of the corresponding index are obtained to obtain the predicted bounding box of the final target.

[0010] Step 4: Input the predicted bounding box of the final target into the judgment module. Based on the two judgment conditions of the number of target prediction centers and the target displacement range, the network output is judged to be reliable. If so, the network result is used as the output.

[0011] As a further technical solution, if the network output is unreliable, the output result of the motion model is called.

[0012] As a further technical solution, step 2 further includes:

[0013] The template features of the initial frame are encoded into memory units through the memory encoding module, and the memory is initialized to obtain a storage memory rich in target temporal information. The memory encoding module consists of three stacked convolutional layers and a spatial-channel attention block. The convolutional layer consists of a convolution block and a residual connection.

[0014] The stored memory is input into the memory decoding module, which uses temporal information to guide the current search feature to focus on the key area, highlight the target features, and obtain the decoding memory. The memory decoding module consists of three 3D convolution block branches. Two branches use the multi-head attention mechanism to realize multi-level information interaction and highlight the discriminative features of the target. The other branch is used as the residual and connected to the output of the multi-head attention mechanism.

[0015] The decoding memory is used to guide the current search feature to focus on the key area in a local and global collaborative manner, and the search feature guided by temporal information is obtained.

[0016] As a further technical solution, in step 3:

[0017] The mask prediction head is composed of multiple stacked convolution blocks, and the bounding box prediction head is composed of three-branch convolution blocks. The search features guided by timing information are input into the mask prediction head to obtain the target prediction mask. The search features guided by timing information are input into the three-branch bounding box prediction head to obtain the target prediction center, target prediction offset, and target prediction size.

[0018] As a further technical solution, step 4 further includes:

[0019] When there is only one high-value point in the target prediction center and the current predicted target position and the target position of the previous frame are not greater than the historical average width and height of the target, the network output is considered reliable.

[0020] As a further technical solution, step 4 further includes:

[0021] When multiple high-value points appear in the target prediction center, the network output is considered unreliable and the output result of the motion model is called.

[0022] As a further technical solution, the output result of the motion model is called, further comprising:

[0023] The target state is predicted by using the historical target state and the method of simple polynomial fitting, and the target state is predicted by the motion model.

[0024] According to one aspect of the present invention, a device for satellite video target tracking is provided that combines prior knowledge and a temporal memory Transformer network, including:

[0025] The first main module is used to extract target features of the template image and the search image respectively using the Transformer feature extraction network;

[0026] The second main module is used to introduce temporal information in the tracking process. It uses the temporal memory module to store the historical discriminative features of the target, guide the network to focus on the key areas of the search image, and finally obtain the search features guided by temporal information.

[0027] The third main module is used to input the search features guided by temporal information into the mask prediction head and the bounding box prediction head respectively to obtain the target prediction mask, target prediction center, target prediction offset and target prediction size. According to the maximum probability index of the prediction center, the prediction offset and prediction size of the corresponding index are obtained to obtain the predicted bounding box of the final target;

[0028] The fourth main module is used to input the predicted bounding box of the final target into the judgment module. Based on the two judgment conditions of the number of target prediction centers and the target displacement range, it is judged whether the network output is reliable. If so, the network result is used as the output.

[0029] According to one aspect of the present invention, a device for satellite video target tracking that combines prior knowledge and temporal memory Transformer networks is provided, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method for satellite video target tracking that combines prior knowledge and temporal memory Transformer networks.

[0030] According to one aspect of the present invention, a computer software product is provided, comprising computer program instructions, which, when executed, implement the method for satellite video target tracking using a Transformer network combining prior knowledge and temporal memory.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) In order to make full use of the temporal information in the tracking process, the present invention proposes a memory module. By introducing the temporal information encoding as memory, the memory is locally and globally coordinated to interact with the current search features, highlighting the target discriminative features, all of which are aimed at achieving a more reliable match between the template image and the search image, thereby improving the model performance.

[0033] (2) To simply and effectively address occlusion and improve model efficiency, the present invention introduces a simple but effective method for predicting target trajectories and proposes an adaptive judgment module that does not require threshold hyperparameters. The present invention uses polynomials to fit the target's historical trajectory and predict the target's current position. Through the adaptive judgment module, the reliability of the identification network output is identified, both of which improve the model's ability to cope with occlusion and the efficiency of real-time tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 It is a schematic diagram of the principle of the method provided by an embodiment of the present invention.

[0036] Figure 2 It is a structural diagram of a memory module provided by an embodiment of the present invention.

[0037] Figure 3 This is a diagram of the head prediction network structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0039] Object tracking is a crucial area in computer vision, with applications in target surveillance and motion analysis. Traditional and deep learning-based methods have achieved impressive performance in satellite video object tracking. However, existing methods still face challenges such as insufficient utilization of temporal information and complex occlusion resolution. To address these issues, this paper proposes a satellite video object tracking method that combines prior knowledge with a temporal memory Transformer network.

[0040] In order to solve the problem of insufficient utilization of temporal information, the present invention focuses on the temporal information of the target during the tracking process. Since the state of the target in the video sequence will change over time, such as texture, deep semantics and other information, the target features will change accordingly. Therefore, the method of the present invention pays attention to the temporal information of the target and proposes a memory module for storing the historical features of the target and guiding the current search features to focus on key areas. In response to the problem of complex occlusion methods in existing methods, the method of the present invention starts from the characteristics of satellite video imaging and is based on the observation of target trajectories in satellite videos. It is found that the trajectories of most targets are very simple, linear or curved, so an extremely simple but effective method is proposed. The method uses the polynomial function fitting method to predict the target trajectory, and proposes an adaptive judgment module to help the model judge whether the network output is reliable, thereby further improving the performance in the field of satellite video target tracking.

[0041] Please see Figure 1 The satellite video target tracking method provided by this embodiment, which combines prior knowledge and a temporal memory Transformer network, includes the following steps:

[0042] Step 1: Template image based on the first frame The search image at time t is expressed as Search image at time t+1 Among them, 3, H, and W are the number of channels, length, and width of the image respectively. The Transformer feature extraction network is used to extract the template image T and the search image S respectively. t Target characteristics Among them, 16 represents the step size of the Transformer feature extraction network, C represents the number of channels of the feature map, and H T 、W T Respectively represent the length and width of the template image, H S 、W S Represent the length and width of the search image respectively.

[0043] In one embodiment, we first use a ViT feature extraction network with candidate deletion trained on a tracking dataset using a mask strategy based on the DropTrack tracker. Extract template image T and search image S t and S t+1 The general feature F of the target T 、 and In the feature extraction stage, the template image and the search image can perform extensive information communication to effectively highlight the target features in the search area.

[0044] Step 2: Introduce the temporal information in the tracking process, use the temporal memory module to store the target's historical discriminative features, guide the network to focus on the key areas of the search image, and finally obtain the search features guided by temporal information.

[0045] In one embodiment, the template features of the initial frame are first encoded into memory units through a memory encoding module to initialize the memory. The memory encoding module consists of three stacked convolutional layers and a spatial-channel attention block. The convolutional layer consists of a convolution block and a residual connection. T and stencil mask Input to the memory encoding module, first passes through the first convolution layer M l1 , get the first layer memory encoding features

[0046]

[0047] M l1 =Conv_Block(x)+Conv(x)

[0048] Conv_Block(x)=Conv(ReLU(Norm(Conv(x))))

[0049] Among them, concat represents the connection operation in the channel dimension, and the target mask information is used to guide the extraction of target-related features. Conv_Block(·) represents the convolution block, Conv, Norm, and ReLU represent convolution, normalization, and activation functions respectively. Then, the first layer memory encoding features and the template mask are input together into the second convolution layer M l2 , and finally obtain the third layer memory encoding feature containing rich target feature information Next, through the spatial-channel attention block SC_Block, the weights of effective spatial information and channel information are enhanced, the interference of complex background is suppressed, and finally enhanced encoding memory is obtained

[0050]

[0051] SC_Block(x)=sp_attn(ch_attn(x)*x)*x

[0052] ch_attn(x)=σ(MLP(AvgPool(x))+MLP(MaxPool(x)))

[0053] sp_attn(x)=σ(Conv(concat(AvgPool(x),Maxpool(x)))

[0054] Where ch_attn(·) and sp_attn(·) represent spatial attention and channel attention respectively, MaxPool, AvgPool, MLP(·) represent maximum pooling, average pooling and simple multi-layer perceptron operation respectively, σ represents sigmoid function. At the next moment, the input is the search feature and target prediction mask M S ,Since the search features and the template features have different ,spatial scales, when the input is the search features, in order to unify the ,memory size, the first convolution layer M l1 After that, a maximum pooling operation is performed. Considering the limitation of GPU memory in actual application, the maximum number of stored memories is 5. Initially, the number of memories is less than 5, and the initial memory is used to fill the vacancies to obtain a storage memory with rich information about the target time series. Next, the stored memory is input into the memory decoding module, which guides the current search feature to focus on the key area through the temporal information and highlights the target features. First, the stored memory aggregates the temporal information features through the decoding block to effectively represent the discriminative features of the current target. The decoding block consists of three 3D convolution blocks Conv3d_Block branches to obtain the multi-faceted discriminative features of the target.

[0055]

[0056] Conv3d_Block(x)=ReLU(Norm(Flatten(Conv3d(x))))

[0057] Conv3d represents 3D convolution, and Flatten(·) represents flattening the first and second dimensions of the feature. Two branches use the multi-head attention mechanism to achieve multi-level information interaction and highlight the discriminative features of the target. The other branch is connected to the output of the multi-head attention mechanism as the residual, and then passes through the channel-spatial attention block to further adjust the spatial-channel weights and suppress background interference to obtain the decoding memory.

[0058]

[0059] MultiHead represents the multi-head attention mechanism. Then, the decoding memory M rich in temporal information is used de The current search feature is guided to focus on key areas in a local and global collaborative manner. First, the decoding memory and search features are respectively subjected to a two-dimensional convolution to adjust the number of channels, and then split into C / 3 in the channel dimension to obtain the target high-level semantic features and low-level texture and other detail features:

[0060]

[0061] Split represents the feature channel segmentation operation. Then, a global, local, and residual multi-branch structure is used to highlight the current target features and enhance network perception. The global approach focuses on the high-level semantic information of the target, the local approach highlights the target's texture, shape, and other detailed features, and the residual mapping preserves the original features:

[0062]

[0063] in and Represent local enhancement features, global enhancement features and residual mapping features, and their shapes are * indicates the deep cross-correlation operation. Next, the three-branch features are spliced ​​in the channel dimension, and the multi-branch features are fused through a simple two-dimensional convolution, and the weights of different branches are adaptively adjusted to obtain the search features guided by the time series information.

[0064]

[0065] Step 3: Input the search features guided by temporal information into the mask prediction head and the bounding box prediction head respectively to obtain the target prediction mask Target Prediction Center Target prediction offset and target prediction size

[0066] In one embodiment, the structure of the mask prediction head is composed of a stack of multiple convolution blocks Conv-block, and the bounding box prediction head is composed of a stack of three-branch convolution blocks. The search features guided by the timing information are input to the mask prediction head Mask_Pred to obtain the target prediction mask

[0067]

[0068] Mask_Pred(x)=Conv(Conv-block ×3 (x))

[0069] Conv-block(x)=ReLU(Norm(Conv(x)))

[0070] The search features guided by temporal information are input into the three-branch bounding box prediction head to obtain the target prediction center Target prediction offset and target prediction size

[0071]

[0072] Then, according to the index value with the largest prediction center probability, the corresponding prediction offset and prediction size are queried, and finally the target prediction bounding box B∈[0,1] is obtained. 4 ={cx,cy,w,h}, where cx, cy, w, h represent the target center coordinates and target size respectively.

[0073] In step 4, the target prediction bounding box is input into the adaptive judgment module to determine whether the network output is reliable. Specifically, the target prediction bounding box is input into the judgment module, and the target prediction center probability map is calculated to see if there is more than one high-value point. If multiple high-value points are present, the target is considered invisible. Otherwise, the next judgment is carried out. If the difference in the target center position between two adjacent frames is less than the historical target size, the target is considered visible and the network result is used as the output. Otherwise, the target is considered invisible and the motion model result is used as the output.

[0074] In one embodiment, there are two judgment conditions: the number of target prediction centers and the target displacement range. When there are multiple high-value points in the target prediction center, the network output is considered unreliable and the output result of the motion model is called. When there is only one high-value point in the target prediction center and the current predicted target position and the target position of the previous frame are not greater than the target historical average width and height, the network output is considered reliable. Otherwise, the network output is considered unreliable. The judgment process is as follows:

[0075]

[0076] Where φ(·) and ψ(·) represent the high-value point calculation function and the distance calculation function respectively. is the average size of the target history, A represents the target attention map, B t represents the bounding box of the target prediction at time t.

[0077] In step 5, if the judgment module calls the motion model, the target state is predicted based on the simple polynomial fitting method using the historical target state, and the target state predicted by the motion model is finally obtained. M ∈[0,1] 4 .

[0078] In one embodiment, based on the polynomial fitting method, the target's historical state is used to construct a motion model to predict the target state, solving the problem of frequent target occlusion. First, the target's historical coordinates are decoupled into the horizontal coordinate X = [x1, x2, x3, ..., x n-1 ] and the vertical coordinate Y=[y1,y2,y3,…,y n-1 ], respectively, polynomial motion modeling is performed on the horizontal and vertical coordinates:

[0079] MM x =Polyfit(X)

[0080] MM y =Polyfit(Y)

[0081] Polyfit represents the polynomial fitting function, where MM x and MM y Respectively represent the horizontal and vertical coordinate motion modeling. When the number of historical coordinates stored is insufficient, the polynomial fitting motion model is converted to the instantaneous velocity model. The average velocity of the satellite video motion within 5 frames can be approximately considered as the instantaneous velocity of the target at the current moment. The instantaneous velocity model can be expressed as:

[0082]

[0083] x n =v x +x n-1 ,y n =vy +y n-1

[0084] where v x and v y They are respectively expressed as the instantaneous speed of the target at the current moment. The target size is obtained by the average size of historical targets, and finally

[0085] In one embodiment, the head prediction module in step 3 has two branches, and has classification loss and regression loss during training, where the classification loss includes prediction mask loss and target prediction center loss:

[0086] L cls =Focal(M gt ,M S )+Focal(C,C gt )

[0087] Where Focal represents the focus loss, M gt and C gt They are respectively represented as the target real mask and the Gaussian map generated by the target real coordinate box. The regression loss is supervised by the GIoU function:

[0088] L reg =GIoU(B,B gt )

[0089] Among them B gt Represents the target ground truth bounding box. The total loss is weighted by the classification loss and regression loss:

[0090] L total =λ1L cls +λ2L reg

[0091] where λ1 and λ2 are loss weights.

[0092] The beneficial effects of the present invention are verified by comparative experiments below.

[0093] This example is written in Python and implemented using the classic deep learning framework PyTorch. The model is trained on large public datasets including TrackingNet, LaSOT, GOT-10K, and WebUAV3M using four GTX 4090 graphics cards. The training process involves 220 rounds, each using 10,000 images.

[0094] In this experiment, four public satellite video object tracking datasets, namely "SatSOT," "SV248S," "OOTB," and "VISO," were selected to validate the model's effectiveness and verify the accuracy and reliability of our proposed method. The SatSOT dataset contains 105 video sequences, including 26 trains, 9 airplanes, 65 cars, and 5 ships. The video sequences range in length from 120 to 750 frames and include 11 challenging attributes. The SV248S dataset contains 248 video sequences with three categories: airplanes, cars, and ships. The SV248S dataset focuses on small object tracking, so over 90% of the videos contain cars. Its video sequences range in length from 500 to 753 frames and include 10 challenging attributes. To comprehensively evaluate the performance of our proposed method, a total of 17 methods were compared, including both classic traditional methods and highly performing deep learning methods. The OOTB dataset is a new satellite video dataset designed specifically for directional SOT in 2024 and consists of four categories: vehicles, airplanes, trains, and ships. This dataset focuses primarily on oriented bounding box prediction, a relatively simple tracking scheme compared to the other two datasets. In experiments, we convert oriented bounding boxes to horizontal bounding boxes for testing. The VISO dataset consists of 47 satellite video sequences captured by the Jilin-1 satellite platform, with each image having a resolution of 12,000 × 5,000. We adhere to Viso's official SOT protocol, ranging from 25 to 27 official SOT protocols, for testing video sequences totaling 1.12 million frames and 559 objects.

[0095] Experimental setup: All experiments were conducted on the RTX 4090, following the one-shot evaluation protocol. The selected comparison methods all used official public codes and training weight parameters, and the hyperparameters were all set to default.

[0096] Evaluation indicators: Area Under Curve (AUC), also known as Success Score (S.), and Precision Score (P.) with a threshold of 5 are commonly used performance evaluation indicators for target tracking.

[0097] The calculation formulas for these three evaluation indicators are as follows:

[0098]

[0099] Among them, Bbox pre and Bbox gt Represent the predicted box and the real box respectively, N is the total number of frames in a video, K is the total number of videos in a dataset, Δ is the threshold, x pre ,ypre , x gt and y gt Represent the predicted center coordinates and the target's true center coordinates, respectively. The comparative test results are shown in Table 1, where the red ones are the best results. I(.) represents the indicator function, which is 1 if the condition is met and 0 otherwise. IoU ji It represents the intersection-over-union score between the predicted result and the true frame of the i-th frame in the j-th video in the dataset.

[0100] Table 1 Comparative test results

[0101]

[0102]

[0103]

[0104] From the data comparison in Table 1, it can be clearly observed that the method of the present invention demonstrates superior performance compared to other methods. In terms of two key performance indicators, success image score and accuracy score, the present invention achieves excellent performance, far exceeding that of other methods. In terms of speed, the method of the present invention can still achieve performance exceeding 30 frames per second, which can meet the requirements of real-time tracking. This invention aims to improve the performance of satellite video target tracking and provide strong support for the research and application of satellite video target tracking.

[0105] The implementation basis of each embodiment of the present invention is achieved through programmed processing by a device with processor functionality. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, and in addition to the above-mentioned embodiments, an embodiment of the present invention provides a device for satellite video target tracking that combines prior knowledge and temporal memory with a Transformer network. This device is used to implement the satellite video target tracking method described in the above-mentioned method embodiment.

[0106] The device includes: a first main module, which is used to use the Transformer feature extraction network to extract target features of the template image and the search image respectively; a second main module, which is used to introduce temporal information in the tracking process, use the temporal memory module to store the target historical discriminative features, guide the network to focus on the key areas of the search image, and finally obtain the search features guided by the temporal information; a third main module, which is used to input the search features guided by the temporal information into the mask prediction head and the bounding box prediction head respectively, obtain the target prediction mask, target prediction center, target prediction offset and target prediction size, obtain the prediction offset and prediction size of the corresponding index according to the maximum probability index of the prediction center, and obtain the prediction bounding box of the final target; a fourth main module, which is used to input the prediction bounding box of the final target into the judgment module, and judge whether the network output is reliable according to the two judgment conditions of the number of target prediction centers and the target displacement range. If so, the network result is used as the output.

[0107] The embodiment of the present invention provides a Transformer network that combines prior knowledge and temporal memory for use in satellite video target tracking equipment. It addresses the problems of insufficient utilization of temporal information and complex methods for solving occlusion. It adopts several of the aforementioned modules and introduces a Transformer architecture, a temporal memory module, a trajectory prediction module, and an occlusion judgment module to complete the target tracking task, thereby improving the accuracy, reliability, and efficiency of satellite video target tracking.

[0108] It should be noted that the device embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set, and the principles are basically the same as the principles of the above-mentioned device embodiments provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned device embodiments, obtain the corresponding technical means and the technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, they will improve the devices in the above-mentioned device embodiments to obtain corresponding device class embodiments for implementing the methods in other method class embodiments.

[0109] Based on the same inventive concept as the aforementioned embodiment, an embodiment of the present invention also provides a device for satellite video target tracking that combines prior knowledge and temporal memory Transformer network, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method for satellite video target tracking that combines prior knowledge and temporal memory Transformer network.

[0110] Based on the same inventive concept as the aforementioned embodiment, an embodiment of the present invention further provides a computer software product, which includes computer program instructions. When the computer program instructions are executed, they implement the method for satellite video target tracking that combines prior knowledge and temporal memory Transformer network.

[0111] In summary, the present invention discloses a method, device and software product for satellite video target tracking that combines prior knowledge and a time memory transformation network, and completes the target tracking task based on the Transformer architecture, a temporal memory module, a trajectory prediction module and an occlusion judgment module. The present invention extracts template image and search image features based on the Transformer architecture. By introducing temporal information and the proposed temporal memory module, the present invention extracts rich discriminative features of the target and improves the model's ability to accurately match. In addition, the present invention effectively solves the problem of frequent target occlusion by introducing an occlusion judgment module that is adaptive based on target size and a simple but effective trajectory prediction module, further improving the model's robustness to occlusion scenes.

[0112] The terms "including" and "having" and any variations thereof in the description and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. Combining prior knowledge and temporal memory Transformer network for satellite video target tracking method, characterized by: include: Step 1: Use the Transformer feature extraction network to extract target features of the template image and the search image respectively; Step 2: Introduce the temporal information in the tracking process and use the temporal memory module to store the target's historical discriminative features, guiding the network to focus on the key areas of the search image, and ultimately obtain the search features guided by temporal information; Step 3: Input the search features guided by temporal information into the mask prediction head and the bounding box prediction head respectively to obtain the target prediction mask, target prediction center, target prediction offset and target prediction size. According to the maximum probability index of the prediction center, the prediction offset and prediction size of the corresponding index are obtained to obtain the predicted bounding box of the final target. Step 4: Input the predicted bounding box of the final target into the judgment module. Based on the two judgment conditions of the number of target prediction centers and the target displacement range, the network output is judged to be reliable. If so, the network result is used as the output.

2. The method for satellite video target tracking based on prior knowledge and temporal memory Transformer network according to claim 1, characterized in that: If the network output is unreliable, the output of the motion model is called.

3. The method for satellite video target tracking based on prior knowledge and temporal memory Transformer network according to claim 1, characterized in that: The step 2 further comprises: The template features of the initial frame are encoded into memory units through the memory encoding module, and the memory is initialized to obtain a storage memory rich in target temporal information. The memory encoding module consists of three stacked convolutional layers and a spatial-channel attention block. The convolutional layer consists of a convolution block and a residual connection. The stored memory is input into the memory decoding module, which uses temporal information to guide the current search feature to focus on the key area, highlight the target features, and obtain the decoding memory. The memory decoding module consists of three 3D convolution block branches. Two branches use the multi-head attention mechanism to realize multi-level information interaction and highlight the discriminative features of the target. The other branch is used as the residual and connected to the output of the multi-head attention mechanism. The decoding memory is used to guide the current search feature to focus on the key area in a local and global collaborative manner, and the search feature guided by temporal information is obtained.

4. The method for satellite video target tracking based on prior knowledge and temporal memory Transformer network according to claim 1, characterized in that: In step 3: The mask prediction head is composed of multiple stacked convolution blocks, and the bounding box prediction head is composed of three-branch convolution blocks. The search features guided by timing information are input into the mask prediction head to obtain the target prediction mask. The search features guided by timing information are input into the three-branch bounding box prediction head to obtain the target prediction center, target prediction offset, and target prediction size.

5. The method for satellite video target tracking based on prior knowledge and temporal memory Transformer network according to claim 1, characterized in that: The step 4 further comprises: When there is only one high-value point in the target prediction center and the current predicted target position and the target position of the previous frame are not greater than the historical average width and height of the target, the network output is considered reliable.

6. The method for satellite video target tracking based on prior knowledge and temporal memory Transformer network according to claim 5, characterized in that: The step 4 further comprises: When multiple high-value points appear in the target prediction center, the network output is considered unreliable and the output result of the motion model is called.

7. The method for satellite video target tracking using a Transformer network combining prior knowledge and temporal memory according to claim 6, characterized in that: The output results of calling the motion model also include: The target state is predicted by using the historical target state and the method of simple polynomial fitting, and the target state is predicted by the motion model.

8. Combining prior knowledge and temporal memory Transformer network for satellite video target tracking equipment, characterized by: include: The first main module is used to extract target features of the template image and the search image respectively using the Transformer feature extraction network; The second main module is used to introduce temporal information in the tracking process. It uses the temporal memory module to store the historical discriminative features of the target, guide the network to focus on the key areas of the search image, and finally obtain the search features guided by temporal information. The third main module is used to input the search features guided by temporal information into the mask prediction head and the bounding box prediction head respectively to obtain the target prediction mask, target prediction center, target prediction offset and target prediction size. According to the maximum probability index of the prediction center, the prediction offset and prediction size of the corresponding index are obtained to obtain the predicted bounding box of the final target; The fourth main module is used to input the predicted bounding box of the final target into the judgment module. Based on the two judgment conditions of the number of target prediction centers and the target displacement range, it is judged whether the network output is reliable. If so, the network result is used as the output.

9. A Transformer network combining prior knowledge and temporal memory for satellite video target tracking equipment, characterized in that: The invention comprises a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method for satellite video target tracking combining prior knowledge and temporal memory Transformer network as described in any one of claims 1 to 7.

10. A computer software product, characterized in that The computer software product includes computer program instructions, which, when executed, implement the method for satellite video target tracking combining prior knowledge and temporal memory Transformer network as described in any one of claims 1 to 7.

Images