Target tracking method and device, electronic equipment and storage medium

Abstract

The application provides a target tracking method and device, electronic equipment and storage medium, and relates to the technical field of positioning and tracking. The method comprises: obtaining related information for tracking a target object, the related information comprising a satellite image, a text description of the target object included in the satellite image, a target satellite image of the target object intercepted from the satellite image, and a historical trajectory sequence of the target object obtained based on a plurality of historical satellite images before the satellite image; and inputting the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object. Thus, the joint positioning and sequence tracking model based on deep learning combines the text description and the historical trajectory sequence of the target object on the basis of the satellite image to jointly track the target, can realize target tracking in a satellite video scene, and improves the accuracy of target tracking in the satellite video scene.

Description

Technical Field

[0001] This application relates to the field of positioning and tracking technology, and in particular to a target tracking method, apparatus, electronic device and storage medium. Background Technology

[0002] Compared to target tracking in general video scenarios, target objects in satellite video scenarios are characterized by small size, weak visual features, low resolution, and complex backgrounds. This results in poor distinguishability between target objects and between target objects and the background, as well as background occlusion, which poses certain challenges to target tracking tasks.

[0003] In related target tracking technologies, various target tracking algorithms already exist. For example, one approach decomposes the target tracking task into three sub-tasks: localization, tracking, and ensemble, and processes each sub-task separately through three modules to achieve target tracking. Alternatively, a target tracking task can be defined using natural language specifications, providing a dedicated platform for natural language-specific tracking tasks. A new natural language-based tracking benchmark, TNL2K, has been released, proposing two baselines initialized with natural language and natural language with bounding boxes, respectively, to achieve target tracking.

[0004] However, the aforementioned target tracking algorithms are only suitable for target tracking in general video scenarios and are not suitable for target tracking in satellite video scenarios. Therefore, how to achieve target tracking in satellite video scenarios is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This application provides a target tracking method, apparatus, electronic device, and storage medium to address the existing technology of target tracking in satellite video scenarios, thereby improving the accuracy of target tracking in satellite video scenarios.

[0006] This application provides a target tracking method, including:

[0007] Acquire relevant information for target object tracking, including satellite images, text descriptions of the target object included in the satellite images, target satellite images of the target object extracted from the satellite images, and historical trajectory sequences of the target object obtained based on multiple historical satellite images preceding the satellite images;

[0008] The satellite image, the text description, the target satellite image, and the historical trajectory sequence are input into the joint positioning and sequence tracking model to obtain the target tracking result of the target object;

[0009] The joint localization and sequence tracking model is trained based on multiple relevant information samples and the tracking result labels corresponding to each relevant information sample.

[0010] According to a target tracking method provided in this application, the joint localization and sequence tracking model includes a natural language encoder, a visual encoder, a feature fusion module, and an autoregressive decoder;

[0011] The step of inputting the satellite image, the text description, the target satellite image, and the historical trajectory sequence into the joint positioning and sequence tracking model to obtain the target tracking result of the target object includes:

[0012] The text description is input into the natural language encoder to extract text encoding features;

[0013] The satellite image and the target satellite image are input into the visual encoder to extract the first image coding feature of the satellite image and the second image coding feature of the target satellite image;

[0014] The text encoding features, the first image encoding features, and the second image encoding features are input into the feature fusion module to obtain the target fusion features;

[0015] The target fusion features and the historical trajectory sequence are input into the autoregressive decoder to obtain the target tracking result.

[0016] According to a target tracking method provided in this application, the feature fusion module includes a first linear projection unit, a second linear projection unit, and a multi-source correlation fusion unit;

[0017] The step of inputting the text encoding features, the first image encoding features, and the second image encoding features into the feature fusion module to obtain the target fusion features includes:

[0018] The text encoding features are input into the first linear projection unit, and the text encoding features are linearly projected to obtain the target text encoding features;

[0019] The first image encoding feature and the second image encoding feature are input into the second linear projection unit, and the first image encoding feature and the second image encoding feature are linearly projected respectively to obtain the corresponding first target image encoding feature and second target image encoding feature. The dimensions of the target text encoding feature, the second target image encoding feature and the second target image encoding feature are the same.

[0020] The target fusion feature is determined based on the target text encoding feature, the second target image encoding feature, the second target image encoding feature, and the multi-source association fusion unit.

[0021] According to a target tracking method provided in this application, determining the target fusion feature based on the target text encoding feature, the second target image encoding feature, the second target image encoding feature, and the multi-source association fusion unit includes:

[0022] The target text encoding features, the encoding features corresponding to the empty target satellite image of the target object, and the first target image encoding features are concatenated to obtain the first concatenated feature;

[0023] The target text encoding features, the second target image encoding features, and the first target image encoding features are concatenated to obtain the second concatenated feature;

[0024] The first splicing feature and the second splicing feature are input into the multi-source association fusion unit to obtain the target fusion feature.

[0025] According to a target tracking method provided in this application, the autoregressive decoder includes a multi-head attention unit and a feedforward neural network unit;

[0026] The step of inputting the target fusion features and the historical trajectory sequence into the autoregressive decoder to obtain the target tracking result includes:

[0027] The target fusion features and the historical trajectory sequence are input into the multi-head attention unit to obtain multi-head attention features;

[0028] The multi-head attention features are input into the feedforward neural network unit to obtain the target decoding features;

[0029] The target tracking result is determined by the first sum of the multi-head attention features and the target decoding features.

[0030] According to a target tracking method provided in this application, the multi-head attention unit includes a multi-head cross attention layer and a masked multi-head self-attention layer;

[0031] The step of inputting the target fusion features and the historical trajectory sequence into the multi-head attention unit to obtain multi-head attention features includes:

[0032] The historical trajectory sequence is input into the masked multi-head self-attention layer to obtain the masked multi-head self-attention features;

[0033] The historical trajectory sequence, the second sum of the mask multi-head self-attention features, and the target fusion features are input into the multi-head cross-attention layer to obtain the multi-head cross-attention features;

[0034] The sum of the multi-head cross-attention feature and the second sum value is determined as the multi-head attention feature.

[0035] This application also provides a target tracking device, including:

[0036] The acquisition unit is used to acquire relevant information for target object tracking. The relevant information includes satellite images, text descriptions of the target object included in the satellite images, target satellite images of the target object extracted from the satellite images, and historical trajectory sequences of the target object obtained based on multiple historical satellite images preceding the satellite images.

[0037] The processing unit is used to input the satellite image, the text description, the target satellite image, and the historical trajectory sequence into the joint positioning and sequence tracking model to obtain the target tracking result of the target object;

[0038] The joint localization and sequence tracking model is trained based on multiple relevant information samples and the tracking result labels corresponding to each relevant information sample.

[0039] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target tracking method as described above.

[0040] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target tracking method as described above.

[0041] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the target tracking method as described above.

[0042] The target tracking method, apparatus, electronic device, and storage medium provided in this application first acquire relevant information for target object tracking during target tracking. This information includes satellite images, text descriptions of the target object included in the satellite images, target satellite images of the target object extracted from the satellite images, and historical trajectory sequences of the target object obtained based on multiple previous historical satellite images. The satellite images, text descriptions, target satellite images, and historical trajectory sequences are then input into a joint localization and sequence tracking model to obtain the target object tracking result. This deep learning-based joint localization and sequence tracking model, combining text descriptions and historical trajectory sequences of the target object with satellite images, enables target tracking in satellite video scenarios and improves the accuracy of target tracking in such scenarios. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating a target tracking method provided in an embodiment of this application.

[0045] Figure 2 This is a schematic diagram of a target tracking framework provided in an embodiment of this application.

[0046] Figure 3 This is a schematic diagram of the structure of a joint localization and sequence tracking model provided in an embodiment of this application.

[0047] Figure 4 This is a schematic flowchart of a method for obtaining target tracking results of a target object, provided in an embodiment of this application.

[0048] Figure 5 This is a schematic diagram of the structure of an autoregressive decoder provided in an embodiment of this application.

[0049] Figure 6 This is a schematic diagram of the structure of a target tracking device provided in an embodiment of this application.

[0050] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0053] Target objects in satellite video scenes are characterized by small size, weak visual features, low resolution, and complex backgrounds. This results in poor distinguishability between target objects and between target objects and the background, as well as background occlusion, which poses certain challenges to target tracking tasks.

[0054] Considering that most relevant target tracking technologies are only suitable for target tracking in general video scenarios and not for target tracking in satellite video scenarios, this application provides a target tracking method applicable to target tracking in satellite video scenarios in order to achieve target tracking in satellite video scenarios and improve the accuracy of target tracking in satellite video scenarios. The execution subject of this method can be an electronic device such as a computer, server, or target tracking equipment, or a target tracking device installed in such electronic device. The target tracking device can be implemented through software, hardware, or a combination of both.

[0055] The target tracking method provided in this application will be described in detail below through several specific embodiments. It is understood that these specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0056] Figure 1 This is a flowchart illustrating a target tracking method provided in an embodiment of this application. For example, please refer to... Figure 1 As shown, the target tracking method may include:

[0057] S101. Obtain relevant information for target object tracking, including satellite images, text descriptions of target objects included in satellite images, target satellite images of target objects extracted from satellite images, and historical trajectory sequences of target objects obtained based on multiple historical satellite images preceding the satellite images.

[0058] The satellite image can be the current frame from satellite video, and the text description can be natural language text, which may include the target object's specific location in the satellite image, its appearance features such as color, texture, and shape, its motion status such as speed, acceleration, and direction of motion, and other attribute information. The specific settings can be customized according to actual needs. The target satellite image is a satellite image extracted from the satellite image that only includes the target object, and the historical trajectory sequence is a sequence of trajectories formed by tracking results obtained from each frame of historical satellite imagery.

[0059] For example, suppose the target tracking result of the target object to be acquired this time can be denoted as: ;in, and This indicates the coordinates of the target object in the current satellite image. This indicates the width of the target object in the current satellite image. This represents the altitude of the target object in the current satellite image; the tracking result obtained based on the previous historical satellite image can be denoted as... ;in, and This indicates the coordinates of the target object in the previous frame of historical satellite imagery. This indicates the width of the target object in the previous frame of historical satellite imagery. This indicates the altitude of the target object in the previous historical satellite image; the tracking result obtained based on the previous two historical satellite images can be recorded as:

[0060] .in, and This indicates the coordinates of the target object in the first two historical satellite images. This indicates the width of the target object in the first two frames of historical satellite imagery. This indicates the altitude of the target object in the first two historical satellite images, and so on.

[0061] In this application embodiment, in order to achieve target tracking in satellite video scenes, when tracking targets using satellite images, for example, see [link to relevant documentation]. Figure 2 As shown, Figure 2This is a schematic diagram of a target tracking framework provided in an embodiment of this application. Based on satellite imagery, it combines textual descriptions of the target object as a basis for target tracking to enhance the target's localization capability, enabling more accurate target capture and tracking. Furthermore, it utilizes historical trajectory sequences of the target object for auxiliary tracking (see S102 below), which can predict its potential future movement trends. Using historical trajectory sequences significantly improves the continuity and stability of tracking. Even when the target object temporarily leaves the field of view or is behind obstructions, effective prediction can still be made based on the historical trajectory sequence, reducing the risk of tracking loss. This helps improve the understanding and localization capability of the joint localization and sequence tracking model for small-sized targets, especially when the target is obscured by the background or other objects.

[0062] S102. Input the satellite images, text descriptions, target satellite images, and historical trajectory sequences into the joint positioning and sequence tracking model to obtain the target tracking results of the target object.

[0063] The joint localization and sequence tracking model is trained based on multiple relevant information samples and the tracking result labels corresponding to each relevant information sample.

[0064] As can be seen from the embodiments of this application, when performing target tracking, relevant information for target object tracking is first obtained. This information includes satellite images, text descriptions of the target object included in the satellite images, target satellite images of the target object extracted from the satellite images, and historical trajectory sequences of the target object obtained based on multiple previous historical satellite images. The satellite images, text descriptions, target satellite images, and historical trajectory sequences are then input into the joint localization and sequence tracking model to obtain the target tracking result. This deep learning-based joint localization and sequence tracking model, combining text descriptions and historical trajectory sequences of the target object with satellite images, enables target tracking in satellite video scenarios and improves the accuracy of target tracking in such scenarios.

[0065] Based on the above Figure 1 The illustrated embodiment, for example, in the embodiments of this application, the joint localization and sequence tracking model may include a natural language encoder, a visual encoder, a feature fusion module, and an autoregressive decoder, as can be seen in [reference needed]. Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a joint localization and sequence tracking model provided in an embodiment of this application.

[0066] Combination Figure 3The joint localization and sequence tracking model shown above, in S102, inputs satellite images, text descriptions, target satellite images, and historical trajectory sequences into the joint localization and sequence tracking model to obtain the target tracking results of the target object. For a detailed implementation of this, please refer to the following... Figure 4 The example shown.

[0067] Figure 4 This application provides a flowchart illustrating a method for obtaining target tracking results of a target object, which may include:

[0068] S401. Input the text description into the natural language encoder and extract the text encoding features.

[0069] For example, in the embodiments of this application, Bidirectional Encoder Representations from Transformers (BERT) can be used as a natural language encoder.

[0070] When inputting text descriptions into a natural language encoder, the text descriptions can first be tokenized, and token CLS and SEP can be added to the beginning and end of the tokenized language query to generate a word series token, which can be denoted as . Where N is the maximum length of the language query. (This refers to the word series.) The input is fed into a natural language encoder, which extracts text encoding features. For example, these text encoding features can be denoted as... , It is the dimension of the output embedding, thereby obtaining the text encoding features of the text description.

[0071] S402. Input the satellite image and the target satellite image into the visual encoder to extract the first image coding feature of the satellite image and the second image coding feature of the target satellite image.

[0072] For example, in the embodiments of this application, considering that the vanilla Swin Transformer has good performance in image feature learning, the vanilla Swin Transformer can be used as a visual encoder.

[0073] When extracting the first image coding features of the satellite image and the second image coding features of the target satellite image through the visual encoder, considering that the output feature resolution of the last stage of the Swin Transformer is low, only the first three stages of the Swin Transformer can be retained for image coding feature extraction.

[0074] For example, when locating a target object, assume that the satellite image can be denoted as... This information can then be input into a visual encoder to extract the first image coding feature of the satellite image. For example, this first image coding feature can be denoted as... .

[0075] When tracking a target object, assume that the satellite image can be denoted as... A target satellite image that includes only the target object can be denoted as... Then, both the satellite image and the target satellite image can be input into the visual encoder, which extracts the first image coding features of the satellite image and the second image coding features of the target satellite image. For example, the first image coding feature can be denoted as... The second image coding feature can be denoted as: , This allows us to obtain the first image coding features of the satellite image and the second image coding features of the target satellite image.

[0076] Based on the above description, after obtaining the text encoding features, the first image encoding features, and the second image encoding features, the following S403 can be executed:

[0077] S403. Input the text encoding features, the first image encoding features, and the second image encoding features into the feature fusion module to obtain the target fusion features.

[0078] For example, combined Figure 2 As shown in the embodiments of this application, the feature fusion module may include a first linear projection unit, a second linear projection unit, and a multi-source correlation fusion unit; when inputting text encoding features, first image encoding features, and second image encoding features into the feature fusion module to obtain the target fusion features, it may include:

[0079] The text encoding features are input into the first linear projection unit, and linear projection is performed on the text encoding features to obtain the target text encoding features. The first image encoding features and the second image encoding features are input into the second linear projection unit, and linear projection is performed on the first image encoding features and the second image encoding features respectively to obtain the corresponding first target image encoding features and second target image encoding features. The dimensions of the target text encoding features, the first target image encoding features, and the second target image encoding features are the same. Then, based on the target text encoding features, the second target image encoding features, the second target image encoding features, and the multi-source association fusion unit, the target fusion features are jointly determined.

[0080] The first linear projection unit and the second linear projection unit are used to unify the dimensions of the text coding features, the first image coding features and the second image coding features, that is, to project them to the same dimension, and use the target text coding features, the first target image coding features and the second target image coding features with the same dimension as the input of the multi-source association fusion unit.

[0081] For example, in embodiments of this application, determining target fusion features based on target text encoding features, second target image encoding features, second target image encoding features, and multi-source association fusion unit may include:

[0082] The target text encoding features, the encoding features corresponding to the empty target satellite image of the target object, and the first target image encoding features are concatenated to obtain the first concatenated feature; the target text encoding features, the second target image encoding features, and the first target image encoding features are concatenated to obtain the second concatenated feature; the first and second concatenated features are input into the multi-source association fusion unit to obtain the target fusion feature.

[0083] Here, "empty target satellite image" refers to a satellite image where the target satellite image is empty, containing only the target object. For empty target satellite images, a zero-filling tensor can be used. The zero-padding tensor is used as a placeholder to fill in the encoded features corresponding to the empty target satellite image. Furthermore, the mask value of the zero-padding tensor can be set to 1 to mask it during the self-attention calculation process to prevent contamination of other useful information.

[0084] For example, the multi-source correlation fusion unit can accommodate different localization and tracking references to achieve cross-modal and cross-temporal tracking relationship modeling through natural language specifications. Considering the strong ability of self-attention operations to capture global correlations, a transformer encoder with a main stack of self-attention layers can be used as the multi-source correlation fusion unit.

[0085] When obtaining target fusion features through the feature fusion module, the text encoding features, the first image encoding features, and the second image encoding features can be projected using the first and second linear projection units respectively, to obtain target text encoding features, second target image encoding features, and second target image encoding features of the same dimension. For example, the target text encoding features obtained after projecting the text encoding features can be denoted as... The first target image coding feature obtained by projecting the first image coding feature can be denoted as: The second target image coding feature obtained by projecting the second image coding feature can be denoted as: .

[0086] After obtaining the target text encoding features respectively Encoded features corresponding to empty target satellite images of the target object First target image coding features Second target image coding features Then, features can be encoded for the target text. Encoded features corresponding to empty target satellite images of the target object and the first target image encoding features After splicing, the first splicing feature is obtained, which can be denoted as: ; and encode features for the target text Second target image coding features and the first target image encoding features After splicing, the second splicing feature is obtained, which can be denoted as: Then, the first splicing feature Second splicing features The input is fed into the multi-source association fusion unit, which processes the first spliced ​​feature. Process, for example ,get ; and through the multi-source correlation fusion unit, the second splicing features are analyzed. Process, for example ,get Based on and , to obtain target fusion features .

[0087] Among them, target text encoding features Coding features corresponding to satellite images of empty targets Target text encoding features used for target object localization Second target image coding features Used for target object tracking.

[0088] After obtaining the target fusion features based on satellite imagery, text description, and target satellite imagery, the target fusion features and historical trajectory sequences can be input into the autoregressive decoder, i.e., the following S404 is executed:

[0089] S404. Input the target fusion features and historical trajectory sequence into the autoregressive decoder to obtain the target tracking results.

[0090] For example, in the embodiments of this application, it can be combined with Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of an autoregressive decoder provided in an embodiment of this application. The autoregressive decoder may include a multi-head attention unit and a feedforward neural network unit.

[0091] When the target fusion features and historical trajectory sequences are input into the autoregressive decoder to obtain the target tracking result, the target fusion features and historical trajectory sequences can be input into the multi-head attention unit to obtain the multi-head attention features; the multi-head attention features can be input into the feedforward neural network unit to obtain the target decoding features; and the first sum of the multi-head attention features and the target decoding features is used to determine the target tracking result.

[0092] For example, in the embodiments of this application, it can be combined with Figure 5 As shown, the multi-head attention unit includes a multi-head cross-attention layer and a masked multi-head self-attention layer. Historical trajectory sequences can be input into the masked multi-head self-attention layer to obtain masked multi-head self-attention features. The second sum of the historical trajectory sequence and the masked multi-head self-attention features, along with the target fusion features, are input into the multi-head cross-attention layer to obtain multi-head cross-attention features. The sum of the multi-head cross-attention features and the second sum is determined as the multi-head attention feature. In this way, the multi-head cross-attention layer and the masked multi-head self-attention layer are decoupled and stacked separately, allowing for parallel cross-attention of visual features, thereby effectively improving the efficiency of target tracking result acquisition.

[0093] As can be seen, in this embodiment of the application, when determining the target tracking result through a joint localization and sequence tracking model by combining satellite images, text descriptions, target satellite images, and historical trajectory sequences, the text description can be input into a natural language encoder to extract text encoding features; the satellite images and target satellite images can be input into a visual encoder to extract the first image encoding features of the satellite images and the second image encoding features of the target satellite images; the text encoding features, the first image encoding features, and the second image encoding features can be input into a feature fusion module to obtain target fusion features; and the target fusion features and historical trajectory sequences can be input into an autoregressive decoder to obtain the target tracking result. In this way, the joint localization and sequence tracking model based on deep learning, by combining the text description of the target object and the historical trajectory sequence on the basis of satellite images, can perform target tracking in satellite video scenarios and improve the accuracy of target tracking in satellite video scenarios.

[0094] The target tracking device provided in this application is described below. The target tracking device described below can be referred to in correspondence with the target tracking method described above.

[0095] Figure 6 This is a schematic diagram of the structure of a target tracking device provided in an embodiment of this application. For example, please refer to [link to relevant documentation]. Figure 6 As shown, the target tracking device 60 may include:

[0096] The acquisition unit 601 is used to acquire relevant information for target object tracking. The relevant information includes satellite images, text descriptions of the target object included in the satellite images, target satellite images of the target object extracted from the satellite images, and historical trajectory sequences of the target object obtained based on multiple historical satellite images preceding the satellite images.

[0097] Processing unit 602 is used to input the satellite image, the text description, the target satellite image and the historical trajectory sequence into the joint positioning and sequence tracking model to obtain the target tracking result of the target object;

[0098] The joint localization and sequence tracking model is trained based on multiple relevant information samples and the tracking result labels corresponding to each relevant information sample.

[0099] For example, in the embodiments of this application, the joint localization and sequence tracking model includes a natural language encoder, a visual encoder, a feature fusion module, and an autoregressive decoder;

[0100] The processing unit 602 is configured to input the satellite image, the text description, the target satellite image, and the historical trajectory sequence into the joint positioning and sequence tracking model to obtain the target tracking result of the target object, including:

[0101] The text description is input into the natural language encoder to extract text encoding features;

[0102] The satellite image and the target satellite image are input into the visual encoder to extract the first image coding feature of the satellite image and the second image coding feature of the target satellite image;

[0103] The text encoding features, the first image encoding features, and the second image encoding features are input into the feature fusion module to obtain the target fusion features;

[0104] The target fusion features and the historical trajectory sequence are input into the autoregressive decoder to obtain the target tracking result.

[0105] For example, in an embodiment of this application, the feature fusion module includes a first linear projection unit, a second linear projection unit, and a multi-source correlation fusion unit;

[0106] The step of inputting the text encoding features, the first image encoding features, and the second image encoding features into the feature fusion module to obtain the target fusion features includes:

[0107] The text encoding features are input into the first linear projection unit, and the text encoding features are linearly projected to obtain the target text encoding features;

[0108] The first image encoding feature and the second image encoding feature are input into the second linear projection unit, and the first image encoding feature and the second image encoding feature are linearly projected respectively to obtain the corresponding first target image encoding feature and second target image encoding feature. The dimensions of the target text encoding feature, the second target image encoding feature and the second target image encoding feature are the same.

[0109] The target fusion feature is determined based on the target text encoding feature, the second target image encoding feature, the second target image encoding feature, and the multi-source association fusion unit.

[0110] For example, in an embodiment of this application, the processing unit 602 is configured to determine the target fusion feature based on the target text encoding feature, the second target image encoding feature, the second target image encoding feature, and the multi-source association fusion unit, including:

[0111] The target text encoding features, the encoding features corresponding to the empty target satellite image of the target object, and the first target image encoding features are concatenated to obtain the first concatenated feature;

[0112] The target text encoding features, the second target image encoding features, and the first target image encoding features are concatenated to obtain the second concatenated feature;

[0113] The first splicing feature and the second splicing feature are input into the multi-source association fusion unit to obtain the target fusion feature.

[0114] For example, in an embodiment of this application, the autoregressive decoder includes a multi-head attention unit and a feedforward neural network unit;

[0115] The processing unit 602 is configured to input the target fusion features and the historical trajectory sequence into the autoregressive decoder to obtain the target tracking result, including:

[0116] The target fusion features and the historical trajectory sequence are input into the multi-head attention unit to obtain multi-head attention features;

[0117] The multi-head attention features are input into the feedforward neural network unit to obtain the target decoding features;

[0118] The target tracking result is determined by the first sum of the multi-head attention features and the target decoding features.

[0119] For example, in an embodiment of this application, the multi-head attention unit includes a multi-head cross-attention layer and a masked multi-head self-attention layer;

[0120] The processing unit 602 is used to input the target fusion features and the historical trajectory sequence into the multi-head attention unit to obtain multi-head attention features, including:

[0121] The historical trajectory sequence is input into the masked multi-head self-attention layer to obtain the masked multi-head self-attention features;

[0122] The historical trajectory sequence, the second sum of the mask multi-head self-attention features, and the target fusion features are input into the multi-head cross-attention layer to obtain the multi-head cross-attention features;

[0123] The sum of the multi-head cross-attention feature and the second sum value is determined as the multi-head attention feature.

[0124] The target tracking device 60 provided in this application embodiment can execute the technical solution of the target tracking method in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the target tracking method. Please refer to the implementation principle and beneficial effects of the target tracking method. It will not be repeated here.

[0125] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a target tracking method. This method includes: acquiring relevant information for tracking a target object, the relevant information including satellite images, a text description of the target object included in the satellite images, a target satellite image of the target object extracted from the satellite images, and a historical trajectory sequence of the target object obtained based on multiple historical satellite images preceding the satellite images; inputting the satellite images, the text description, the target satellite image, and the historical trajectory sequence into a joint localization and sequence tracking model to obtain a target tracking result for the target object; wherein the joint localization and sequence tracking model is trained based on multiple relevant information samples and tracking result labels corresponding to each relevant information sample.

[0126] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target tracking method provided by the above methods. The method includes: acquiring relevant information for tracking a target object, the relevant information including satellite images, text descriptions of the target object included in the satellite images, target satellite images of the target object extracted from the satellite images, and historical trajectory sequences of the target object obtained based on multiple historical satellite images preceding the satellite images; inputting the satellite images, the text descriptions, the target satellite images, and the historical trajectory sequences into a joint localization and sequence tracking model to obtain a target tracking result of the target object; wherein the joint localization and sequence tracking model is trained based on multiple relevant information samples and tracking result labels corresponding to each relevant information sample.

[0128] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the target tracking method provided by the above methods. The method includes: acquiring relevant information for tracking a target object, the relevant information including satellite images, a text description of the target object included in the satellite images, a target satellite image of the target object extracted from the satellite images, and a historical trajectory sequence of the target object obtained based on multiple historical satellite images preceding the satellite images; inputting the satellite images, the text description, the target satellite image, and the historical trajectory sequence into a joint localization and sequence tracking model to obtain a target tracking result for the target object; wherein the joint localization and sequence tracking model is trained based on multiple relevant information samples and tracking result labels corresponding to each of the relevant information samples.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A target tracking method, characterized in that, include: Acquire relevant information for target object tracking, including satellite images, text descriptions of the target object included in the satellite images, target satellite images of the target object extracted from the satellite images, and historical trajectory sequences of the target object obtained based on multiple historical satellite images preceding the satellite images; The satellite image, the text description, the target satellite image, and the historical trajectory sequence are input into the joint positioning and sequence tracking model to obtain the target tracking result of the target object; The joint localization and sequence tracking model is trained based on multiple relevant information samples and the tracking result labels corresponding to each relevant information sample; The joint localization and sequence tracking model includes a natural language encoder, a visual encoder, a feature fusion module, and an autoregressive decoder. The autoregressive decoder includes a multi-head attention unit and a feedforward neural network unit. The step of inputting the satellite image, the text description, the target satellite image, and the historical trajectory sequence into the joint positioning and sequence tracking model to obtain the target tracking result of the target object includes: The text description is input into the natural language encoder to extract text encoding features; The satellite image and the target satellite image are input into the visual encoder to extract the first image coding feature of the satellite image and the second image coding feature of the target satellite image; The text encoding features, the first image encoding features, and the second image encoding features are input into the feature fusion module to obtain target fusion features. This includes: inputting the text encoding features into a first linear projection unit and performing linear projection on the text encoding features to obtain target text encoding features; inputting the first image encoding features and the second image encoding features into a second linear projection unit and performing linear projection on the first image encoding features and the second image encoding features respectively to obtain corresponding first target image encoding features and second target image encoding features, wherein the dimensions of the target text encoding features, the dimensions of the second target image encoding features, and the dimensions of the second target image encoding features are... The features are identical in dimension; the target text encoding features, the encoding features corresponding to the empty target satellite image of the target object, and the first target image encoding features are concatenated to obtain a first concatenated feature, wherein the empty target satellite image is a satellite image that only includes the target object when the target satellite image is empty; the target text encoding features, the second target image encoding features, and the first target image encoding features are concatenated to obtain a second concatenated feature; the first concatenated feature and the second concatenated feature are input into a multi-source association fusion unit to obtain the target fusion feature; wherein, the feature fusion module includes a first linear projection unit, a second linear projection unit, and a multi-source association fusion unit; The target fusion feature and the historical trajectory sequence are input into the autoregressive decoder to obtain the target tracking result, including: inputting the target fusion feature and the historical trajectory sequence into the multi-head attention unit to obtain multi-head attention features; inputting the multi-head attention features into the feedforward neural network unit to obtain target decoding features; and determining the target tracking result by using a first sum of the multi-head attention features and the target decoding features.

2. The target tracking method according to claim 1, characterized in that, The multi-head attention unit includes a multi-head cross-attention layer and a masked multi-head self-attention layer; The step of inputting the target fusion features and the historical trajectory sequence into the multi-head attention unit to obtain multi-head attention features includes: The historical trajectory sequence is input into the masked multi-head self-attention layer to obtain the masked multi-head self-attention features; The historical trajectory sequence, the second sum of the mask multi-head self-attention features, and the target fusion features are input into the multi-head cross-attention layer to obtain the multi-head cross-attention features; The sum of the multi-head cross-attention feature and the second sum value is determined as the multi-head attention feature.

3. A target tracking device, characterized in that, include: The acquisition unit is used to acquire relevant information for target object tracking. The relevant information includes satellite images, text descriptions of the target object included in the satellite images, target satellite images of the target object extracted from the satellite images, and historical trajectory sequences of the target object obtained based on multiple historical satellite images preceding the satellite images. The processing unit is used to input the satellite image, the text description, the target satellite image, and the historical trajectory sequence into the joint positioning and sequence tracking model to obtain the target tracking result of the target object; The joint localization and sequence tracking model is trained based on multiple relevant information samples and the tracking result labels corresponding to each relevant information sample; The joint localization and sequence tracking model includes a natural language encoder, a visual encoder, a feature fusion module, and an autoregressive decoder. The autoregressive decoder includes a multi-head attention unit and a feedforward neural network unit. The processing unit is configured to input the satellite image, the text description, the target satellite image, and the historical trajectory sequence into the joint positioning and sequence tracking model to obtain the target tracking result of the target object, including: The text description is input into the natural language encoder to extract text encoding features; The satellite image and the target satellite image are input into the visual encoder to extract the first image coding feature of the satellite image and the second image coding feature of the target satellite image; The text encoding features, the first image encoding features, and the second image encoding features are input into the feature fusion module to obtain target fusion features. This includes: inputting the text encoding features into a first linear projection unit and performing linear projection on the text encoding features to obtain target text encoding features; inputting the first image encoding features and the second image encoding features into a second linear projection unit and performing linear projection on the first image encoding features and the second image encoding features respectively to obtain corresponding first target image encoding features and second target image encoding features, wherein the dimensions of the target text encoding features, the dimensions of the second target image encoding features, and the dimensions of the second target image encoding features are... The features are identical in dimension; the target text encoding features, the encoding features corresponding to the empty target satellite image of the target object, and the first target image encoding features are concatenated to obtain a first concatenated feature, wherein the empty target satellite image is a satellite image that only includes the target object when the target satellite image is empty; the target text encoding features, the second target image encoding features, and the first target image encoding features are concatenated to obtain a second concatenated feature; the first concatenated feature and the second concatenated feature are input into a multi-source association fusion unit to obtain the target fusion feature; wherein, the feature fusion module includes a first linear projection unit, a second linear projection unit, and a multi-source association fusion unit; The target fusion feature and the historical trajectory sequence are input into the autoregressive decoder to obtain the target tracking result, including: inputting the target fusion feature and the historical trajectory sequence into the multi-head attention unit to obtain multi-head attention features; inputting the multi-head attention features into the feedforward neural network unit to obtain target decoding features; and determining the target tracking result by using a first sum of the multi-head attention features and the target decoding features.

4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the target tracking method as described in any one of claims 1 to 2.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the target tracking method as described in any one of claims 1 to 2.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the target tracking method as described in any one of claims 1 to 2.

Images