Target tracking method and device, electronic equipment and storage medium

Abstract

The invention provides a target tracking method and device, electronic equipment and a storage medium, and relates to the technical field of positioning and tracking. The method comprises the following steps: acquiring related information for tracking a target object, the related information comprising a satellite image, a text description of the target object included in the satellite image, and a target satellite image of the target object intercepted from the satellite image, obtaining a historical track sequence of the target object based on multiple frames of historical satellite images before the satellite image; and inputting the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object. According to the deep learning-based joint positioning and sequence tracking model, the text description and the historical trajectory sequence of the target object are combined on the basis of the satellite image to jointly perform target tracking, so that target tracking in a satellite video scene can be realized, and the accuracy of target tracking in the satellite video scene is improved.

Description

Technical Field

[0001] The present application relates to the field of positioning and tracking technology, and in particular to a target tracking method, device, electronic device and storage medium. Background Art

[0002] Compared with target tracking in general video scenes, the target objects in satellite video scenes have the characteristics of small size, weak visual features, low resolution, and complex background. This makes the target objects difficult to distinguish from each other and from the background, and there is background occlusion, which brings certain challenges to the target tracking task.

[0003] There are many different target tracking algorithms in related target tracking technologies. For example, target tracking tasks are decomposed into three subtasks: positioning, tracking, and integration, and each subtask is processed by three modules to achieve target tracking. Alternatively, target tracking tasks are defined by natural language specifications, providing a dedicated platform for natural language specification tracking tasks, releasing a new natural language-based tracking benchmark TNL2K, and proposing two baselines initialized by natural language and natural language with bounding boxes to achieve target tracking.

[0004] However, the above target tracking algorithm is only for target tracking in general video scenes, and is not suitable for target tracking in satellite video scenes. Therefore, how to achieve target tracking in satellite video scenes is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0005] The present application provides a target tracking method, device, electronic device and storage medium to solve the problem of how to achieve target tracking in satellite video scenes in the prior art, thereby improving the accuracy of target tracking in satellite video scenes.

[0006] The present application provides a target tracking method, comprising: Acquire relevant information for tracking a target object, the relevant information including a satellite image, a text description of the target object included in the satellite image, a target satellite image of the target object captured from the satellite image, and a historical trajectory sequence of the target object obtained based on multiple frames of historical satellite images before the satellite image; Inputting the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object; The joint positioning and sequence tracking model is obtained by training based on a plurality of relevant information samples and tracking result labels corresponding to each of the relevant information samples.

[0007] According to a target tracking method provided by the present application, the joint positioning and sequence tracking model includes a natural language encoder, a visual encoder, a feature fusion module and an autoregressive decoder; The step of inputting the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object includes: Inputting the text description into the natural language encoder to extract text encoding features; Inputting the satellite image and the target satellite image into the visual encoder, extracting a first image coding feature of the satellite image and a second image coding feature of the target satellite image; Inputting the text coding feature, the first image coding feature and the second image coding feature into the feature fusion module to obtain a target fusion feature; The target fusion feature and the historical trajectory sequence are input into the autoregressive decoder to obtain the target tracking result.

[0008] According to a target tracking method provided by the present application, the feature fusion module includes a first linear projection unit, a second linear projection unit and a multi-source association fusion unit; The step of inputting the text coding feature, the first image coding feature and the second image coding feature into the feature fusion module to obtain a target fusion feature includes: Inputting the text encoding feature into the first linear projection unit, performing linear projection on the text encoding feature, and obtaining a target text encoding feature; Inputting the first image coding feature and the second image coding feature into the second linear projection unit, linearly projecting the first image coding feature and the second image coding feature respectively, to obtain corresponding first target image coding features and second target image coding features, wherein the dimension of the target text coding feature, the dimension of the second target image coding feature and the dimension of the second target image coding feature are the same; The target fusion feature is determined based on the target text encoding feature, the second target image encoding feature, the second target image encoding feature and the multi-source association fusion unit.

[0009] According to a target tracking method provided by the present application, the target fusion feature is determined based on the target text encoding feature, the second target image encoding feature, the second target image encoding feature and the multi-source association fusion unit, including: Performing splicing processing on the target text coding feature, the coding feature corresponding to the empty target satellite image of the target object, and the first target image coding feature to obtain a first splicing feature; Performing splicing processing on the target text coding feature, the second target image coding feature and the first target image coding feature to obtain a second splicing feature; The first splicing feature and the second splicing feature are input into the multi-source association fusion unit to obtain the target fusion feature.

[0010] According to a target tracking method provided by the present application, the autoregressive decoder includes a multi-head attention unit and a feedforward neural network unit; The step of inputting the target fusion feature and the historical trajectory sequence into the autoregressive decoder to obtain the target tracking result includes: Inputting the target fusion feature and the historical trajectory sequence into the multi-head attention unit to obtain a multi-head attention feature; Inputting the multi-head attention feature into the feedforward neural network unit to obtain the target decoding feature; The target tracking result is determined by taking a first sum of the multi-head attention feature and the target decoding feature.

[0011] According to a target tracking method provided by the present application, the multi-head attention unit includes a multi-head cross attention layer and a masked multi-head self-attention layer; The step of inputting the target fusion feature and the historical trajectory sequence into the multi-head attention unit to obtain the multi-head attention feature includes: Inputting the historical trajectory sequence into the masked multi-head self-attention layer to obtain masked multi-head self-attention features; Inputting the second sum of the historical trajectory sequence and the masked multi-head self-attention feature, and the target fusion feature into the multi-head cross attention layer to obtain a multi-head cross attention feature; The sum of the multi-head cross-attention feature and the second sum value is determined as the multi-head attention feature.

[0012] The present application also provides a target tracking device, comprising: an acquisition unit, configured to acquire relevant information for tracking a target object, wherein the relevant information includes a satellite image, a text description of the target object included in the satellite image, a target satellite image of the target object captured from the satellite image, and a historical trajectory sequence of the target object obtained based on a plurality of frames of historical satellite images before the satellite image; A processing unit, configured to input the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object; The joint positioning and sequence tracking model is obtained by training based on a plurality of relevant information samples and tracking result labels corresponding to each of the relevant information samples.

[0013] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the target tracking method described in any one of the above is implemented.

[0014] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the target tracking method as described in any one of the above is implemented.

[0015] The present application also provides a computer program product, including a computer program, which implements any of the target tracking methods described above when executed by a processor.

[0016] The target tracking method, device, electronic device and storage medium provided by the present application first obtain relevant information for tracking the target object when performing target tracking. The relevant information includes satellite images, text descriptions of the target objects included in the satellite images, target satellite images of the target objects captured from the satellite images, and the historical trajectory sequence of the target objects obtained based on multiple frames of historical satellite images before the satellite images; the satellite images, text descriptions, target satellite images and historical trajectory sequences are input into the joint positioning and sequence tracking model to obtain the target tracking results of the target object. In this way, the joint positioning and sequence tracking model based on deep learning combines the text description and historical trajectory sequence of the target object on the basis of satellite images to jointly perform target tracking, which can realize target tracking in satellite video scenes and improve the accuracy of target tracking in satellite video scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a target tracking method provided in an embodiment of the present application.

[0019] Figure 2A schematic diagram of a target tracking framework provided in an embodiment of the present application.

[0020] Figure 3 A schematic diagram of the structure of a joint positioning and sequence tracking model provided in an embodiment of the present application.

[0021] Figure 4 A flowchart of a method for obtaining target tracking results of a target object provided in an embodiment of the present application.

[0022] Figure 5 A schematic diagram of the structure of an autoregressive decoder provided in an embodiment of the present application.

[0023] Figure 6 A schematic diagram of the structure of a target tracking device provided in an embodiment of the present application.

[0024] Figure 7 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] In the embodiments of the present application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of the present application, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0027] The target objects in satellite video scenes are small in size, weak in visual features, low in resolution, and have complex backgrounds. This results in poor distinguishability between target objects and between target objects and backgrounds, as well as background occlusion, which poses certain challenges to the target tracking task.

[0028] Considering that most of the relevant target tracking technologies are only for target tracking in general video scenes, and are not suitable for target tracking in satellite video scenes, therefore, in order to achieve target tracking in satellite video scenes and improve the accuracy of target tracking in satellite video scenes, the embodiment of the present application provides a target tracking method, which can be applicable to target tracking in satellite video scenes. The execution subject of the method can be an electronic device such as a computer, a server, or a target tracking device, or a target tracking device set in the electronic device, and the target tracking device can be implemented by software, hardware, or a combination of the two.

[0029] The target tracking method provided by the present application will be described in detail below through the following specific embodiments. It is understandable that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0030] Figure 1 A flow chart of a target tracking method provided in an embodiment of the present application, for example, see Figure 1 As shown, the target tracking method may include: S101. Obtain relevant information for tracking a target object, where the relevant information includes a satellite image, a text description of the target object included in the satellite image, a target satellite image of the target object captured from the satellite image, and a historical trajectory sequence of the target object obtained based on multiple frames of historical satellite images before the satellite image.

[0031] The satellite image may be the current frame image in the satellite video, and the text description may be a natural language text description, for example, it may include the specific location of the target object in the satellite image, the appearance features of the target object, such as color, texture, shape, etc., the motion state of the target object, such as speed, acceleration, motion direction, etc., and of course, other attribute information of the target object, etc., which may be set according to actual needs. The target satellite image is a satellite image that only includes the target object and is intercepted from the satellite image, and the historical trajectory sequence is a trajectory sequence composed of the tracking results obtained from each frame of the historical satellite image.

[0032] For example, assuming that the target tracking result of the target object to be acquired this time can be recorded as ;in, and Indicates the coordinates of the target object in the current satellite image. Indicates the width of the target object in the current satellite image. represents the height of the target object in the current satellite image; then the tracking result obtained based on the previous historical satellite image of the satellite image can be recorded as ;in, and Indicates the coordinates of the target object in the previous frame of historical satellite image. Indicates the width of the target object in the previous frame of historical satellite image. Indicates the height of the target object in the previous frame of historical satellite image; the tracking result obtained based on the previous two frames of historical satellite image can be recorded as: .in, and Represents the coordinates of the target object in the previous two frames of historical satellite images, Indicates the width of the target object in the previous two frames of historical satellite images. Indicates the height of the target object in the previous two frames of historical satellite images, and so on.

[0033] In the embodiment of the present application, in order to realize target tracking in a satellite video scene, when tracking a target through a satellite image, for example, see Figure 2 As shown, Figure 2 A schematic diagram of a target tracking framework provided for an embodiment of the present application, based on satellite images, on the one hand, a text description of the target object is used as the basis for target tracking to enhance the target's positioning capability, and the target can be captured and tracked more accurately; on the other hand, the historical trajectory sequence of the target object is combined for auxiliary tracking, as shown in S102 below, and its possible future movement trend can be predicted. The use of the historical trajectory sequence can significantly improve the continuity and stability of tracking. When the target object temporarily leaves the field of view or is behind an obstruction, it can still be effectively predicted based on the historical trajectory sequence to reduce the risk of tracking loss, which helps to improve the joint positioning and sequence tracking model's understanding and positioning capabilities for small-sized targets, especially when the target is obscured by the background or other objects.

[0034] S102: input the satellite image, text description, target satellite image and historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object.

[0035] Among them, the joint positioning and sequence tracking model is trained based on multiple relevant information samples and the tracking result labels corresponding to each relevant information sample.

[0036] It can be seen that in the embodiment of the present application, when performing target tracking, relevant information for tracking the target object is first obtained, and the relevant information includes satellite images, text descriptions of the target objects included in the satellite images, target satellite images of the target objects captured from the satellite images, and the historical trajectory sequence of the target objects obtained based on multiple frames of historical satellite images before the satellite images; the satellite images, text descriptions, target satellite images, and historical trajectory sequences are input into the joint positioning and sequence tracking model to obtain the target tracking results of the target object. In this way, the joint positioning and sequence tracking model based on deep learning combines the text description and historical trajectory sequence of the target object on the basis of satellite images to jointly perform target tracking, which can realize target tracking in satellite video scenarios and improve the accuracy of target tracking in satellite video scenarios.

[0037] Based on the above Figure 1 In the embodiment shown, for example, in the embodiment of the present application, the joint positioning and sequence tracking model may include a natural language encoder, a visual encoder, a feature fusion module and an autoregressive decoder, which can be seen in Figure 3 As shown, Figure 3 A schematic diagram of the structure of a joint positioning and sequence tracking model provided in an embodiment of the present application.

[0038] Combination Figure 3 In the joint positioning and sequence tracking model shown in the above S102, the satellite image, text description, target satellite image and historical trajectory sequence are input into the joint positioning and sequence tracking model to obtain the target object tracking result. For a specific implementation, please refer to the following Figure 4 The embodiment shown.

[0039] Figure 4 A schematic flow chart of a method for obtaining a target tracking result of a target object provided in an embodiment of the present application, the method may include: S401: Input the text description into a natural language encoder to extract text encoding features.

[0040] For example, in an embodiment of the present application, Bidirectional Encoder Representations from Transformers (BERT) may be used as a natural language encoder.

[0041] When inputting the text description into the natural language encoder, the text description can be tokenized first, and the tokens CLS and SEP are added at the beginning and end of the tokenized language query to generate a word series token, which can be recorded as , where N is the maximum length of a language query. Input into the natural language encoder, and extract the text encoding features through the natural language encoder. For example, the text encoding features can be recorded as , is the dimension of the output embedding, thereby obtaining the text encoding features of the text description.

[0042] S402: Input the satellite image and the target satellite image into a visual encoder, and extract a first image coding feature of the satellite image and a second image coding feature of the target satellite image.

[0043] For example, in the embodiment of the present application, considering that the vanilla Swin Transformer has good performance in image feature learning, the vanilla Swin Transformer can be used as the visual encoder.

[0044] When extracting the first image coding features of the satellite image and the second image coding features of the target satellite image through the visual encoder, considering that the output feature resolution of the last level of the Swin Transformer is low, only the first three levels of the Swin Transformer can be retained for image coding feature extraction.

[0045] For example, when locating the target object, it is assumed that the satellite image can be recorded as , it can be input into the visual encoder, and the first image coding feature of the satellite image can be extracted by the visual encoder. For example, the first image coding feature can be recorded as .

[0046] When tracking the target object, it is assumed that the satellite image can be recorded as , the target satellite image including only the target object can be recorded as , then both the satellite image and the target satellite image can be input into the visual encoder, and the first image coding feature of the satellite image and the second image coding feature of the target satellite image can be extracted by the visual encoder. For example, the first image coding feature can be recorded as , the second image coding feature can be recorded as , , thereby obtaining the first image coding feature of the satellite image and the second image coding feature of the target satellite image.

[0047] In combination with the above description, after respectively acquiring the text coding feature, the first image coding feature, and the second image coding feature, the following S403 may be performed: S403: Input the text coding feature, the first image coding feature and the second image coding feature into a feature fusion module to obtain a target fusion feature.

[0048] For example, combined with Figure 2 As shown, in the embodiment of the present application, the feature fusion module may include a first linear projection unit, a second linear projection unit and a multi-source association fusion unit; when the text coding feature, the first image coding feature and the second image coding feature are input into the feature fusion module and the target fusion feature is obtained, it may include: The text coding feature is input into the first linear projection unit, and the text coding feature is linearly projected to obtain the target text coding feature; the first image coding feature and the second image coding feature are input into the second linear projection unit, and the first image coding feature and the second image coding feature are linearly projected respectively to obtain the corresponding first target image coding feature and the second target image coding feature; wherein the dimension of the target text coding feature, the dimension of the first target image coding feature and the dimension of the second target image coding feature are the same; then based on the target text coding feature, the second target image coding feature, the second target image coding feature and the multi-source association fusion unit, the target fusion feature is jointly determined.

[0049] Among them, the role of the first linear projection unit and the second linear projection unit is to achieve dimensional unification of the text coding features, the first image coding features and the second image coding features, that is, to project them into the same dimension, and use the target text coding features, the first target image coding features, and the second target image coding features with the same dimension as the input of the multi-source association fusion unit.

[0050] For example, in the embodiment of the present application, determining the target fusion feature based on the target text coding feature, the second target image coding feature, the second target image coding feature and the multi-source association fusion unit may include: The target text coding feature, the coding feature corresponding to the empty target satellite image of the target object and the first target image coding feature are spliced ​​to obtain a first spliced ​​feature; the target text coding feature, the second target image coding feature and the first target image coding feature are spliced ​​to obtain a second spliced ​​feature; the first spliced ​​feature and the second spliced ​​feature are input into a multi-source association fusion unit to obtain a target fusion feature.

[0051] Among them, the empty target satellite image is a satellite image when the target satellite image that only includes the target object is empty. For the empty target satellite image, you can use a zero-filled tensor As a placeholder, it is filled to obtain the encoding features corresponding to the empty target satellite image. In addition, the mask value of the zero-filled tensor can be set to 1 to mask it during the self-attention calculation process to prevent contamination of other useful information.

[0052] For example, the multi-source correlation fusion unit can accommodate different localization and tracking references to achieve cross-modal and cross-temporal tracking relationship modeling through natural language specifications. Considering that the self-attention operation has a strong ability to capture global correlation, the transformer encoder with mainly stacked self-attention layers can be used as a multi-source correlation fusion unit.

[0053] When obtaining the target fusion feature through the feature fusion module, the text encoding feature, the first image encoding feature and the second image encoding feature can be projected respectively through the first linear projection unit and the second linear projection unit therein to obtain the target text encoding feature, the second target image encoding feature and the second target image encoding feature of the same dimension. For example, the target text encoding feature obtained after the text encoding feature is projected can be recorded as , the first target image coding feature obtained after the projection of the first image coding feature can be recorded as , the second target image coding feature obtained after the projection of the second image coding feature can be recorded as .

[0054] After obtaining the target text encoding features respectively , the coding features corresponding to the empty target satellite image of the target object , the first target image encoding feature and the second target image encoding feature After that, the target text can be encoded with features , the coding features corresponding to the empty target satellite image of the target object and the first target image encoding feature Perform splicing processing to obtain the first splicing feature, which can be recorded as ; and encode features of the target text , the second target image encoding feature and the first target image encoding feature After splicing, the second splicing feature is obtained, which can be recorded as ; Then the first splicing feature and the second stitching feature Input to the multi-source association fusion unit, and the multi-source association fusion unit is used to fusion the first splicing feature To process, for example ,get ; and through the multi-source association fusion unit to the second splicing feature To process, for example ,get , thus based on and , get the target fusion feature .

[0055] Among them, the target text encoding features Encoded features corresponding to satellite images of empty targets Used to locate the target object and the target text encoding features and the second target image encoding feature Used for target object tracking.

[0056] After obtaining the target fusion features based on the satellite image, the text description and the target satellite image, the target fusion features and the historical trajectory sequence can be input into the autoregressive decoder, that is, the following S404 is executed: S404: Input the target fusion features and the historical trajectory sequence into the autoregressive decoder to obtain the target tracking result.

[0057] For example, in the embodiments of the present application, it can be combined with Figure 5 As shown, Figure 5 A structural schematic diagram of an autoregressive decoder provided in an embodiment of the present application, the autoregressive decoder may include a multi-head attention unit and a feedforward neural network unit.

[0058] The target fusion features and the historical trajectory sequence are input into the autoregressive decoder. When the target tracking result is obtained, the target fusion features and the historical trajectory sequence can be input into the multi-head attention unit to obtain the multi-head attention features; the multi-head attention features are input into the feedforward neural network unit to obtain the target decoding features; and then the first sum of the multi-head attention features and the target decoding features is used to determine the target tracking result.

[0059] For example, in the embodiments of the present application, it can be combined with Figure 5 As shown, the multi-head attention unit includes a multi-head cross attention layer and a masked multi-head self-attention layer. The historical trajectory sequence can be input into the masked multi-head self-attention layer to obtain the masked multi-head self-attention feature; the second sum of the historical trajectory sequence and the masked multi-head self-attention feature and the target fusion feature are input into the multi-head cross attention layer to obtain the multi-head cross attention feature; the sum of the multi-head cross attention feature and the second sum is determined as the multi-head attention feature. In this way, the multi-head cross attention layer and the masked multi-head self-attention layer are decoupled and stacked separately, and the cross attention of visual features can be performed in parallel, thereby effectively improving the efficiency of obtaining target tracking results.

[0060] It can be seen that in the embodiment of the present application, when the target tracking result is determined by combining the satellite image, text description, target satellite image and historical trajectory sequence through the joint positioning and sequence tracking model, the text description can be input into the natural language encoder to extract the text coding features; the satellite image and the target satellite image are input into the visual encoder to extract the first image coding features of the satellite image and the second image coding features of the target satellite image; the text coding features, the first image coding features and the second image coding features are input into the feature fusion module to obtain the target fusion features; the target fusion features and the historical trajectory sequence are then input into the autoregressive decoder to obtain the target tracking result. In this way, the joint positioning and sequence tracking model based on deep learning combines the text description and the historical trajectory sequence of the target object on the basis of the satellite image to jointly perform target tracking, which can realize target tracking in the satellite video scene and improve the accuracy of target tracking in the satellite video scene.

[0061] The target tracking device provided in the present application is described below. The target tracking device described below and the target tracking method described above can be referenced to each other.

[0062] Figure 6 A schematic diagram of a target tracking device provided in an embodiment of the present application is provided. For example, see Figure 6 As shown, the target tracking device 60 may include: An acquisition unit 601 is used to acquire relevant information for tracking a target object, wherein the relevant information includes a satellite image, a text description of the target object included in the satellite image, a target satellite image of the target object captured from the satellite image, and a historical trajectory sequence of the target object obtained based on multiple frames of historical satellite images before the satellite image; A processing unit 602 is used to input the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object; The joint positioning and sequence tracking model is obtained by training based on a plurality of relevant information samples and tracking result labels corresponding to each of the relevant information samples.

[0063] For example, in an embodiment of the present application, the joint positioning and sequence tracking model includes a natural language encoder, a visual encoder, a feature fusion module and an autoregressive decoder; The processing unit 602 is used to input the satellite image, the text description, the target satellite image and the historical trajectory sequence into the joint positioning and sequence tracking model to obtain the target tracking result of the target object, including: Inputting the text description into the natural language encoder to extract text encoding features; Inputting the satellite image and the target satellite image into the visual encoder, extracting a first image coding feature of the satellite image and a second image coding feature of the target satellite image; Inputting the text coding feature, the first image coding feature and the second image coding feature into the feature fusion module to obtain a target fusion feature; The target fusion feature and the historical trajectory sequence are input into the autoregressive decoder to obtain the target tracking result.

[0064] For example, in an embodiment of the present application, the feature fusion module includes a first linear projection unit, a second linear projection unit and a multi-source association fusion unit; The step of inputting the text coding feature, the first image coding feature and the second image coding feature into the feature fusion module to obtain a target fusion feature includes: Inputting the text encoding feature into the first linear projection unit, performing linear projection on the text encoding feature, and obtaining a target text encoding feature; Inputting the first image coding feature and the second image coding feature into the second linear projection unit, linearly projecting the first image coding feature and the second image coding feature respectively, to obtain corresponding first target image coding features and second target image coding features, wherein the dimension of the target text coding feature, the dimension of the second target image coding feature and the dimension of the second target image coding feature are the same; The target fusion feature is determined based on the target text encoding feature, the second target image encoding feature, the second target image encoding feature and the multi-source association fusion unit.

[0065] For example, in the embodiment of the present application, the processing unit 602 is used to determine the target fusion feature based on the target text encoding feature, the second target image encoding feature, the second target image encoding feature and the multi-source association fusion unit, including: Performing splicing processing on the target text coding feature, the coding feature corresponding to the empty target satellite image of the target object, and the first target image coding feature to obtain a first splicing feature; Performing splicing processing on the target text coding feature, the second target image coding feature and the first target image coding feature to obtain a second splicing feature; The first splicing feature and the second splicing feature are input into the multi-source association fusion unit to obtain the target fusion feature.

[0066] For example, in an embodiment of the present application, the autoregressive decoder includes a multi-head attention unit and a feedforward neural network unit; The processing unit 602 is used to input the target fusion feature and the historical trajectory sequence into the autoregressive decoder to obtain the target tracking result, including: Inputting the target fusion feature and the historical trajectory sequence into the multi-head attention unit to obtain a multi-head attention feature; Inputting the multi-head attention feature into the feedforward neural network unit to obtain the target decoding feature; The target tracking result is determined by taking a first sum of the multi-head attention feature and the target decoding feature.

[0067] For example, in an embodiment of the present application, the multi-head attention unit includes a multi-head cross attention layer and a masked multi-head self-attention layer; The processing unit 602 is used to input the target fusion feature and the historical trajectory sequence into the multi-head attention unit to obtain the multi-head attention feature, including: Inputting the historical trajectory sequence into the masked multi-head self-attention layer to obtain masked multi-head self-attention features; Inputting the second sum of the historical trajectory sequence and the masked multi-head self-attention feature, and the target fusion feature into the multi-head cross attention layer to obtain a multi-head cross attention feature; The sum of the multi-head cross-attention feature and the second sum value is determined as the multi-head attention feature.

[0068] The target tracking device 60 provided in the embodiment of the present application can execute the technical solution of the target tracking method in any of the above-mentioned embodiments. Its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the target tracking method. Please refer to the implementation principle and beneficial effects of the target tracking method, and no further details will be given here.

[0069] Figure 7 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the target tracking method, which includes: obtaining relevant information for tracking the target object, the relevant information including a satellite image, a text description of the target object included in the satellite image, a target satellite image of the target object intercepted from the satellite image, and a historical trajectory sequence of the target object obtained based on multiple frames of historical satellite images before the satellite image; inputting the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object; wherein the joint positioning and sequence tracking model is trained based on multiple relevant information samples and tracking result labels corresponding to each of the relevant information samples.

[0070] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0071] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target tracking method provided by the above methods, the method including: obtaining relevant information for tracking a target object, the relevant information including a satellite image, a text description of the target object included in the satellite image, a target satellite image of the target object captured from the satellite image, and a historical trajectory sequence of the target object obtained based on multiple frames of historical satellite images before the satellite image; inputting the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object; wherein the joint positioning and sequence tracking model is trained based on multiple relevant information samples and tracking result labels corresponding to each of the relevant information samples.

[0072] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the target tracking method provided by the above methods, the method comprising: obtaining relevant information for tracking a target object, the relevant information comprising a satellite image, a text description of the target object included in the satellite image, a target satellite image of the target object captured from the satellite image, and a historical trajectory sequence of the target object obtained based on multiple frames of historical satellite images before the satellite image; inputting the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object; wherein the joint positioning and sequence tracking model is trained based on multiple relevant information samples and tracking result labels corresponding to each of the relevant information samples.

[0073] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A target tracking method, characterized in that: include: Acquire relevant information for tracking a target object, the relevant information including a satellite image, a text description of the target object included in the satellite image, a target satellite image of the target object captured from the satellite image, and a historical trajectory sequence of the target object obtained based on multiple frames of historical satellite images before the satellite image; Inputting the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object; The joint positioning and sequence tracking model is obtained by training based on a plurality of relevant information samples and tracking result labels corresponding to each of the relevant information samples.

2. The target tracking method according to claim 1, characterized in that: The joint positioning and sequence tracking model includes a natural language encoder, a visual encoder, a feature fusion module and an autoregressive decoder; The step of inputting the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object includes: Inputting the text description into the natural language encoder to extract text encoding features; Inputting the satellite image and the target satellite image into the visual encoder, extracting a first image coding feature of the satellite image and a second image coding feature of the target satellite image; Inputting the text coding feature, the first image coding feature and the second image coding feature into the feature fusion module to obtain a target fusion feature; The target fusion feature and the historical trajectory sequence are input into the autoregressive decoder to obtain the target tracking result.

3. The target tracking method according to claim 2, characterized in that: The feature fusion module includes a first linear projection unit, a second linear projection unit and a multi-source association fusion unit; The step of inputting the text coding feature, the first image coding feature and the second image coding feature into the feature fusion module to obtain a target fusion feature includes: Inputting the text encoding feature into the first linear projection unit, performing linear projection on the text encoding feature, and obtaining a target text encoding feature; Inputting the first image coding feature and the second image coding feature into the second linear projection unit, linearly projecting the first image coding feature and the second image coding feature respectively, to obtain corresponding first target image coding features and second target image coding features, wherein the dimension of the target text coding feature, the dimension of the second target image coding feature and the dimension of the second target image coding feature are the same; The target fusion feature is determined based on the target text encoding feature, the second target image encoding feature, the second target image encoding feature and the multi-source association fusion unit.

4. The target tracking method according to claim 3, characterized in that: The determining the target fusion feature based on the target text encoding feature, the second target image encoding feature, the second target image encoding feature and the multi-source association fusion unit includes: Performing splicing processing on the target text coding feature, the coding feature corresponding to the empty target satellite image of the target object, and the first target image coding feature to obtain a first splicing feature; Performing splicing processing on the target text coding feature, the second target image coding feature and the first target image coding feature to obtain a second splicing feature; The first splicing feature and the second splicing feature are input into the multi-source association fusion unit to obtain the target fusion feature.

5. The target tracking method according to any one of claims 2 to 4, characterized in that: The autoregressive decoder includes a multi-head attention unit and a feedforward neural network unit; The step of inputting the target fusion feature and the historical trajectory sequence into the autoregressive decoder to obtain the target tracking result includes: Inputting the target fusion feature and the historical trajectory sequence into the multi-head attention unit to obtain a multi-head attention feature; Inputting the multi-head attention feature into the feedforward neural network unit to obtain the target decoding feature; The target tracking result is determined by taking a first sum of the multi-head attention feature and the target decoding feature.

6. The target tracking method according to claim 5, characterized in that: The multi-head attention unit includes a multi-head cross attention layer and a masked multi-head self-attention layer; The step of inputting the target fusion feature and the historical trajectory sequence into the multi-head attention unit to obtain the multi-head attention feature includes: Inputting the historical trajectory sequence into the masked multi-head self-attention layer to obtain masked multi-head self-attention features; Inputting the second sum of the historical trajectory sequence and the masked multi-head self-attention feature, and the target fusion feature into the multi-head cross attention layer to obtain a multi-head cross attention feature; The sum of the multi-head cross-attention feature and the second sum value is determined as the multi-head attention feature.

7. A target tracking device, characterized in that: include: an acquisition unit, configured to acquire relevant information for tracking a target object, wherein the relevant information includes a satellite image, a text description of the target object included in the satellite image, a target satellite image of the target object captured from the satellite image, and a historical trajectory sequence of the target object obtained based on a plurality of frames of historical satellite images before the satellite image; A processing unit, configured to input the satellite image, the text description, the target satellite image and the historical trajectory sequence into a joint positioning and sequence tracking model to obtain a target tracking result of the target object; The joint positioning and sequence tracking model is obtained by training based on a plurality of relevant information samples and tracking result labels corresponding to each of the relevant information samples.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the target tracking method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target tracking method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the target tracking method according to any one of claims 1 to 6 is implemented.

Images