Remote sensing image multi-target tracking method and device based on text guidance

Abstract

The invention provides a remote sensing image multi-target tracking method and device based on text guidance. The method comprises the following steps: acquiring a remote sensing image video and text description of a to-be-tracked target; and inputting the remote sensing image video and the text description of the target to be tracked into a pre-trained remote sensing multi-target tracking model based on text guidance, and outputting a multi-frame tracking result of the target conforming to the text description. The method can automatically detect and track specific airplanes or ships and other types of targets in the remote sensing video image according to the text description of the target appearance and the motion trail. A target with specific semantic features can be automatically retrieved and tracked in a remote sensing video according to natural language description, and manual screening is avoided. The method supports simultaneous tracking of a plurality of targets meeting description, and is suitable for the attention demand of a plurality of targets in an actual task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a text-guided remote sensing image multi-target tracking method and device. BACKGROUND

[0002] In practical application scenarios such as maritime traffic management, civil aviation flight safety, maritime and aerial search and rescue, environmental monitoring, and combating illegal activities, there is an urgent need for efficient and accurate remote sensing monitoring technology to solve the identification and tracking of targets such as aircraft and ships with specific semantic features. Traditional remote sensing multi-target tracking methods usually use computer vision algorithms to first detect and track all specific category targets in images or videos, and then use artificial methods or additional rule programs to filter the target of interest. This method can still be applied in static or small-scale scenarios, but when faced with a large amount of remote sensing image data, complex environmental interference, or time-sensitive scenarios (such as disaster response, emergency warning), it is difficult to meet the requirements of high real-time performance and high accuracy due to high cost, low efficiency, high error rate, and lack of flexibility and universality. SUMMARY

[0003] The present application provides a text-guided remote sensing image multi-target tracking method and device to solve the defects of existing remote sensing multi-target tracking methods in the face of a large amount of remote sensing image data, complex environmental interference, or time-sensitive scenarios (such as disaster response, emergency warning), which are difficult to meet the requirements of high real-time performance and high accuracy. The technical solution proposed by the present application is as follows: In a first aspect, the present application provides a text-guided remote sensing image multi-target tracking method, comprising: obtaining a remote sensing image video and a text description of a target to be tracked; inputting the remote sensing image video and the text description of the target to be tracked into a pre-trained text-guided remote sensing multi-target tracking model, and outputting multi-frame tracking results of the target conforming to the text description.

[0004] Optionally, the text-guided remote sensing multi-target tracking model is trained by the following method: obtain a plurality of remote sensing video samples containing a plurality of moving targets, and label the remote sensing video samples to obtain a remote sensing target tracking data set containing text descriptions; use the remote sensing target tracking data set containing text descriptions to train a pre-established text-guided remote sensing multi-target tracking model.

[0005] Optionally, the remote sensing video samples containing a plurality of moving targets are labeled by the following method: frame the remote sensing video sample containing a plurality of moving targets according to a preset frame rate to obtain a continuous sequence of static remote sensing images; use a target detection labeling tool to label targets in each frame of the sequence of static remote sensing images; extract appearance features, position features, and motion features of the moving targets; input the appearance features, position features, and motion features of the moving targets into a large language model to output a target text description; load the remote sensing video sample, the target label, and the target text description, determine the starting frame and the ending frame of the target text description corresponding to the moving target in the remote sensing video sample, and complete the time consistency binding between the target text description and the moving target according to the starting frame and the ending frame; wherein each target text description corresponds to at least one or more target identifiers and their frame range of appearance.

[0006] Optionally, the remote sensing target tracking dataset includes a remote sensing video sample, target labels of each frame in the video, and target identifiers in each frame in the video that meet the target text description; wherein the target label at least contains bounding box coordinates, target identifiers, and class labels, and the target text description covers the appearance features, position features, and motion features of the target.

[0007] Optionally, the text-guided remote sensing multi-target tracking model includes a text encoder, an image encoder, a cross-modal information fusion module, a target tracking module, and a perceptible text memory module. input the remote sensing video and the text description of the target to be tracked into a pre-trained text-guided remote sensing multi-target tracking model, and output a multi-frame tracking result of the target meeting the text description, including: the text encoder converts the text description of the target to be tracked into a semantic feature vector, and outputs a text feature; the image encoder extracts multi-scale image features of each frame in the remote sensing video; the cross-modal information fusion module fuses the text feature and the image feature to generate a multi-modal feature map containing visual information and semantic information; obtain a target query vector of a historical frame, and the target tracking module generates a current query vector according to the multi-modal feature map and the target query vector of the historical frame; wherein the current query vector represents a candidate target state in the current frame; the perceptible text memory module performs multi-layer cross-attention fusion on the current query vector, the target query vector of the historical frame, and the text feature to output a semantic-enhanced query vector; Generate multi-frame tracking results of the target conforming to the text description based on the semantic enhanced query vector; wherein the multi-frame tracking results at least include bounding box coordinates of each target, target representation, class prediction and tracking confidence.

[0008] Optionally, the cross-modal information fusion module fuses the text features and the image features through feature splicing, attention mechanism or cross-modal attention network.

[0009] In the second aspect, the present application further provides a text-guided remote sensing image multi-target tracking device, comprising the following modules: The data acquisition module is configured to acquire a remote sensing image video and a text description of a target to be tracked. The target tracking module is configured to input the remote sensing image video and the text description of the target to be tracked into a pre-trained text-guided remote sensing multi-target tracking model, and output multi-frame tracking results of the target conforming to the text description.

[0010] In the third aspect, the present application further provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor implements the text-guided remote sensing image multi-target tracking method of the first aspect when executing the computer program.

[0011] In the fourth aspect, the present application further provides a non-transitory computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the text-guided remote sensing image multi-target tracking method of the first aspect.

[0012] In the fifth aspect, the present application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the text-guided remote sensing image multi-target tracking method of the first aspect.

[0013] Based on the above technical solutions, the present application has the following beneficial effects compared with the prior art: The text-guided remote sensing image multi-target tracking method and device provided by the application directly input remote sensing image video and text description of a target to be tracked into a pre-trained model, without manually screening targets in a large amount of image data one by one, relying on model automatic processing, greatly improving processing efficiency, reducing cost, and being able to effectively deal with a large amount of remote sensing image data. The text-guided remote sensing multi-target tracking model pre-trained by the application can contain various semantic feature information of the target, and the model can more accurately identify and distinguish the target meeting the text description in a complex environment according to the rich semantic information, thereby enhancing the flexibility and universality of the method, so as to overcome the interference of the complex environment. Through model automatic processing, the multi-frame tracking result of the target meeting the text description is quickly output after the input data, greatly shortening the processing time, being able to provide information for the related scene in time, and meeting the high real-time requirement. The text-guided remote sensing multi-target tracking model provided by the application uses the accurate semantic information of the text description to identify and track the target, reduces the error caused by human factors, can more accurately output the target tracking result meeting the requirement, and meets the high accuracy requirement.

[0014] Other features and advantages of the present application will be set forth in the descriptions below, and in part will become apparent to those skilled in the art from the descriptions, or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and achieved by the structures particularly pointed out in the description and the drawings.

[0015] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0017] Figure 1 is a flowchart of the text-guided remote sensing image multi-target tracking method provided by the application.

[0018] Figure 2 is a training flowchart of the text-guided remote sensing multi-target tracking model provided by the application.

[0019] Figure 3 is a data labeling flowchart provided by the application.

[0020] Figure 4A framework flow schematic diagram of a text-guided remote sensing multi-target tracking model provided by the present application.

[0021] Figure 5 A structural schematic diagram of a text-aware memory module provided by the present application.

[0022] Figure 6 A structural schematic diagram of a text-guided remote sensing image multi-target tracking device provided by the present application.

[0023] Figure 7 A structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described below in conjunction with the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0025] Remote sensing visual positioning tasks usually focus on locating a single target in a single remote sensing image that matches a specific natural language description. The emergence of multi-modal large models provides more possibilities for remote sensing image and language understanding. Some models have achieved multi-modal matching in static images. However, existing methods are still generally limited to static image tasks and cannot be extended to dynamic association and long-term tracking of multiple targets in remote sensing videos.

[0026] In particular, in remote sensing videos, high-value targets such as aircraft and ships often exist in multiple instances and present challenges such as continuous movement, similar appearance between targets, small size, and frequent occlusion. In this context, users often want to directly guide the model to focus on and track multiple targets that match the description by inputting a text description (such as “a large gray ship approaching the shore”) and avoid analyzing all targets one by one.

[0027] However, unlike natural scene videos, remote sensing videos have characteristics such as small target size, top-down perspective, complex background, high acquisition cost, and sparse semantic information, which bring multiple difficulties to model design and data annotation. First, obtaining high-quality remote sensing videos requires special satellite task scheduling, imaging planning, and ground reception, which is costly. Second, providing complete multi-target trajectory annotation for each frame of image and semantic matching information with natural language queries requires a lot of manpower and professional knowledge. In addition, the lack of automatic and low-cost large-scale annotation procedures for remote sensing videos further limits the development of this direction.

[0028] Therefore, there is an urgent need for a new method that can combine remote sensing images with text semantic information, support users to quickly and accurately detect and track multiple dynamic targets that meet the description in remote sensing videos through natural language instructions, reduce human intervention, improve task efficiency, and adapt to diverse practical application needs.

[0029] To solve the problems of low target screening efficiency, lack of semantic constraints, and high human intervention in the process of remote sensing multi-target tracking, the present application provides a text-guided remote sensing image multi-target tracking method and device, which can automatically detect and track specific aircraft or ship targets in remote sensing video images based on text descriptions of target appearance and motion trajectory.

[0030] The remote sensing video studied by the present application is a very low resolution, large scene remote sensing video containing high-value targets such as aircraft and ships taken by satellites outside the earth. The video tracking target studied by the present application is a moving target in the remote sensing video, such as an aircraft or a ship. Such targets generally have different appearance characteristics, such as color, size, and model; they are located at different positions in the video; and they also have different motion characteristics when moving, such as speed, trajectory shape, and motion direction. Targets with different characteristics have different tracking values, so the present application designs a text-guided remote sensing image multi-target tracking method for this scenario.

[0031] Referring to Figure 1 The text-guided remote sensing image multi-target tracking method includes the following steps: S110, obtaining a remote sensing image video and a text description of a target to be tracked.

[0032] An imaging device carried by a satellite, unmanned aerial vehicle, or other remote sensing platform collects image sequences of a target area at certain time intervals and spatial resolutions, forming a remote sensing image video. These videos cover a wealth of geographic information and can reflect changes in the target area over time, providing basic data for subsequent target tracking.

[0033] A user uses natural language to describe the target to be tracked in detail according to actual needs. The text description can include the category of the target (such as aircraft, ships, etc.), appearance characteristics (color, shape, size, etc.), spatial position characteristics (relative / absolute position relationship between the target and the background or other ground objects), motion characteristics (speed, acceleration, direction change, trajectory shape, etc.), and other related attribute information. For example, "a large gray ship approaching the port" or "a white jet aircraft accelerating for takeoff."

[0034] The text description provided by the user should be as accurate and complete as possible to enable the model to better understand the characteristics of the target. When describing the target category, use clear terms; for the description of appearance features and motion features, try to be specific and avoid vague expressions. For example, when describing the color of a vehicle, use specific color names such as "red" or "blue" instead of vague descriptions such as "bright color".

[0035] S120, input the remote sensing image video and the text description of the target to be tracked into a pre-trained text-guided remote sensing multi-target tracking model, and output multi-frame tracking results of the target conforming to the text description.

[0036] The acquired remote sensing image video and the text description of the target to be tracked are jointly input into the pre-trained text-guided remote sensing multi-target tracking model. This model has been trained on a large amount of data and can understand the correspondence between the text description and the target features in the remote sensing image, so as to accurately locate and track the target conforming to the text description in each frame of the video and output multi-frame tracking results. The tracking results include the bounding box coordinates of each target, target identification, category prediction, tracking confidence, etc., forming a complete text-guided target tracking bounding box sequence.

[0037] The text-guided remote sensing multi-target tracking model adopts a deep learning architecture, combining computer vision and natural language processing techniques. The model uses a convolutional neural network (such as ResNet, EfficientNet, etc.) to extract image features from remote sensing images, uses a pre-trained language model to extract text features, and uses a recurrent neural network or cross-modal attention network to fuse image features and text features to achieve target tracking. The fused features are used to predict the position of the target in each frame, and a tracking method based on the Transformer architecture can be used.

[0038] In order to enable the model to accurately track the target conforming to the text description, a large amount of labeled data needs to be used for training. The labeled data should include remote sensing image videos, corresponding text descriptions, and the true position information of the target in each frame. During training, a supervised learning method is used, a suitable loss function (such as mean square error loss, cross-entropy loss, etc.) is defined to measure the difference between the model's prediction results and the true labels, and the model's parameters are updated through the backpropagation algorithm to minimize the loss function.

[0039] The traditional remote sensing multi-target tracking method has high artificial screening cost and low efficiency when facing a large amount of data. The remote sensing image multi-target tracking method based on text guidance provided by the application directly inputs the remote sensing image video and the text description of the target to be tracked into the pre-trained model, without the need for artificial screening of the targets in a large amount of image data one by one, relying on automatic processing of the model, greatly improving the processing efficiency and reducing the cost, and being able to effectively deal with a large amount of remote sensing image data. The traditional remote sensing multi-target tracking method may need to manually set a large number of parameters and rules to adapt to different targets and scenes, which is tedious and inefficient. The application only needs the user to provide a simple text description, and the model can automatically complete the tracking task, greatly reducing the manual intervention and improving the tracking efficiency.

[0040] The traditional remote sensing multi-target tracking method is difficult to accurately identify and track targets in complex environments due to lack of flexibility and universality. The method of the application uses a pre-trained remote sensing multi-target tracking model based on text guidance. The text description can contain various semantic feature information of the target. The model can more accurately identify and distinguish the target meeting the text description in complex environments according to the rich semantic information, enhancing the flexibility and universality of the method, thereby overcoming the interference of complex environments.

[0041] In time-sensitive scenarios such as disaster response and emergency warning, the traditional remote sensing multi-target tracking method has low artificial screening efficiency and cannot provide results in time. The method automatically processes the model, quickly outputs multiple frames of tracking results of the target meeting the text description after inputting the data, greatly shortens the processing time, and can provide information in time for related scenarios, meeting the high real-time requirement. The traditional remote sensing multi-target tracking method has a large error rate in artificial screening, which is difficult to ensure accuracy. The remote sensing multi-target tracking model based on text guidance uses the accurate semantic information of the text description to identify and track the target, reduces the errors caused by human factors, can more accurately output the target tracking results meeting the requirements, and meets the high accuracy requirement.

[0042] Traditional remote sensing multi-target tracking methods mainly rely on visual features of targets, such as color, shape, etc. However, these features are easily disturbed in complex backgrounds, leading to inaccurate tracking. The method of the present application can utilize detailed text descriptions provided by users to accurately describe targets from multiple dimensions, enabling the model to more accurately identify and track targets and reducing the cases of false tracking and missed tracking. This method combines visual information of remote sensing images and semantic information of text descriptions, fully leveraging the advantages of both information sources. Visual information provides intuitive appearance and motion information of targets, while semantic information can understand the characteristics and attributes of targets from a higher level, complementing each other and improving the model's representation ability of targets, thereby improving the accuracy of tracking. Users can use natural language to flexibly describe the target to be tracked, without being limited by the target category and features. Whether it is a common target (such as an airplane or a ship) or a rare target (such as a specific aircraft or building), as long as it can be described in text, the model can attempt to track it. This flexibility enables the method to adapt to various tracking tasks and scenarios. Remote sensing images are affected by various factors such as weather, lighting, and shooting angle, and the appearance and motion state of targets may change greatly. The present application can provide stable semantic information of targets through text descriptions, enabling the model to be less disturbed by these environmental changes to a certain extent and maintaining good tracking performance. For example, even if the color of the target changes in the image due to changes in lighting, as long as other features mentioned in the text description (such as shape, category) remain unchanged, the model can still accurately track the target.

[0043] In some embodiments, referring to Figure 2 The text-guided remote sensing multi-target tracking model is trained by the following methods: S210, a plurality of remote sensing video samples containing a plurality of moving targets are obtained, and the remote sensing video samples are labeled to obtain a remote sensing target tracking dataset containing text descriptions.

[0044] Remote sensing video samples containing moving targets such as airplanes and ships are obtained by satellite, unmanned aerial vehicle, etc. The data source can be a public dataset or obtained by task scheduling satellite shooting. The video is cut into a continuous static image sequence at a frame rate of 24 frames per second (24FPS), providing basic data for subsequent labeling.

[0045] The moving targets (such as airplanes and ships) in each frame of image are labeled with bounding boxes using a target detection tool, recording their coordinates (x min , y min , x max , y max), and a unique identification ID. The appearance features (color, size, model) of the target, the spatial position features (relative / absolute position between the target and the background or other ground objects) and the motion features (speed, trajectory shape, etc.) are extracted by combining artificial and algorithmic methods.

[0046] Natural language descriptions are generated using large language models (such as GPT), such as "white jet accelerating for takeoff" or "large gray ship near the shore", ensuring that the descriptions are fluent and cover multiple dimensions. The text descriptions are bound to the time range (start frame / end frame) of the target in the video using specialized tools, with each text corresponding to at least one target identification and its appearance frame range. Artificially check the fluency of the text, the correspondence between the target and the description, and the continuity of the trajectory, revise ambiguous or ambiguous descriptions, and ensure the usability of the dataset.

[0047] S220, using the remote sensing target tracking dataset containing text descriptions to train a pre-established text-guided remote sensing multi-target tracking model.

[0048] The above-mentioned text-guided remote sensing multi-target tracking model includes a text encoder, an image encoder, a cross-modal information fusion module, a target tracking module, and a perceptible text memory module. The text encoder uses a pre-trained language model (such as BERT, RoBERTa) to convert the input text into a semantic feature vector, capturing the appearance, position, and motion information of the target. The image encoder uses a convolutional neural network structure such as ResNet to extract multi-scale features of the remote sensing image, with low-level features preserving small target details and high-level features providing global semantics. The cross-modal information fusion module fuses text features and image features through attention mechanisms or feature concatenation to generate a multi-modal feature map rich in semantics, focusing on semantically relevant areas. The target tracking module is based on the Transformer architecture, with multiple encoder-decoder layers for cross-frame target matching and position prediction, including a target association mechanism to maintain identity consistency. The perceptible text memory module introduces a memory bank to store historical query vectors, combining current text semantics through cross-attention mechanisms to enhance target temporal consistency and avoid drift or mismatch.

[0049] The constructed and labeled remote sensing target tracking dataset containing text descriptions is used to train the above-constructed text-guided remote sensing multi-target tracking model. During training, the model receives image sequences and text descriptions simultaneously, and the matched target trajectories and semantic relationships are used as supervision signals. The loss function during training is a weighted combination of detection loss and tracking loss. The detection loss includes target classification loss (cross-entropy loss) and bounding box regression loss. The regression loss can be in the form of IoU, GIoU, DIoU, CIoU, etc. The tracking loss further introduces a reference loss to guide the model to accurately identify and continuously track the target that meets the text description. The detection loss and tracking loss are weighted in proportion to balance the accuracy, consistency and semantic matching ability. The model parameters are optimized through gradient descent to make the model have the joint ability of semantic understanding, multi-target detection and time sequence association. The target classification loss measures the prediction accuracy of the model for target categories. The class prediction probability distribution (such as Softmax output) output by the model and the true class label are used to calculate the cross-entropy loss.

[0050] In the formula, represents the target classification loss, represents the number of samples (such as the number of targets in each frame). C represents the number of categories (such as two categories of aircraft and ship). represents the true class label (1 if target i belongs to category c, otherwise 0), represents the probability that target i predicted by the model belongs to category c.

[0051] The bounding box regression loss measures the positional deviation between the target bounding box predicted by the model and the true bounding box. Taking IoU as an example, , IoU(pred,gt) is the intersection over union of the target bounding box pred predicted by the model and the true bounding box gt, and L IoU is the bounding box regression loss.

[0052] The tracking loss measures the accuracy of the model in associating the same target across frames to avoid target ID switching or drifting. When calculating, the target query vector output by the model is combined with the query vector saved in the historical frame, and the text semantic features are combined.

[0053] In the formula, L ref represents the tracking loss. N represents the number of targets in the current frame. M represents the number of negative samples (different targets) in the historical frame. q i represents the current query vector of the i-th target in the current frame (output by the target tracking module). represents the query vector in the historical frame that is closest to q iQuery vectors of the same target (read from the memory module). sim(qi, qj) represents the similarity between qi and qj. i Query vectors of different targets (randomly sampled from the memory module). sim(.) represents the similarity function, such as cosine similarity. represents the temperature coefficient, controlling the sharpness of the similarity distribution (set to 0.1-0.5).

[0054] The present application proposes a complete set of data set construction process suitable for remote sensing multi-target semantic tracking, including target feature extraction, automatic generation of text description and target-text matching annotation, which improves the annotation efficiency and reduces the cost, and provides data support for large-scale remote sensing semantic tracking task. By annotating the remote sensing video samples and generating a data set containing text description, the model can deeply learn the characteristics of different moving targets. The text description can cover the appearance features, motion characteristics and other information of the target, so that the model can more accurately identify various targets in training, reduce misjudgment and omission. For example, when monitoring urban traffic, it can accurately distinguish vehicles of different types and driving directions. Text description helps the model understand the association between targets in different frames. By learning the description of target features and motion trends in the text, the model can better associate the same target in different time image frames, even in complex situations such as target occlusion and deformation, it can achieve accurate tracking, improve the continuity and stability of tracking. Obtain remote sensing video samples in multiple different scenes for annotation and training, so that the model can be exposed to rich environmental information, including different geographical areas, weather conditions, light intensity, etc. This enables the model to learn the characteristics and motion rules of targets in various scenes, so as to better adapt to different remote sensing monitoring scenes in practical applications, improve the generality and practicality of the model. The annotated remote sensing target tracking data set contains various types of moving targets, such as vehicles, ships, aircraft, etc., and their performance in different motion states. By learning these diverse target information, the model can better handle complex target tracking tasks and effectively track various rare or special targets.

[0055] During the training process, the model can extract key features related to the target based on the annotated text description. Compared with traditional unsupervised or weakly supervised learning methods, this supervised training method can avoid extracting a large number of irrelevant features, reduce the computational load and time cost of data processing, and improve the efficiency and accuracy of feature extraction. The remote sensing target tracking dataset containing text descriptions provides a more explicit learning goal and rich semantic information for the model. The model can quickly understand the characteristics and motion rules of the target based on the text description, thereby accelerating the parameter update and model convergence during the training process, shortening the training time, and improving the development efficiency. The interactive mode based on the text description allows users to easily intervene and correct the tracking results of the model. If the user finds that the tracking results of the model are inaccurate, the user can modify the text description or provide additional text information to guide the model to re-track, thereby enhancing the user's control ability and interactive experience of the model.

[0056] Existing remote sensing image multi-target tracking methods cannot filter or select targets based on user input natural language descriptions such as target color, trajectory, direction, etc. The semantic expression is disconnected from target selection, and the interactivity is poor. Moreover, existing remote sensing multi-modal methods are mostly used for text-guided target positioning in single images, and cannot be extended to multi-target dynamic identification and cross-frame tracking in remote sensing videos. Unlike existing methods that track first and then filter, the remote sensing multi-target tracking model based on text guidance of the present application directly integrates text description into the tracking process, realizes semantic-driven full-process from input to output, and improves processing efficiency and interactive intelligence. It supports cross-frame long-term tracking of multiple semantic targets in remote sensing videos, and adapts to complex motion scenarios such as rapid movement and occlusion of aircraft and ships. Current remote sensing multi-modal technologies are mostly limited to target positioning tasks in static images, while the present application extends to cross-frame semantic consistency modeling of multiple dynamic targets in remote sensing video sequences, realizing long-term stable tracking of complex moving targets such as aircraft and ships.

[0057] In some embodiments, efficient annotation is achieved by frame cutting + static, reducing the cost of manual annotation. A large language model is used to generate target text description and bind the time range, realizing the association of text-target-trajectory, and providing structured training data for subsequent multi-modal tracking models. Specifically, as shown in FIG. 3, a remote sensing video sample containing multiple moving targets is annotated in the following manner: Figure 3 S310, frame cutting the remote sensing video sample containing multiple moving targets at a preset frame rate to obtain a continuous sequence of static remote sensing images.

[0058] ​The remote sensing video sample is cut into continuous static image frames at a preset frame rate (such as 24 FPS per second). The frame rate can be adjusted according to the target motion speed. Similarity detection is performed on adjacent frames, highly repetitive frames are deleted, and the labeling amount is reduced. By converting dynamic video labeling into static image labeling, the complexity of manual operation is reduced. Key motion information can be retained while reducing data redundancy.

[0059] S320, using a target detection labeling tool, target labeling is performed on each frame of remote sensing image in the sequence of static remote sensing images.

[0060] Using a target detection labeling tool (such as LabelImg, CVAT), a rectangular frame is labeled for the target in each image, and attributes such as class (such as airplane, ship) and occlusion state are recorded. For each moving target (such as an airplane or a ship) in the image, its position in the image is labeled in the form of a bounding box, and its upper left and lower right coordinates (x min , y min , x max , y max ), class label (such as airplane, ship) and unique target identifier are recorded for subsequent tracking association.

[0061] S330, extracting appearance features, position features and motion features of the moving target.

[0062] Combining manual labeling and algorithm assistance, the key attribute information of each target in the time sequence of image instances is extracted. The attributes include appearance features (such as color, size, shape, model, etc., such as red, airplane), spatial position features (relative / absolute position relationship between the target and the background or other ground object targets, such as next to the terminal), and motion features (speed, acceleration, direction change, trajectory shape, etc., such as acceleration, straight line, turning, etc.). This step can extract features with the aid of image processing algorithms and motion pattern classification algorithms, and can also be supplemented and corrected by manual operation.

[0063] S340, inputting the appearance features, position features and motion features of the moving target into a large language model to output a target text description.

[0064] The extracted appearance features, position features, and motion features are taken as inputs to introduce a large language model (such as GPT, DeepSeek, etc. open source model) to generate natural language text description. The model is guided to automatically select appropriate attribute combinations and output smooth and accurate description sentences to represent the appearance and behavior characteristics of a specific target, such as "a large gray ship approaching the port" or "a white jet accelerating for takeoff." Multiple equivalent text expressions are generated for each type of target to improve diversity and model robustness. Semantic-rich text labels are generated, surpassing the information content of traditional category labels. Text guidance is provided for multi-modal tracking models to enhance their adaptability to complex scenes.

[0065] S350, load the remote sensing video sample, the target label, and the target text description, determine the starting frame and the ending frame of the target text description corresponding to the moving target in the remote sensing video sample, and complete the time consistency binding between the target text description and the moving target according to the starting frame and the ending frame; wherein each target text description corresponds to at least one or more target identifiers and their frame range.

[0066] Using a dedicated annotation tool, load the video image and the labeled target information, and input the generated text description. The starting frame and the ending frame of the target corresponding to the text description in the video are determined by manual operation, and the time consistency binding between the text and the specific target is completed by clicking the annotation. Finally, each text corresponds to at least one or more target identifiers and their frame range. Based on static annotation, use IOU or Hungarian algorithm to associate cross-frame targets to generate complete trajectories (starting frame → ending frame); for each trajectory, aggregate all target text descriptions in the covered frames (such as intersection or weighted fusion). For different frame text descriptions of the same target (such as "gray ship" and "large ship"), generate a unified description ("large gray ship") through LLM. If multiple trajectories match the same text description, determine the unique binding through feature similarity (such as cosine distance) or manual verification. Generate structured annotation results, with each text description corresponding to one or more target representations and their frame range (such as {"description": "large gray ship", "IDs": [1, 2], "frames": [10, 20]}); support training of end-to-end multi-modal tracking models to realize direct mapping from text query to target trajectory.

[0067] Perform manual quality inspection and text data specification verification on all labeled data to ensure that the text description is smooth and accurate, the target and description correspondence is correct, and the target trajectory annotation is coherent. Ambiguous, redundant, or ambiguous descriptions can be revised to ensure that the generated data set has good training usability and expandability.

[0068] The remote sensing video sample is cut according to the preset frame rate, so that the play can be paused, the previous frame or the next frame can be selected, the annotation result is facilitated to be managed and stored, and subsequent review and modification are facilitated. The appearance feature, position feature and motion feature of the moving target are extracted, and the features are input into a large language model to generate a target text description, so that the automation of part of the annotation work is realized. Compared with the traditional manual text description writing mode, this mode greatly saves time and labor cost. For example, for a large number of remote sensing video samples, if the manual text description writing mode is used, a large amount of time and effort is required, and through automatic feature extraction and text generation, a large amount of annotation work can be completed in a short time.

[0069] The target detection annotation tool is used to annotate each frame of image in the static remote sensing image sequence, so that the position and boundary of the moving target in the image can be accurately determined. On this basis, the appearance feature, position feature and motion feature of the moving target are extracted, and these detailed feature extraction provides a solid foundation for subsequent generation of accurate text description. The extracted moving target features are input into a large language model, and the powerful language generation and understanding ability of the large language model can output high-quality and accurate target text description. The large language model can comprehensively consider various feature information to generate natural, detailed and accurate description sentences, avoiding the problem of inaccurate or incomplete description that may occur in manual annotation. For example, for a ship on the sea, the large language model can generate an accurate description such as "a large gray ship is approaching the port" or "a white jet is accelerating for takeoff" according to the appearance feature and motion feature of the ship.

[0070] The starting frame and ending frame of the moving target in the remote sensing video sample corresponding to the target text description are determined, and the time consistency binding is completed, so that the target text description can be accurately corresponding to the moving target in the whole video time period. Each target text description corresponds to at least one or more target identifiers and their frame range, so that the annotation result has continuity and consistency in the time dimension, avoiding annotation errors caused by unclear time information. For example, when annotating a long video, it can be clearly known that a certain target starts to appear from which frame and ends at which frame, as well as the feature change in the whole process. The annotator only needs to click the target in the starting and ending frames, and the frames in between are also automatically annotated, without the need for frame-by-frame selection, saving annotation cost.

[0071] The annotation data obtained by the above annotation method contains detailed feature information and accurate text description of the moving target, as well as time consistency binding information, providing rich and diverse data for the training of a remote sensing multi-target tracking model based on text guidance. These data can help the model better learn the feature representation of the moving target, the correspondence between the text description and the target, and the change rule of the target in the time dimension, thereby improving the performance and generalization ability of the model. For example, the model can learn that the text description of moving targets with different appearance features corresponds to different ways of describing, and the change of target features at different time points affects the tracking results. The text description of the target provides rich semantic information for the model, enabling the model to understand the semantic meaning of the moving target. During the training process, the model can learn how to associate the text description with the target in the image, thereby enhancing the semantic understanding ability of the model and helping the model more accurately identify and track the target in a complex environment. For example, when multiple similar appearance targets appear in the video, the model can distinguish different targets according to the semantic information in the text description (such as the category, motion state, etc. of the target), thereby improving the accuracy of tracking.

[0072] In some embodiments, the remote sensing target tracking dataset includes remote sensing video samples, target annotations for each frame in the video, and target identification that meets the target text description in each frame of the video; wherein the target annotation at least contains bounding box coordinates, target identification and category label, and the target text description covers the appearance feature, position feature and motion feature of the target.

[0073] The target annotation in the dataset contains bounding box coordinates, target identification and category label, and the text description covers appearance, position and motion features. This enables the model to learn the features of the target from multiple dimensions during training. For example, through the bounding box coordinates, the model can learn the spatial position and size information of the target; the category label helps the model to distinguish different types of targets; the appearance feature description (such as color, shape, texture) enables the model to capture the visual characteristics of the target; the spatial position feature (such as the relative position in the image, the relationship with the surrounding environment) and the motion feature (such as the direction of motion, the speed) provide the model with dynamic information of the target. By integrating these information, the model can construct a more comprehensive and accurate target representation, thereby improving the accuracy of tracking. The rich training information enables the model to encounter target features in various situations. For example, the appearance of the target in different scenes may vary due to factors such as lighting, angle, etc., and the motion feature may also be different due to the intention of the target or the influence of the environment. By learning these diverse features, the model can better adapt to complex and variable remote sensing scenes in actual applications, improving the ability to handle unseen data, i.e. enhancing the generalization ability.

[0074] The data set contains both text descriptions and visual annotations, providing a foundation for multi-modal fusion training. The model can fuse text information (such as the appearance, location, and motion description of the target) with visual information (such as features extracted from the image), allowing for better understanding of the target. For example, color information mentioned in the text description can guide the model to focus more on areas with corresponding colors in the image, improving the accuracy of target detection and tracking. This multi-modal fusion training can fully leverage the strengths of both text and visual information, making up for the shortcomings of a single modality. During training, the model needs to learn to map text features and visual features into the same feature space, achieving cross-modal feature alignment. Through a large number of text-visual corresponding samples in the data set, the model can continuously adjust its parameters so that the text features and visual features of the same target are as close as possible in the feature space, while the features of different targets are far apart. This cross-modal feature alignment helps the model more accurately match the text description and the target in the image during tracking, improving the robustness of tracking.

[0075] The bounding box coordinates in target annotation provide precise location information of the target in the image for the model. During training, the model can learn how to accurately detect the location of the target based on these annotation information, reducing false positives and missed detections. For example, by optimizing the loss function, the model can minimize the difference between the predicted bounding box and the annotated bounding box, thereby improving the accuracy of target positioning. Class labels help the model learn the feature differences of different types of targets, enabling accurate classification of targets. In remote sensing images, there may be multiple types of targets, such as airplanes, ships, vehicles, etc. Through the class labels in the training data set, the model can learn the unique features of each type of target and accurately determine the class of the target during tracking, avoiding confusion between different classes of targets.

[0076] In some embodiments, referring to Figure 4 As shown, the text-guided remote sensing multi-target tracking model includes text encoders, image encoders, cross-modal information fusion modules, target tracking modules, and perceptible text memory modules, forming a complete end-to-end network architecture. The modules work together to achieve deep fusion of text and image information in spatial and temporal dimensions. The text encoder is used to convert natural language descriptions into computable semantic features, guiding the image encoder to focus on relevant targets and achieving text-visual dual driving. Through the cross-modal information fusion module and the perceptible text memory module, the continuity of target state in the time dimension is modeled; multi-frame tracking results containing target location, class, and confidence are directly generated, supporting open-vocabulary tracking (Open-Vocabulary Tracking).

[0077] As inputs, a remote sensing video and a text description of a target to be tracked are provided to a pre-constructed and trained text-guided remote sensing multi-target tracking model. The remote sensing video contains multiple consecutive images, recording the dynamic changes of multiple targets in space and time; the text description of the target to be tracked is presented in natural language, covering the category, appearance features (color, shape, etc.), motion features (speed, direction, etc.), and other key information of the target. The remote sensing video and the text description of the target to be tracked are input into the pre-trained text-guided remote sensing multi-target tracking model, and the output is a multi-frame tracking result of the target conforming to the text description, including: S1201, the text encoder converts the text description of the target to be tracked into a semantic feature vector, and outputs a text feature.

[0078] A pre-trained language model (such as BERT, RoBERTa, etc.) is used as a text encoder. The encoder receives the text description of the target to be tracked and uses the powerful language understanding capability of the pre-trained model to deeply analyze the text. Through a word embedding layer, each word or subword in the text is converted into a high-dimensional vector, and then a multi-layer Transformer structure is used for semantic encoding, finally converting the natural language into a fixed-dimensional semantic feature vector, i.e. the text feature. This text feature can capture the appearance, position, and motion of the target, and other high-value description information in the text, serving as the basic semantic input for subsequent multi-modal fusion and target discrimination. The generated text feature is output and passed to the cross-modal information fusion module for fusion with the image feature.

[0079] S1202, the image encoder extracts multi-scale image features of each frame in the remote sensing video.

[0080] A convolutional neural network such as ResNet is used as a fine-grained image encoder to process each frame of the remote sensing video. ResNet can effectively extract image features through the stacking of multiple residual blocks. Low-level feature maps undergo fewer convolution and pooling operations, preserving fine appearance information of small targets such as edges, textures, and other details; high-level feature maps undergo multiple convolutions and pooling, providing global context semantics such as the overall layout of the scene where the target is located. The combination of the two can enhance the representation ability of small targets and complex scenes. The image encoder encodes each frame of the image into multiple scale feature maps and outputs these feature maps, which are passed to the cross-modal information fusion module to provide visual basis for subsequent semantic alignment and fusion.

[0081] S1203, the cross-modal information fusion module fuses the text feature and the image feature to generate a multi-modal feature map containing both visual and semantic information.

[0082] The cross-modal information fusion module receives the text features output by the text encoder and the image features output by the image encoder. Various fusion methods are adopted, such as feature concatenation, which concatenates the text features and image features in the channel dimension to form a new feature vector; or using attention mechanisms, such as a cross-modal attention network (Transformer-based fusion), which calculates the attention weights between the text features and image features to perform weighted fusion. Point-by-point weighting, channel mapping, and other methods can also be used to achieve multi-modal alignment at the semantic level. After the above fusion operations, a joint feature map containing both visual and language information is generated. This multi-modal feature map not only contains the appearance and location information of the target in the image, but also incorporates the semantic constraints in the text description, which can guide subsequent detection and tracking operations to focus more on semantic-related regions.

[0083] Taking the feature concatenation-based fusion method as an example, the text features and multi-scale image features are concatenated in a specific dimension, thereby combining information from different modalities. Suppose the text features are encoded to obtain a vector of dimension D text , and the multi-scale image features are processed to obtain a set of feature maps, each with a dimension of C img ×H×W (where C img is the number of channels, H is the height, and W is the width). To perform concatenation, the text feature vector needs to be expanded to the same spatial dimension as the image feature maps. The text feature vector can be replicated to match the height and width dimensions of the image features Figure 1 , resulting in a feature tensor with a dimension of D text ×H×W. The expanded text feature tensor is concatenated with the multi-scale image feature maps in the channel dimension. For example, if the number of channels of the image feature maps is C img , the number of channels of the concatenated multi-modal feature maps will become C img +D text , resulting in a new feature map that contains both visual and semantic information. This fusion method is simple to implement, has a small computational load, and can preserve all the information of the original features.

[0084] Taking the fusion mode based on attention mechanism as an example, firstly, self-attention mechanism is applied to text features and multi-scale image features respectively to enhance their own expression ability, and then fusion is performed. Specifically, for text features, linear transformation is performed to map them into three matrices of query (Query), key (Key) and value (Value). Then the dot product of the query matrix and the key matrix is calculated to obtain an attention weight matrix, and normalization processing (such as using the Softmax function) is performed on the attention weight matrix. Finally, the normalized attention weight matrix is multiplied by the value matrix to obtain enhanced text features. For multi-scale image feature maps, linear transformation is also performed on them in the channel dimension to obtain query, key and value matrices. In the same way as the text features, self-attention is calculated to obtain enhanced image feature maps. The enhanced text features and image features can be fused by weighted summation or splicing. For example, the text feature vector is copied to the same spatial dimension as the image feature map, and then element-wise addition is performed with the image feature map to obtain a multi-modal feature map. This fusion mode can enhance the expression ability of the features themselves, capture the long-term dependencies within the features, and help improve the quality of the fused features.

[0085] Taking cross-attention mechanism fusion as an example, the multi-scale image feature map is taken as the query (Query), and the text feature is taken as the key (Key) and value (Value) after linear transformation. The dot product of the image query and the text key is calculated to obtain an attention weight matrix, and then normalization processing is performed. Finally, the normalized attention weight matrix is multiplied by the text value matrix to obtain image features enhanced based on text semantics. Or, the text feature is taken as the query, and the image feature map is taken as the key and value after linear transformation. Cross-attention is calculated in the same way as above to obtain text features enhanced based on image visual information. The image features enhanced based on text semantics and the original image features (or the text features enhanced based on image visual information and the original text features) are fused, for example, by weighted summation or splicing, to obtain the final multi-modal feature map. This fusion mode can effectively capture the semantic association between different modalities, realize information interaction and complementarity, and improve the understanding and expression ability of the fused features for the target.

[0086] For example, based on the Transformer-based fusion method, the text features and multi-scale image features are embedded and mapped to the same dimensional space. At the same time, position encoding is added to the text features and image features to preserve their spatial and temporal information. The embedded text features and image features are concatenated and input into the multi-modal encoder. The encoder consists of multiple self-attention layers and feedforward neural network layers, which interact and encode the concatenated features through the self-attention mechanism to capture the intrinsic relationship between different modal features. The multi-modal decoder receives the output of the encoder and performs cross-attention calculation with the text features (or image features). The decoder also consists of multiple cross-attention layers and feedforward neural network layers, which further fuse the text and image information through the cross-attention mechanism to generate a multi-modal feature map containing both visual and semantic information. The Transformer-based fusion method can be effectively trained on large-scale data and has strong feature extraction and fusion capabilities, which can handle complex multi-modal information.

[0087] In S1204, a target query vector of a historical frame is obtained, and the target tracking module generates a current query vector according to the multi-modal feature map and the target query vector of the historical frame; wherein the current query vector represents a candidate target state in the current frame.

[0088] The target query vector of the historical frame is obtained from the historical state of the model, such as the target query vector of the previous frame (T-1 frame). The target query vector is a representation of the target state in the historical frame, containing information such as the position and motion state of the target. A target tracking module based on the Transformer architecture is adopted, which receives the fused multi-modal feature map and the target query vector of the previous frame. A multi-layer Transformer encoder and decoder are used to model the consistency between the current frame and the historical target. The Transformer encoder encodes the multi-modal feature map to extract key information, and the decoder generates a new query vector, i.e. the query vector of the current frame (T frame), representing the candidate target state in the current frame, by combining the target query vector of the previous frame. At the same time, the module also contains a target matching mechanism, which calculates the similarity between the candidate target in the current frame and the target in the historical frame to associate the target identity across frames.

[0089] The following describes the process of generating a current query vector by the target tracking module based on the Transformer architecture, according to the multi-modal feature map and the target query vector of the historical frame: First, the historical query vector is processed using the self-attention mechanism: the target query vector of the historical frame is copied and expanded to have the same dimension as the multi-modal feature map ​The spatial dimensions of the multi-modal feature maps are matched. Assume that the multi-modal feature maps have a dimension of CxHxW (where C is the number of channels, H is the height, and W is the width), and the history query feature map is expanded into a tensor Q expanded with a dimension of CxHxW expanded . The self-attention mechanism is applied to Q expanded . Q enhanced is linearly transformed in the channel dimension to obtain query (Query), key (Key), and value (Value) matrices. The dot product of the query and the key is calculated to obtain a self-attention weight matrix, which is then normalized. Finally, the normalized self-attention weight matrix is multiplied by the value matrix to obtain an enhanced history query feature map Q enhanced .

[0090] The cross-attention mechanism is used to fuse the multi-modal features: the enhanced history query feature map Q fused is taken as the query (Query), and the multi-modal feature map is taken as the key (Key) and the value (Value). The dot product of the query and the key is calculated to obtain a cross-attention weight matrix, which is then normalized. The normalized cross-attention weight matrix is multiplied by the value matrix to obtain a query feature map Q fused enhanced based on multi-modal features.

[0091] Global average pooling is performed on Q t to compress the spatial dimension features into a vector to obtain a current query vector q t . This current query vector q t represents the state of the candidate target in the current frame, which integrates the target information of the history frames and the visual and semantic information of the current frame, and can be used for subsequent target position prediction and tracking.

[0092] In S1205, the memory module of the perceivable text fuses the current query vector with the target query vector of the history frame and the text feature through multi-layer cross-attention to output a semantic-enhanced query vector.

[0093] The memory module for text-awareness is constructed, including a historical query vector memory bank, a semantic enhancement decoding structure, a self-attention and cross-attention mechanism. The memory module receives the current query vector output by the target tracking module, and combines the target query vector in the historical frame (stored in the historical query vector memory bank) and the current text semantics (text features output by the text encoder) for information fusion. Through the self-attention mechanism, the current query vector can focus on different parts of itself to mine internal information; through the cross-attention mechanism, the current query vector can interact with the historical query information and the text semantics to capture the temporal evolution and semantic consistency of the target. After a series of attention mechanism processing, the semantic enhanced query vector is output. This semantic enhanced query vector further strengthens the guiding role of text information for target tracking, while considering the temporal dynamic changes of the target. The processed semantic query is also written into the memory bank for subsequent frame calling, so as to realize cross-frame continuous tracking of the target and maintenance of text consistency.

[0094] Reference Figure 5 As shown in the figure, the memory module for text-awareness includes a self-attention layer, a cross-attention layer, a feedforward neural network layer, a regular decoder layer and a memory module. The module performs multi-layer cross-attention fusion on the current query vector, the target query vector of the historical frame and the text features, and the process of outputting the semantic enhanced query vector is as follows: Firstly, the target tracking module outputs the current query vector, including Q (query), K (key) and V (value) three components. These vectors are intermediate representations generated after the target tracking module performs preliminary processing on the current frame image. The Q, K and V vectors are used as inputs of the self-attention layer to capture the preliminary context information of the target in the current frame.

[0095] The query vector enters the self-attention layer, and the internal relationship and features of the vector are captured through self-attention calculation. The self-attention mechanism calculates the similarity between Q, K and V, generates attention weights, and performs weighted summation on V to obtain a feature representation fused with internal information. The feature vector output after the self-attention layer processing (denoted as F self ) contains the preliminary context information of the target in the current frame. The Q component (denoted as Q self ) in F self interacts with the text features and the target query vector of the historical frame in the cross-attention layer.

[0096] The query vector output by the self-attention layer (denoted as Q self ) is first calculated with the historical frame target query vector (denoted as H query ) provided by the memory module in the cross-attention layer. By calculating the similarity between Q self and the K of H query , the historical information guided attention weight is generated, and Hquery The weighted sum is performed to realize information fusion in the time dimension, and a feature vector (denoted as F history ) fused with historical frame information is output, enhancing the consistency modeling of the target identity.

[0097] Subsequently, F history is taken as input and calculated with text features (denoted as T feat , providing K, V) in another cross-attention layer. By calculating the similarity of F history and the K of T feat , text-guided attention weights are generated, and the V of T feat is weighted and summed to realize the fusion of text semantics and visual features. The output is a text-enhanced visual feature (denoted as F text ), which focuses on the image region related to the text description.

[0098] F text serves as cross-modal information into a feedforward neural network, which further extracts and integrates features through nonlinear transformation. The feedforward neural network contains multiple fully connected layers and nonlinear activation functions (such as ReLU), which can learn complex feature mapping relationships and output higher-level abstract feature representations (denoted as F abstract ), containing richer semantic and visual information. F abstract serves as input to the regular decoder layer to generate the final query representation.

[0099] The decoder layer decodes the information (F abstract ) processed by the feedforward neural network. The output is a decoded feature representation (denoted as F decoded ). F decoded serves as the final query representation, i.e., the semantic-enhanced query vector, which is used to output multi-frame tracking results after the detection head.

[0100] The memory module continuously stores the target query vectors (H query ) of historical frames throughout the process, forming a historical query vector memory library. These historical information is used for cross-attention calculation in subsequent frames to realize cross-frame target association. The processed information (such as the semantic-enhanced query vector) is saved back to the memory module to update the historical information library. The dynamically updated memory module provides the target query vectors (H query ) of historical frames for subsequent frames, supporting cross-frame target association.

[0101] S1206, based on the semantic-enhanced query vector, generating a multi-frame tracking result of a target consistent with the text description; wherein the multi-frame tracking result at least includes the bounding box coordinates, target representation, class prediction and tracking confidence of each target.

[0102] Based on the semantic enhanced query vector output by the perceptible text based memory module, the model uses a detection head (such as a fully connected layer or a convolutional layer) to predict the target in the current frame. The prediction content includes the bounding box coordinates of each target, which is used to determine the position of the target in the image; target representation, which can be further used for target identification and classification; class prediction, which judges the category to which the target belongs; and tracking confidence, which represents the reliability of target tracking.

[0103] The above process is repeated for each frame in the remote sensing video, and finally a multi-frame tracking result of the target conforming to the text description is generated. The multi-frame tracking result can show the motion trajectory and state change of the target in the video, providing valuable information for remote sensing monitoring, target analysis and other applications.

[0104] The present application uses a text encoder to convert the text description of the target to be tracked into a semantic feature vector, outputting text features. The text encoder can deeply understand the text description of the target to be tracked and capture the rich semantic information therein. Key semantic elements such as color, target category, shape feature, etc. can be accurately extracted and converted into a high-dimensional semantic feature vector. This accurate semantic capture provides a solid foundation for the subsequent fusion of image features, enabling the model to track the target based on the text description. The generated semantic feature vector is universal and can effectively interact and match in different remote sensing scenarios and image feature spaces. No matter what background environment or shooting angle the target in the remote sensing image is in, the text features can serve as a unified semantic reference to help the model locate and identify the target conforming to the text description in complex visual information.

[0105] The size and detail level of the target in the remote sensing image vary greatly. The image encoder can capture target information at different levels by extracting multi-scale image features. For example, on a low-resolution feature map, the overall contour and approximate position of the target can be obtained; while on a high-resolution feature map, detailed features such as texture, edges, etc. of the target can be captured. This multi-scale feature extraction method enables the model to adapt to target tracking tasks of different sizes and complexities, improving the accuracy and robustness of tracking. The image encoder can convert the original remote sensing image data into compact and discriminative feature representations. These features not only retain important information in the image, but also eliminate redundant data, reducing computational complexity and storage requirements. At the same time, efficient feature representation helps subsequent cross-modal information fusion and processing of the target tracking module, improving the overall efficiency of the model.

[0106] By fusing text features with image features, the cross-modal information fusion module can introduce semantic information from text into visual features, enhancing the model's understanding of targets in images. For example, when color information of a target is mentioned in the text description, the fused multi-modal feature map can highlight the target area with that color more prominently, allowing the model to more accurately identify and distinguish targets during tracking. In remote sensing images, there may be targets with similar appearances, making it difficult to accurately distinguish them based solely on visual features. The cross-modal information fusion module uses additional semantic information provided by text features to effectively eliminate such semantic ambiguities.

[0107] The target tracking module achieves effective use of temporal information by obtaining the target query vector of the historical frame and generating the current query vector using the multi-modal feature map. The target query vector of the historical frame contains the motion state and feature information of the target in the previous frames, combined with the multi-modal feature map of the current frame, the model can predict the possible position and state of the target in the current frame. The use of this temporal information allows the model to better track the motion trajectory of the target, especially when the target moves quickly or is occluded, improving the continuity and accuracy of tracking. The current query vector, as a representation of the candidate target state in the current frame, has rich information content. It not only contains the visual and semantic features of the target, but also integrates the temporal motion information of the target. Based on the current query vector, the model can more accurately assess the matching degree of the candidate target and the target to be tracked, thereby selecting the target that best meets the requirements for tracking.

[0108] Through the perceptible text memory module, the perceptible text memory mechanism is introduced to enhance the target consistency modeling capability. By constructing a historical target query vector memory bank and combining text semantic features for semantic enhancement of the current frame target, the model can effectively associate occlusion, multiple instances, and cross-frame targets in complex remote sensing videos, significantly improving the robustness and identity consistency of small target tracking. The perceptible text memory module combines the current query vector (short-term memory) with the target query vector of the historical frame (long-term memory) and text features through multi-layer cross-attention fusion. This combination allows the model to fully utilize information from the historical tracking process while maintaining sensitivity to the current frame target state. For example, when tracking a long-lasting target, the memory module can continuously update and accumulate the target's feature information, improving tracking stability; when the target is temporarily occluded or changes appearance, the memory module can use historical information to help the model re-identify and track the target.

[0109] The multi-layer cross-attention fusion mechanism can highlight the most relevant feature information of the target to be tracked and suppress the interference of irrelevant information, thereby enhancing the discriminability of the query representation. The enhanced query representation can more accurately distinguish the target from the background, the target from other similar objects, and improve the accuracy and robustness of target tracking. Based on the enhanced query representation, the model can generate multi-frame tracking results of the target consistent with the text description; wherein the multi-frame tracking results at least include the bounding box coordinates of each target, target representation, class prediction and tracking confidence. Based on the enhanced query representation, the model can accurately predict the bounding box coordinates of each target in the current frame. The accurate prediction of the bounding box coordinates enables the model to accurately frame the position of the target, providing an accurate basis for subsequent target analysis and processing. The target representation in the multi-frame tracking results not only contains the visual features of the target, but also fuses semantic information and temporal information. Such comprehensive target representation helps to analyze and understand the target more deeply, such as target behavior recognition, state monitoring, etc. The model can accurately predict the class of each target according to the text description and image features. This is of great significance for target classification and identification in remote sensing images, and can help users quickly understand the composition and distribution of targets in the image. The tracking confidence reflects the reliability of the model's target tracking results. By providing tracking confidence, users can understand the accuracy of each target tracking result, and then make further processing and decision-making according to actual needs. For example, for targets with low tracking confidence, manual review or other auxiliary means can be used for confirmation.

[0110] In order to comprehensively evaluate the effect of the model in the actual remote sensing multi-target tracking task, a plurality of special indicators are used for performance evaluation, including HoTA (Higher Order Tracking Accuracy), DetA (Detection Accuracy) and AssA (Association Accuracy) and other multi-target tracking evaluation standards. These indicators can measure the overall performance of the model from multiple dimensions such as detection accuracy, trajectory continuity and target identity preservation. The evaluation process is based on the test set part of the constructed data set, focusing on verifying the accuracy, robustness and practical application reliability of the model in processing complex semantic descriptions, dealing with small target interference and maintaining trajectory continuity, etc.

[0111] The present application supports continuous tracking of multiple semantic targets in remote sensing videos. Current remote sensing multi-modal techniques are mostly limited to target positioning tasks in static images, while the present application extends to cross-frame semantic consistency modeling of multiple dynamic targets in remote sensing video sequences, enabling long-term stable tracking of complex moving targets (such as airplanes and ships). The present application introduces a perceptual text memory module to improve correlation robustness. To address the challenges of similar target appearance, small size, and easy occlusion in remote sensing videos, the present application effectively improves the model's ability to maintain target identity by fusing historical query vectors and text semantic information, avoiding tracking drift and mismatch.

[0112] The text-guided remote sensing image multi-target tracking device provided by the present application is described below. The text-guided remote sensing image multi-target tracking device described below can be referred to in conjunction with the text-guided remote sensing image multi-target tracking method described above.

[0113] The text-guided remote sensing image multi-target tracking device provided by the present application, as shown in Figure 6 , includes: The data acquisition module 410 is configured to acquire a remote sensing image video and a text description of a target to be tracked. The target tracking module 420 is configured to input the remote sensing image video and the text description of the target to be tracked into a pre-trained text-guided remote sensing multi-target tracking model, and output multi-frame tracking results of a target conforming to the text description.

[0114] Figure 7 An example of an entity structure diagram of an electronic device is shown in Figure 7 , which can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540. The processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can invoke logical instructions in the memory 530 to execute the text-guided remote sensing image multi-target tracking method.

[0115] In addition, the logic instructions in the memory 530 described above can be implemented in the form of software function units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0116] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the text-guided remote sensing image multi-target tracking method provided by the above-mentioned methods.

[0117] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the text-guided remote sensing image multi-target tracking method provided by the above-mentioned methods.

[0118] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0119] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary universal hardware platforms, and of course can also be implemented by hardware. Based on such understanding, the technical solutions described above essentially or the parts that contribute to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0120] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features therein can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for text-guided multi-object tracking in remote sensing images, characterized in that, The method comprises the following steps: acquiring a remote sensing video and a text description of a target to be tracked; inputting the remote sensing video and the text description of the target to be tracked into a pre-trained text-guided remote sensing multi-target tracking model, and outputting a multi-frame tracking result of a target conforming to the text description. 2.The text-guided remote sensing image multi-target tracking method of claim 1, wherein, The text-guided remote sensing multi-target tracking model is trained in the following manner: acquiring a plurality of remote sensing video samples containing a plurality of moving targets, and labeling the remote sensing video samples to obtain a remote sensing target tracking data set containing text descriptions; training a pre-established text-guided remote sensing multi-target tracking model using the remote sensing target tracking data set containing text descriptions. 3.The text-guided multi-target tracking method for remote sensing images based on the text guidance according to claim 2, characterized in that, The remote sensing video samples containing a plurality of moving targets are labeled in the following manner: frame the remote sensing video samples containing a plurality of moving targets at a preset frame rate to obtain a continuous sequence of static remote sensing images; use a target detection labeling tool to label each frame of the remote sensing image sequence; extract the appearance feature, position feature and motion feature of the moving target; input the appearance feature, position feature and motion feature of the moving target into a large language model to output a target text description; load the remote sensing video sample, the target label and the target text description, determine the starting frame and the ending frame of the target text description corresponding to the moving target in the remote sensing video sample, and complete the time consistency binding between the target text description and the moving target according to the starting frame and the ending frame; wherein each target text description corresponds to at least one or more target identifiers and their frame range of appearance. 4.The text-guided multi-target tracking method for remote sensing images based on the text guidance of claim 2, wherein, The remote sensing target tracking data set includes remote sensing video samples, target labels of each frame in the video, and target identifiers conforming to the target text description in each frame in the video; wherein the target label at least contains bounding box coordinates, target identifier and category label, and the target text description covers the appearance feature, position feature and motion feature of the target. 5.The text-guided multi-target tracking method for remote sensing images based on the text guidance of claim 1, wherein, The text-guided remote sensing multi-target tracking model comprises a text encoder, an image encoder, a cross-modal information fusion module, a target tracking module and a perceptible text memory module; The text encoder converts the text description of the target to be tracked into a semantic feature vector and outputs a text feature; The image encoder extracts multi-scale image features of each frame in the remote sensing video; The cross-modal information fusion module fuses the text feature and the image feature to generate a multi-modal feature map containing visual information and semantic information at the same time; acquire a target query vector of a historical frame, and the target tracking module generates a current query vector according to the multi-modal feature map and the target query vector of the historical frame; wherein the current query vector represents the state of a candidate target in the current frame; ​ The memory module of the perceptible text fuses the current query vector with the target query vector of the historical frame and the text feature through multi-layer cross attention, and outputs a semantic enhanced query vector; Based on the semantic enhanced query vector, a multi-frame tracking result of the target conforming to the text description is generated; wherein the multi-frame tracking result at least includes the bounding box coordinates of each target, target representation, class prediction and tracking confidence.

6. The text cue based multi-target tracking method of remote sensing images according to claim 5, characterized in that, The cross-modal information fusion module fuses the text feature and the image feature through feature splicing, attention mechanism or cross-modal attention network.

7. A text guide based remote sensing image multi-target tracking device, characterized in that, Comprise: A data acquisition module for acquiring a remote sensing video and a text description of a target to be tracked; A target tracking module for inputting the remote sensing video and the text description of the target to be tracked into a pre-trained remote sensing multi-target tracking model based on text guidance, and outputting a multi-frame tracking result of the target conforming to the text description.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to realize the remote sensing image multi-target tracking method based on text guidance as claimed in any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the remote sensing image multi-target tracking method based on text guidance as claimed in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the remote sensing image multi-target tracking method based on text guidance as claimed in any one of claims 1 to 6.

Images