Unmanned aerial vehicle multi-modal feature fusion target tracking method and system based on natural language description

Abstract

The application discloses a kind of unmanned aerial vehicle multimodal feature fusion target tracking method and system based on natural language description, belong to computer vision and image processing technical field, solve the problem that existing technology is prone to poor target tracking ability and long-time tracking ability when the image quality collected by unmanned aerial vehicle is poor or image feature is not obvious.The traffic accident scene in the image of the visual angle of the unmanned aerial vehicle is described in natural language, and the language prompt is obtained.A scene-context feature pyramid network is constructed to enhance the context information of the image from the visual angle of the unmanned aerial vehicle to obtain an image with enhanced features.The enhanced image and the language prompt are respectively visually encoded and language encoded to obtain visual features and language feature vectors for visual-language bimodal feature local alignment.The new language features and visual features obtained after alignment are fully fused to obtain multimodal features for target tracking.The application is used for unmanned aerial vehicle multimodal feature fusion target tracking.

Description

Technical Field

[0001] A method and system for UAV multimodal feature fusion target tracking based on natural language description is presented, which is used for UAV multimodal feature fusion target tracking and belongs to the field of computer vision and image processing technology. Background Technology

[0002] With the reduction in the production and application costs of drones, and the rapid development of technologies such as sensors, automation, and artificial intelligence, drones, equipped with cameras, GPS positioning systems, various hardware sensors, and mission payload systems, can now be used in fields such as coverage, tracking, countermeasures, and communication, efficiently completing various tasks that are difficult for humans to perform. Video single-target tracking is one of the fundamental tasks in computer vision, with wide applications in intelligent industries such as robot vision, video security, and sports video analysis. Drones equipped with target tracking algorithms can not only accelerate scene search time but also save significant manpower and resources. Currently, existing target tracking technologies rely solely on the initial visual information of the target, using its initial appearance features to locate moving targets. However, this form of target tracking introduces performance bottlenecks in tracking algorithms and limits the application scope of single-target tracking. On the one hand, the appearance features of moving targets contain rich texture information but only sparse target semantics, resulting in limited discrimination capabilities of single-target trackers. On the other hand, the appearance of a target changes continuously during long-term movement, differing from its initial appearance, making it impossible for single-target trackers to stably track moving targets for extended periods.

[0003] CN 202410369741.0 - A multimodal tracking method for unmanned aerial vehicles (UAVs) utilizes combined features from template images, search region images, and text. Based on a Transformer encoding layer, it performs feature extraction and modal interaction, and inputs features from a portion of the search region into a feedforward neural network for classification and regression. The final bounding box of the tracked target is calculated based on the obtained classification response map, offset, and scale. This method addresses the problem that current target tracking technologies cannot adapt to the rapid changes in the UAV's field of view and high field of view. This application can improve target tracking performance and optimize tracking effects. However, the following technical problems exist:

[0004] Poor image quality or unclear image features acquired by UAVs can easily lead to poor target acquisition and long-term tracking capabilities. This method proposes a contextual image feature enhancement module for UAV images to improve the quality of UAV images used in deep learning algorithms.

[0005] In traffic accident detection and recognition tasks, relying solely on visual target trackers such as DeepSORT is insufficient to capture diverse visual features. Furthermore, traffic accident data often exhibits a long-tailed distribution, and visual differences within the same category remain significant, making it difficult to summarize the different characteristics between accidents using only a single accident category. This paper introduces natural language descriptions, enabling the localization of the referenced target based on a given visual language reference, and uses multimodal enhancement of target features.

[0006] In existing tasks using drones to search for traffic accidents, only pre-trained image features can be used, and the search target cannot be modified or changed in real time according to different scenarios. After introducing natural language description, different search targets can be modified based on evidence. By providing different accident descriptions, the search and identification operation becomes more efficient and accurate.

[0007] Existing natural language tracking algorithms typically address the two tasks of visual grounding and tracking in two steps, deploying separate localization and tracking models to implement these two steps. This presents the following technical problems:

[0008] 1. The separate framework ignores the connection between visual localization and tracking—providing a global basis for localization in these two steps through natural language description. In other words, the separate framework treats visual localization and tracking as two independent processes, which leads to an information gap between the two. This gap hinders the system from continuously and accurately understanding the target object, resulting in problems such as insufficient long-term target tracking capability, poor adaptability to dynamic environments, and insufficient utilization of multimodal information.

[0009] 2. The separate framework makes end-to-end training difficult, resulting in a large number of parameters. Summary of the Invention

[0010] To address the problems mentioned above, the present invention aims to provide a target tracking method and system for UAVs based on natural language description and multimodal feature fusion, which solves the problem that existing technologies often result in poor target tracking and long-term tracking capabilities when the image quality of the UAV is poor or the image features are not obvious.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] A target tracking method for UAVs based on natural language description and multimodal feature fusion includes the following steps:

[0013] Step 1: Acquire traffic accident scene video from the perspective of a drone, convert the video into images from the perspective of a drone, annotate the images, and provide natural language descriptions of the traffic accident scene in the images to obtain language prompts;

[0014] Step 2: Construct a scene-context feature pyramid network to perform contextual information enhancement processing on the image from the UAV's perspective in Step 1 to obtain the feature-enhanced image;

[0015] Step 3: Perform visual encoding and language encoding on the enhanced image from Step 2 and the language prompts obtained from Step 1, respectively, to obtain visual features and language feature vectors;

[0016] Step 4: Process the visual features into visual feature vectors and perform local alignment of the visual-linguistic bimodal features with the corresponding linguistic feature vectors;

[0017] Step 5: Fully fuse the aligned new language features obtained in Step 4 with the visual features obtained in Step 3 to obtain multimodal features;

[0018] Step 6: If the image from the current UAV's perspective is the first frame, input the pre-trained historical features of traffic accidents into the decoder for decoding, and then use the localization head to obtain the final tracking result. At the same time, proceed to step 7. Otherwise, use the tracking result obtained in the previous frame as the historical feature and input it into the decoder along with the multimodal features of the previous frame for decoding, and then use the localization head to obtain the final tracking result. At the same time, proceed to step 7. The final tracking result includes the target bounding box.

[0019] Step 7: If the image from the current UAV's perspective is not the last frame, perform tracking processing on each frame based on the tracking results, perform region of interest pooling on the target region of the multimodal features according to the target bounding box to obtain the target region features, and then perform planarization processing on the target region features to obtain historical features. Then go to step 1 to perform tracking processing on the next frame; otherwise, end the tracking.

[0020] Furthermore, the scene-context feature pyramid network constructed in step 2 is a multi-scale feature extraction network based on the BiFPN framework. This network extracts multi-scale features through the backbone network ResNet50, and then uses the bidirectional feature fusion path of the BiFPN structure to obtain the feature-enhanced image.

[0021] A pre-trained ResNet50 is used as a feature extractor to extract multi-scale features from the input image. Four feature maps of different scales are obtained by sequentially setting four layers of ResNet50, namely 1 / 4, 1 / 8, 1 / 16 and 1 / 32. Then, a 1x1 convolution is used to adjust the number of channels of the four feature maps of different scales extracted by ResNet50 to a uniform number of channels of 512.

[0022] BiFPN includes top-down path feature fusion and bottom-up path feature fusion. The feature maps obtained from the top-down path feature fusion and bottom-up path feature fusion are added together to obtain the final feature map. The final feature map is then converted into the required number of output channels (3) through a 3x3 convolutional layer, and the feature-enhanced image is output.

[0023] BiFPN receives feature maps from ResNet50 at four different scales, arranged in descending order of scale into four levels: E1, E2, E3, and E4.

[0024] The top-down path feature fusion starts by upsampling the feature map of E4 to make it the same size as the feature map of E3. The upsampled feature map of E4 and the feature map of E3 are then weighted and fused using the ReLU activation function with learnable weights self.weights1. The fused feature map continues to be passed down and the same weighted fusion operation is performed on the feature maps of E2 and E1 in turn.

[0025] The bottom-up feature fusion process starts by downsampling the feature map of E1 to make it the same size as the feature map of E2. The downsampled feature map of E1 and the feature map of E2 are then weighted and fused using the ReLU activation function with learnable weights self.weights2. The fused feature map continues to be passed down and the same weighted fusion operation is performed on the feature maps of E3 and E4 in turn.

[0026] Furthermore, in step 3, the Swing Transformer encoder is used as a visual encoder to encode the enhanced image. The specific steps are as follows:

[0027] The visual encoder processes the enhanced image and retains the output feature maps of the first three layers of the Swin Transformer. It flattens each output feature map into a sequence of shape (B, H*W, C), flattens the output features using nn.Flatten, adjusts the feature dimensions using a linear layer, and adds the three feature maps to generate the final feature map, which is the visual feature. Here, B is the batch size, H and W are the height and width of the output feature map of the first three layers of the Swin Transformer, and C is the number of channels.

[0028] The natural language description is encoded using the BERT language transformation model as the language encoder. The specific steps are as follows:

[0029] Preprocessing:

[0030] For natural language descriptions, BERT's WordPiece segmenter is used to segment the words, resulting in a token sequence. A special token [CLS] is added to the beginning of the token sequence, and a special token [SEP] is added to the end of the token sequence. Then, the token sequence is converted into the corresponding token ID and mapped according to the vocabulary of the BERT model. The token sequence includes multiple tokens, and each token includes a query vector, a key vector, and a value vector.

[0031] Embedding layer processing:

[0032] In the embedding layer, the token ID is converted into a word embedding vector. Using the word embedding parameters pre-trained by BERT, each token ID is mapped to a fixed-dimensional word embedding vector. Then, a position embedding vector corresponding to its position is added to each token in the token sequence. The final input representation is the sum of the word embedding vector and the position embedding vector, which constitutes the initial feature vector fed into the encoder layer.

[0033] Encoding layer:

[0034] Each encoder layer includes a self-attention mechanism and a feedforward neural network. In the self-attention mechanism, after the embedding layer is processed, the attention score of each token with the remaining tokens in the token sequence is calculated using the query vector and key vector and normalized to a weight. The value vectors of the tokens are weighted and summed according to the weights to update the representation of each token, thereby capturing the long-distance dependencies in the token sequence. The feedforward neural network performs a non-linear transformation on the output of the self-attention mechanism.

[0035] The final output indicates:

[0036] Through the above steps, BERT outputs a language feature vector for each token in the token sequence after the encoding layer is processed.

[0037] Furthermore, the specific steps of step 4 are as follows:

[0038] First, the visual features obtained in step 3 retain batch and channel information, and the data is standardized by using the StandardScaler() function. The feature maps of different batches and channels are divided into different samples, and GMM clustering is performed. Four cluster centers are defined for each sample. The four cluster centers are obtained and converted into feature vectors. Through this step, several visual feature vectors are generated on different batches and channels.

[0039] Then, the visual feature vector and the language feature vector output from step 3 are mapped to a predefined common dimension through two independent fully connected layers;

[0040] After dimensional matching, for each pair of visual and linguistic feature vectors, their dot product is calculated to obtain a similarity score. These similarity scores are then combined into an attention matrix S of shape (num_visual_features, num_text_embeddings), where num_visual_features refers to the number of visual feature vectors and num_text_embeddings refers to the number of linguistic feature vectors. The attention matrix S represents the pairwise similarity score between all visual and linguistic feature vectors, and each value in the attention matrix reflects the correlation strength between the visual and linguistic feature vectors. Each row of the attention matrix represents a visual feature vector, and each column represents a linguistic feature vector.

[0041] The softmax function in the soft attention method is used to normalize each column of the attention matrix S to generate a weight distribution. Then, the language feature vectors are weighted and summed according to the weights to obtain a new language feature corresponding to the visual feature, that is, the local alignment of visual-language bimodal features is obtained.

[0042] Furthermore, the specific steps of step 5 are as follows:

[0043] Based on the local alignment of visual-linguistic bimodal features, the aligned new linguistic features are added to the original visual features generated in step 3 via residual connections, followed by layer normalization. A position-forward neural network is then used to perform a nonlinear transformation on the normalized features. This position-forward neural network consists of two linear layers and an activation function ReLU, with the following specific structure:

[0044] FFN(x) = Linear(ReLU(Linear(x))), where FFN represents a position-fed forward neural network, Linear represents a linear layer, ReLU represents an activation function, and x is the input data;

[0045] Based on the output of the nonlinear transformation, the output of the position nonlinear transformation is added to the features after the first residual processing through residual connection. Then, layer normalization is performed, and finally multimodal features are output, which is the fused features.

[0046] Furthermore, the decoder framework in step 6 is an improved transformer decoder, whose basic architecture includes a self-attention module, a residual connection layer, and a feedforward neural network layer, with a channel attention module added between the residual connection layer and the feedforward neural network layer.

[0047] The channel attention module includes an input layer that receives the feature map from the previous layer; a global average pooling layer that compresses the spatial information of each channel of the input feature map to generate channel description vectors; a first fully connected layer that reduces the number of output channels of the global average pooling layer to 1 / 16 of its original value; an activation function layer that linearly processes the output of the first fully connected layer; a second fully connected layer that restores the number of channels of the output of the activation function layer to its original value; a sigmoid function layer that generates channel weight vectors from the output of the second fully connected layer; and an output layer that multiplies the channel weight vectors with the input feature map to obtain the output feature map. The localization head is used to... The feature vector is mapped to the bounding box parameters of the target. That is, the localizer uses a 1x1 convolution kernel to reduce the number of channels in the target region to 1, generating a score map, and obtaining a probability score for whether each position is a corner point of the target. Based on the score map, the bounding box of the target is predicted, and four values ​​are output, corresponding to the center coordinates (x, v) and width and height (w, h) of the bounding box. The localizer parameters are updated through backpropagation by minimizing the difference between the bounding box and the ground truth box using the IoU loss function to obtain the final tracking result. The similarity measure of the score map includes dot product and cosine similarity. The final tracking result includes the target box, where the target represents the location of the traffic accident obtained in the entire traffic scene. Furthermore, in step 7, performing region-of-interest (ROI) pooling on the target region of the multimodal features of the image frame based on the target bounding box involves expanding the target bounding box by 1.5 bits within the multimodal feature target region, i.e., increasing the width and height of the target bounding box by a factor of 1.5. The adjusted target bounding box is then converted from center coordinates and width / height format to bounding box format. Finally, the ROI extraction method is used to extract the target region features from the feature map within the adjusted bounding box. The formula for the ROI extraction method is:

[0048]

[0049] RoIAlign is the layer that performs the RoI Align operation, used to calculate the pixel values ​​of floating-point coordinates through bilinear interpolation. "(6, 6)" represents the output size, meaning each RoI is sampled as a 6x6 feature map. `spatial_scale` represents the scaling factor used to adjust the RoI coordinates, i.e., the ratio of the original size of the feature map within the bounding box to the size of the feature map after the RoI Align operation. `self.divisor` represents the downsampling factor, and `sampling_ratio` represents the number of sampling points used within the RoI; setting it to 2 means sampling 2 points each in the height and width directions. A UAV multimodal feature fusion target tracking system based on natural language description includes:

[0050] Information Acquisition Module: Acquires video of traffic accident scenes from the perspective of a drone, converts the video into images from the perspective of a drone, annotates the images, provides natural language descriptions of the traffic accident scenes in the images, and obtains language prompts;

[0051] BiFPN Feature Pyramid Module: Constructs a scene-context feature pyramid network to perform contextual information enhancement processing on images from the UAV's perspective to obtain feature-enhanced images;

[0052] Encoding module: Performs visual encoding and language encoding on the enhanced image and language prompts respectively to obtain visual features and language feature vectors;

[0053] Multimodal feature alignment module: processes visual features into visual feature vectors and performs local alignment of visual-linguistic bimodal features with the corresponding linguistic feature vectors;

[0054] Multimodal feature fusion module: Fully fuses the aligned new language features with visual features to obtain multimodal features;

[0055] Decoder module: If the image from the current UAV's perspective is the first frame, the pre-trained decoder with historical features of traffic accidents is input for decoding; otherwise, the tracking results obtained in the previous frame are used as historical features and input with the multimodal features of the previous frame for decoding.

[0056] The positioning and tracking module is used to obtain the final tracking result by passing the decoded result through the positioning head. The final tracking result includes the target bounding box.

[0057] Historical feature embedding generation module: When the image from the current UAV perspective is not the last frame, after tracking processing of each frame based on the tracking results, the target region of the multimodal features is pooled according to the target box to obtain the target region features. Then, the target region features are planarized to obtain historical features.

[0058] Compared with the prior art, the beneficial effects of this invention are as follows:

[0059] I. This invention incorporates natural language descriptions into target tracking decoding. The descriptions provide high-level semantic information about the target object, including the object's state and its relationship with other objects. This global semantic information is very useful for locating targets in specific environments, especially in complex or ambiguous traffic accident scenarios. It improves long-term target tracking capability, dynamic environment adaptability, and fully utilizes multimodal information and target discrimination capability.

[0060] II. This invention combines visual language grounding (in visual language target tracking, the process of matching text descriptions with the actual positions of objects in images or video frames) with a tracking method to establish an end-to-end training framework. The same network is used to implement the two tasks of visual language grounding and tracking, making the network lightweight and reducing the number of parameters.

[0061] Third, this invention addresses the task of visual target tracking from the perspective of UAVs by proposing a scene-context feature pyramid network for images from the perspective of UAVs, thereby improving the quality of UAV images used in deep learning algorithms.

[0062] Fourth, this invention introduces rich semantic information from natural language description based on visual features, which can be used to improve the target discrimination ability and long-term tracking ability of target trackers, thereby improving the visual-language dual-modal single target tracking task;

[0063] V. In order to fully integrate visual and linguistic features, the present invention first aligns the visual and linguistic features, so that the generated new linguistic features have enhanced information of image features, thereby improving the ability of linguistic features to understand the current visual scene.

[0064] VI. This invention improves the decoder structure by allocating attention weights among different channels to emphasize channel features that are more important to the task, enabling it to perform better on traffic accident datasets.

[0065] VII. The UAV in this invention is equipped with a target tracking algorithm based on natural language description. It can change the UAV search task while switching natural language command descriptions, thereby improving the intelligence of UAV search and tracking and covering more tasks that are difficult for humans to complete.

[0066] 8. In the task of traffic accident identification, traffic accident data often exhibits a long-tail distribution, and the visual differences within the same category are still relatively large. It is difficult to summarize the different characteristics between accidents by relying on a single accident category. After introducing natural language description, the target reference can be located based on a given visual language reference. In the task of using UAVs to search for traffic accidents, it can provide pre-set relevant accident descriptions, making traffic accident identification more efficient and accurate. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the overall framework of the present invention;

[0068] Figure 2This is a schematic diagram of the BiFPN structure in this invention, where P3, P4, P5, and P6 are feature maps of different scales extracted from ResNet50, F4 and F5 are saved intermediate results, and D3, D4, D5, and D6 are the outputs of the four feature maps of different scales after a bidirectional fusion process. The arrows indicate the direction of data fusion.

[0069] Figure 3 This is a schematic diagram of the decoder architecture in this invention, where Add&Norm represents the residual connection layer, Feedforward represents the feedforward neural network layer, SEAttention represents the channel attention module, MHAttention represents the multi-head attention module, and Q represents the historical result feature embedding.

[0070] Figure 4 This is a schematic diagram of the structure of the Swin Transformer visual encoder in this invention for storing the feature maps of the first three layers;

[0071] Figure 5 This is a structural diagram of the Bert encoder module in this invention. Detailed Implementation

[0072] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0073] In existing technologies, targets are typically identified by visual observation of images captured by drones, a method that is time-consuming and labor-intensive. While equipping drones with target tracking algorithms can enable real-time tracking of all categories within the current drone-captured scene, it is difficult to convey the targets that people actually want to track.

[0074] The present invention provides a UAV multimodal feature fusion target tracking method based on natural language description, which can help UAVs complete highly targeted target tracking under the guidance of natural language.

[0075] A target tracking method for UAVs based on natural language description and multimodal feature fusion includes the following steps:

[0076] Step 1: Acquire traffic accident scene video from the perspective of a drone, convert the video into images from the perspective of a drone, annotate the images, and provide natural language descriptions of the traffic accident scene in the images to obtain language prompts;

[0077] Step 2: Construct a scene-context feature pyramid network to perform contextual information enhancement processing on the image from the UAV's perspective to obtain the feature-enhanced image;

[0078] In images captured from a drone's perspective, complex backgrounds, vertical viewpoints, and variations in target type and size make target tracking a challenging task. This work proposes a scene-context feature pyramid network for object detection, considering that object type is often closely related to its surrounding scene. This network aims to strengthen the relationship between the target and the scene, addressing the challenges posed by variations in target size.

[0079] The constructed scene-context feature pyramid network is a multi-scale feature extraction network based on the BiFPN framework. This network extracts multi-scale features through the backbone network ResNet50, and then uses the bidirectional feature fusion path of the BiFPN structure to obtain the feature-enhanced image.

[0080] A pre-trained ResNet50 is used as a feature extractor to extract multi-scale features from the input image. Four feature maps of different scales are obtained by sequentially setting four layers of ResNet50, namely 1 / 4, 1 / 8, 1 / 16 and 1 / 32. Then, a 1x1 convolution is used to adjust the number of channels of the four feature maps of different scales extracted by ResNet50 to a uniform number of channels of 512.

[0081] BiFPN includes top-down path feature fusion and bottom-up path feature fusion. The feature maps obtained from the top-down path feature fusion and bottom-up path feature fusion are added together to obtain the final feature map. The final feature map is then converted into the required number of output channels (3) through a 3x3 convolutional layer, and the feature-enhanced image is output.

[0082] BiFPN receives feature maps from ResNet50 at four different scales, arranged in descending order of scale into four levels: E1, E2, E3, and E4.

[0083] The top-down path feature fusion starts by upsampling the feature map of E4 to make it the same size as the feature map of E3. The upsampled feature map of E4 and the feature map of E3 are then weighted and fused using the ReLU activation function with learnable weights self.weights1. The fused feature map continues to be passed down and the same weighted fusion operation is performed on the feature maps of E2 and E1 in turn.

[0084] The bottom-up feature fusion process starts by downsampling the feature map of E1 to make it the same size as the feature map of E2. The downsampled feature map of E1 and the feature map of E2 are then weighted and fused using the ReLU activation function with learnable weights self.weights2. The fused feature map continues to be passed down and the same weighted fusion operation is performed on the feature maps of E3 and E4 in turn.

[0085] Step 3: Perform visual encoding and language encoding on the enhanced image from Step 2 and the language prompts obtained from Step 1, respectively, to obtain visual features and language feature vectors;

[0086] The enhanced image is encoded using the Swin Transformer encoder as the visual encoder. The specific steps are as follows:

[0087] The visual encoder processes the enhanced image and retains the output feature maps of the first three layers of the Swin Transformer. It flattens each output feature map into a sequence of shape (B, H*W, C), flattens the output features using nn.Flatten, adjusts the feature dimensions using a linear layer, and adds the three feature maps to generate the final feature map, which is the visual feature. Here, B is the batch size, H and W are the height and width of the output feature map of the first three layers of the Swin Transformer, and C is the number of channels.

[0088] The natural language description is encoded using the BERT language transformation model as the language encoder. The specific steps are as follows:

[0089] Preprocessing:

[0090] For natural language descriptions, BERT's WordPiece tokenizer is used for word segmentation, resulting in a token sequence. A special token [CLS] is added to the beginning of the token sequence, and a special token [SEP] is added to the end. Then, the token sequence is converted into corresponding token IDs and mapped according to the BERT model's vocabulary. The token sequence includes multiple tokens, each token consisting of a query vector, a key vector, and a value vector. For example, for the natural language description "A white car collided with a blue truck.", BERT's WordPiece tokenizer is used for word segmentation, resulting in the token sequence ["A", "white", "car", "collided", "with", "a", "blue", "truck", "."]. A special token [CLS] is added to the beginning of the sequence, and a special token [SEP] is added to the end. Then, the token sequence is converted into corresponding token IDs and mapped according to the BERT model's vocabulary.

[0091] Embedding layer processing:

[0092] In the embedding layer, token IDs are converted into word embedding vectors. Using the word embedding parameters pre-trained by BERT, each token ID is mapped to a fixed-dimensional word embedding vector. Then, a positional embedding vector corresponding to its position is added to each token in the token sequence. The final input representation is the sum of the word embedding vectors and the positional embedding vectors, forming the initial feature vector fed into the encoder layer. For example, in the example, the token ID corresponding to "white" is mapped to a 768-dimensional word embedding vector. Then, a vector related to its position is added to each token, enabling the model to capture the sequential information in the sequence.

[0093] Encoding layer:

[0094] Each encoder layer includes a self-attention mechanism and a feedforward neural network. In the self-attention mechanism, after processing by the embedding layer, the attention score of each token with the remaining tokens in the token sequence is calculated using the query vector and key vector, and normalized into weights. The value vectors of the tokens are then weighted and summed according to the weights to update the representation of each token, thereby capturing long-distance dependencies in the token sequence. The feedforward neural network performs a non-linear transformation on the output of the self-attention mechanism. For example, the "collided" token may generate higher attention weights for "white car" and "blue truck" because they are closely related to collision events. The feedforward neural network further enhances the expressive power of the model by performing a non-linear transformation on the output of the self-attention mechanism.

[0095] In the sentence "A car collided with a truck," each token (such as "car") is first transformed into three vectors—a query vector, a key vector, and a value vector—using the self-attention mechanism in the BERT encoder. Next, attention scores are calculated between this token and all other tokens in the sequence (including itself), and these scores are converted into weights using the Softmax function, ensuring they sum to 1 to form a probability distribution. Subsequently, these weights are used to perform a weighted summation of the token's value vectors, generating a new representation of that token. This new representation incorporates information from the entire sentence, capturing richer semantic relationships and long-range dependencies. The new representation of "car" integrates information from "collided," "with," "a," "truck," and itself through the above process, particularly emphasizing its association with "truck," thus better reflecting its meaning in the context.

[0096] The final output indicates:

[0097] Through the above steps, BERT outputs a language feature vector for each token in the token sequence after the encoding layer is processed.

[0098] Step 4: Process the visual features into visual feature vectors and perform local alignment between the visual and linguistic feature vectors; the specific steps are as follows:

[0099] First, the visual features obtained in step 3 retain batch and channel information, and the data is standardized by using the StandardScaler() function. The feature maps of different batches and channels are divided into different samples, and GMM clustering is performed. Four cluster centers are defined for each sample. The four cluster centers are obtained and converted into feature vectors. Through this step, several visual feature vectors are generated on different batches and channels.

[0100] Then, the visual feature vector and the language feature vector output from step 3 are mapped to a predefined common dimension through two independent fully connected layers;

[0101] After dimensional matching, for each pair of visual and linguistic feature vectors, their dot product is calculated to obtain a similarity score. These similarity scores are then combined into an attention matrix S of shape (num_visual_features, num_text_embeddings), where num_visual_features refers to the number of visual feature vectors and num_text_embeddings refers to the number of linguistic feature vectors. The attention matrix S represents the pairwise similarity score between all visual and linguistic feature vectors, and each value in the attention matrix reflects the correlation strength between the visual and linguistic feature vectors. Each row of the attention matrix represents a visual feature vector, and each column represents a linguistic feature vector.

[0102] The softmax function in the soft attention method is used to normalize each column of the attention matrix S to generate a weight distribution. Then, the language feature vectors are weighted and summed according to the weights to obtain a new language feature corresponding to the visual feature, that is, the local alignment of visual-language bimodal features is obtained.

[0103] Step 5: Fully fuse the aligned new language features obtained in Step 4 with the visual features obtained in Step 3 to obtain multimodal features; the specific steps are as follows:

[0104] Based on the local alignment of visual-linguistic bimodal features, the aligned new linguistic features are added to the original visual features generated in step 3 via residual connections, followed by layer normalization. A position-forward neural network is then used to perform a nonlinear transformation on the normalized features. This position-forward neural network consists of two linear layers and an activation function ReLU, with the following specific structure:

[0105] FFN(x) = Linear(ReLU(Linear(x))), where FFN represents a position-fed forward neural network, Linear represents a linear layer, ReLU represents an activation function, and x is the input data;

[0106] Based on the output of the nonlinear transformation, the output of the nonlinear transformation is added to the features after the first residual processing through residual connection, and then layer normalization is performed to finally output multimodal features, that is, the fused features.

[0107] Step 6: If the image from the current UAV's perspective is the first frame, input the pre-trained historical features of traffic accidents into the decoder for decoding, and then use the localization head to obtain the final tracking result. At the same time, proceed to step 7. Otherwise, use the tracking result obtained in the previous frame as the historical feature and input it into the decoder along with the multimodal features of the previous frame for decoding, and then use the localization head to obtain the final tracking result. At the same time, proceed to step 7. The final tracking result includes the target bounding box.

[0108] The decoder framework is an improved transformer decoder. The basic architecture includes a self-attention module, as well as residual connection layers and feedforward neural network layers. A channel attention module is added between the residual connection layers and the feedforward neural network layers.

[0109] The channel attention module includes an input layer that receives the feature map from the previous layer, a global average pooling layer that compresses the spatial information of each channel of the input feature map to generate channel description vectors, a first fully connected layer that reduces the number of output channels of the global average pooling layer to 1 / 16 of the original number, an activation function layer that linearly processes the output of the first fully connected layer, a second fully connected layer that restores the number of channels of the output of the activation function layer to the original number of channels, a sigmoid function layer that generates channel weight vectors from the output of the second fully connected layer, and an output layer that multiplies the channel weight vectors with the input feature map to obtain the output feature map.

[0110] Through a multi-layered attention mechanism, the decoder interacts with the query vector (in this paper, historical features) and the multimodal features output by the encoder to generate a target-specific feature vector corresponding to a possible target. This feature vector is then fed into the localizer for bounding box prediction.

[0111] The localizer is used to map the feature vectors obtained from the decoder to the bounding box parameters of the target. Specifically, the localizer uses a 1x1 convolutional kernel to reduce the number of channels in the target region to 1, generating a score map. It obtains a probability score for whether each location is a corner point of the target. Based on the score map, it predicts the bounding box of the target and outputs four values, corresponding to the center coordinates (x, y) and width and height (w, h) of the bounding box. The localizer parameters are updated through backpropagation by minimizing the difference between the bounding box and the ground truth box using the IoU loss function to obtain the final tracking result. The similarity measure of the score map includes dot product and cosine similarity. The final tracking result includes the target box, where the target represents the location of the traffic accident obtained in the entire traffic scene.

[0112] Step 7: If the image from the current UAV's perspective is not the last frame, perform tracking processing on each frame based on the tracking results, perform region of interest pooling on the target region of the multimodal features according to the target bounding box to obtain the target region features, and then perform planarization processing on the target region features to obtain historical features. Then go to step 1 to perform tracking processing on the next frame; otherwise, end the tracking.

[0113] Region of Interest (ROI) pooling is performed on the target region of the multimodal features of the image frame based on the target bounding box. This involves enlarging the target bounding box by 1.5 bits within the multimodal feature target region, i.e., increasing both its width and height by a factor of 1.5. The adjusted target bounding box is then converted from center coordinates and width / height format to bounding box format. Finally, the ROI extraction method is used to extract the target region features from the feature map within the adjusted bounding box. The formula for the ROI extraction method is as follows:

[0114]

[0115] RoIAlign is the layer that performs the RoI Align operation, which is used to calculate the pixel values ​​of floating-point coordinates through bilinear interpolation. "(6,6)" represents the output size, that is, each RoI is sampled into a 6x6 feature map. spatial_scale represents the scaling factor used to adjust the RoI coordinates, that is, the ratio of the original size of the feature map within the bounding box to the size of the feature map after the RoI Align operation. self.divisor represents the downsampling factor. sampling_ratio represents the number of sampling points used in the RoI. Setting it to 2 means sampling 2 points in both the height and width directions.

[0116] The above are merely representative embodiments among the many specific applications of this invention, and do not constitute any limitation on the scope of protection of this invention. All technical solutions formed by transformation or equivalent substitution fall within the scope of protection of this invention.

Claims

1. A method for unmanned aerial vehicle (UAV) multi-modal feature fusion target tracking based on natural language description, characterized in that, The method comprises the following steps: Step 1, obtaining a traffic accident scene video from the perspective of a UAV, converting the video into an image from the perspective of the UAV, labeling the image, and describing the traffic accident scene in the image in natural language to obtain a language prompt; Step 2, constructing a scene-context feature pyramid network to perform context information enhancement processing on the image from the perspective of the UAV in step 1 to obtain a feature-enhanced image; Step 3, performing visual coding and language coding on the enhanced image in step 2 and the language prompt obtained in step 1 respectively to obtain visual features and language feature vectors; Step 4, processing the visual features into visual feature vectors and performing visual-language bimodal feature local alignment with the corresponding language feature vectors; Step 5, fully fusing the new language features obtained in step 4 with the visual features obtained in step 3 to obtain multimodal features; Step 6, if the current image from the perspective of the UAV is the first frame, decoding the pre-trained historical feature input decoder with the traffic accident, then positioning the decoded result through a positioning head to obtain a final tracking result, and simultaneously proceeding to step 7, otherwise, taking the tracking result obtained in the last frame as the historical feature and inputting the historical feature and the multimodal features of the last frame into the decoder for decoding, then positioning the decoded result through the positioning head to obtain the final tracking result, and simultaneously proceeding to step 7, wherein the final tracking result comprises a target box; Step 7, when the current image from the perspective of the UAV is not the last frame, after tracking processing each frame based on the tracking result, performing region of interest pooling on the target region of the multimodal features according to the target box to obtain target region features, then performing planarization processing on the target region features to obtain historical features, and then proceeding to step 1 for tracking processing of the next frame, otherwise, ending the tracking. 2.The unmanned aerial vehicle multi-modal feature fusion target tracking method based on natural language description of claim 1, characterized in that, The scene-context feature pyramid network constructed in step 2 is a multiscale feature extraction network based on a BiFPN framework, which extracts multiscale features through a backbone network ResNet50, and then obtains a feature-enhanced image through a bidirectional feature fusion path of the BiFPN structure; A pre-trained ResNet50 is used as a feature extractor to extract multiscale features from an input image, and four feature maps of different scales are obtained through four layers of ResNet50 arranged in sequence, which are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 respectively, and then a 1x1 convolution is used to adjust the channel number of the four feature maps of different scales extracted through ResNet50 to a unified channel number of 512; The BiFPN includes a top-down path feature fusion and a bottom-up path feature fusion, and adds the feature maps obtained through the top-down path feature fusion and the bottom-up path feature fusion to obtain a final feature map, and then converts the final feature map into an output channel number of 3 through a 3x3 convolution layer to output a feature-enhanced image; The BiFPN receives four feature maps of different scales from ResNet50, and arranges them into E1, E2, E3 and E4 in order of decreasing scale; The top-down path feature fusion is the same in size as the feature map of E3 through an upsampling operation from the feature map of E4, and the feature map of E4 after upsampling is weighted and fused with the feature map of E3 using a learnable weight self.weights1 through a ReLU activation function, and the fused feature map continues to pass down and is sequentially weighted and fused with the feature maps of E2 and E1 in the same way; The bottom-up path feature fusion is the same in size as the feature map of E2 through a downsampling operation from the feature map of E1, and the feature map of E1 after downsampling is weighted and fused with the feature map of E2 using a learnable weight self.weights2 through a ReLU activation function, and the fused feature map continues to pass down and is sequentially weighted and fused with the feature maps of E3 and E4 in the same way. 3.The unmanned aerial vehicle multi-modal feature fusion target tracking method based on natural language description of claim 2, characterized in that, The step 3 adopts the encoder Swin Transformer as the visual encoder to encode the enhanced image, and the specific steps are as follows: The visual encoder processes the enhanced image and retains the output feature maps of the first three layers of the Swin Transformer, flattens each layer of the output feature map into a sequence with a shape of (B, H*W, C), uses nn.Flatten to flatten the output feature, adjusts the feature dimension using a linear layer, and adds the three layers of feature maps to generate a final feature map, which is the visual feature, wherein B is the batch size, H and W are the height and width of the feature map output by the first three layers of the Swin Transformer, and C is the channel number; The language conversion model BERT is used as the language encoder to encode the natural language description, and the specific steps are as follows: Preprocessing: The natural language description is segmented using the WordPiece tokenizer of BERT to obtain a token sequence, a special token [CLS] is added at the beginning of the token sequence, a special token [SEP] is added at the end of the token sequence, then the token sequence is converted into corresponding token ID according to the vocabulary table of the BERT model, the token sequence includes multiple tokens, and each token includes a query vector, a key vector and a value vector; Embedding layer processing: In the embedding layer, the token ID is converted into a word embedding vector, the word embedding parameter of the BERT pre-training is used to map each token ID into a fixed-dimensional word embedding vector, then a position embedding vector corresponding to the position of each token in the token sequence is added, and the final input representation is the sum of the word embedding vector and the position embedding vector, which constitutes the initial feature vector input into the encoder layer; Encoding layer: Each encoder layer includes a self-attention mechanism and a feed-forward neural network, in the self-attention mechanism, the attention scores of each token to the remaining tokens in the token sequence are calculated by the query vector and the key vector after the embedding layer processing, and are normalized into weights, and the value vector of the token is weighted and summed according to the weights, so that the representation of each token is updated, thereby capturing long-distance dependencies in the token sequence, and the feed-forward neural network performs nonlinear transformation on the output of the self-attention mechanism; Finally, the output representation is outputted: Through the above steps, BERT outputs a language feature vector for each token in the token sequence processed by the encoder layer.

4. The unmanned aerial vehicle multi-modal feature fusion target tracking method based on a natural language description according to claim 3, characterized in that, The specific steps of step 4 are: First, the visual feature batch and channel information obtained in step 3 are reserved, and the data is standardized by the StandardScaler() function, the feature maps of different batches and channels are divided into different samples, GMM clustering is performed, four cluster centers are defined for each sample, the four cluster centers are obtained and converted into feature vectors, and a plurality of visual feature vectors are generated on different batches and channels through this step; Then, the visual feature vectors and the language feature vectors outputted in step 3 are respectively mapped to a pre-defined common dimension through two independent fully connected layers; After the dimension matching, for each pair of visual feature vectors and language feature vectors, a similarity score is calculated by calculating the dot product, and the similarity scores are combined into an attention matrix S with a shape of (num_visual_features, num_text_embeddings), wherein num_visual_features refers to the number of visual feature vectors, num_text_embeddings refers to the number of language feature vectors, and the attention matrix S represents the two-by-two similarity scores between all visual feature vectors and all language feature vectors, each value in the attention matrix reflects the association strength between the visual feature vectors and the language feature vectors; wherein each row of the attention matrix represents a visual feature vector, and each column represents a language feature vector. The softmax function in the soft attention method is applied to normalize each column in the attention matrix S to generate a weight distribution, and then the language feature vectors are weighted and summed according to the weights to obtain a new language feature corresponding to the visual feature, that is, a visual-language dual-modal feature local alignment is obtained.

5. The unmanned aerial vehicle multi-modal feature fusion target tracking method based on natural language description according to claim 3, characterized in that, The specific steps of step 5 are: On the basis of the visual-language dual-modal feature local alignment, the aligned new language feature is added to the original visual feature generated in step 3 through a residual connection, then layer normalization is performed, and a position feed-forward neural network is used to perform nonlinear transformation on the feature after the layer normalization, wherein the position feed-forward neural network includes two linear layers and an activation function ReLU, and the specific structure is: FFN(x) = Linear(ReLU(Linear(x))), wherein FFN represents a position feed forward neural network, Linear represents a linear layer, ReLU represents an activation function, and x is input data; The output of the position non-linear transformation is added to the features after the first residual processing through a residual connection again, and then layer normalization is performed, and finally the multi-modal features are output, that is, the fusion features are obtained. 6.The unmanned aerial vehicle multi-modal feature fusion target tracking method based on natural language description of claim 3, characterized in that, The framework of the decoder in step 6 is an improved transformer decoder, and the basic architecture includes a self-attention module, a residual connection layer and a feed forward neural network layer, and a channel attention module is added between the residual connection layer and the feed forward neural network layer. The channel attention module includes an input layer receiving the feature map transmitted from the previous layer, a global average pooling layer compressing the spatial information of each channel of the feature map input by the input layer to generate a channel description vector, a first fully connected layer reducing the output channel number of the global average pooling layer to 1 / 16 of the original, an activation function layer performing linear processing on the output of the first fully connected layer, a second fully connected layer restoring the channel of the output of the activation function layer to the original channel number, a sigmoid function layer generating a channel weight vector from the output of the second fully connected layer, and an output layer multiplying the channel weight vector with the feature map input by the input layer to obtain an output feature map; the positioning head is used to map the feature vector obtained by the decoder into the bounding box parameters of the target, that is, the positioning head uses a 1x1 convolution kernel to reduce the channel number of the target region to 1 to generate a score map, obtain a probability score for each position whether it is a target corner, and predict the bounding box of the target based on the score map, which will output four values corresponding to the center coordinates (x, y) and the width and height (w, h) of the bounding box. The difference between the bounding box and the real box is minimized by using the IoU loss function to update the parameters of the positioning head through back propagation to obtain the final tracking result, wherein the similarity measure of the score map includes dot product and cosine similarity, and the final tracking result includes the target box, wherein the target represents the traffic accident position obtained in the entire traffic scene.

7. The unmanned aerial vehicle multi-modal feature fusion target tracking method based on a natural language description according to claim 6, characterized in that, In step 7, the target region of the multi-modal features of the current frame image is region of interest pooling according to the target box, which is to expand the target box in the target region of the multi-modal features by 1.5, that is, the width and height of the target box are expanded by 1.5 times, and the adjusted target box is converted from the center coordinates and the width and height format to the bounding box format, and then the ROI extraction method is used to extract the target region features from the feature map within the adjusted bounding box, wherein the formula of the ROI extraction method is: wherein, is a layer performing operation for calculating pixel values of a float coordinate by bilinear interpolation, "(6, 6)" represents the size of the output, i.e., each feature map is sampled into 6x6, represents a scale factor for adjusting coordinates, i.e., the original size of the feature map within the bounding box and the scale of the feature map size after the operation, represents a down-sampling multiple, represents the number of sampling points used within , and setting it to 2 means sampling 2 points in the height and width directions, respectively.

8. A natural language description based unmanned aerial vehicle multi-modal feature fusion target tracking system, characterized in that, It includes: An information acquisition module: acquiring a traffic accident scene video from the perspective of a UAV, converting the video into an image from the perspective of the UAV, labeling the image, and describing the traffic accident scene in the image in natural language to obtain a language prompt; A BiFPN feature pyramid module: constructing a scene-context feature pyramid network to perform context information enhancement processing on the image from the perspective of the UAV to obtain a feature-enhanced image; An encoding module: the enhanced image and the language prompt are respectively visually encoded and language encoded to obtain visual features and language feature vectors; A multi-modal feature alignment module: the visual features are processed into visual feature vectors and locally aligned with the corresponding language feature vectors in visual-language dual modal features; A multi-modal feature fusion module: the new language features and the visual features after alignment are fully fused to obtain multi-modal features; A decoder module: if the current image of the UAV view is the first frame, the pre-trained decoder with the historical features of the traffic accident is decoded, otherwise, the tracking result obtained in the last frame is taken as the historical features and input into the decoder with the multi-modal features of the last frame; A positioning tracking module: used to obtain the final tracking result through the positioning head after decoding, wherein the final tracking result includes a target frame; A historical feature embedding generation module: when the current image of the UAV view is not the last frame, after tracking processing of each frame based on the tracking result, the target region of the multi-modal features is region of interest pooled according to the target frame to obtain target region features, and the target region features are planarized to obtain historical features.

Images