Unmanned aerial vehicle aerial image long-tail target detection method based on multi-modal feature fusion and feature enhancement

Abstract

The invention discloses an unmanned aerial vehicle aerial image long-tail target detection method based on multi-modal feature fusion and feature enhancement, and relates to the technical field of target detection. Performing feature extraction and weighted fusion in a semantic dimension and a visual dimension based on a multi-modal feature fusion module to obtain a fusion enhanced feature vector; after coding is carried out through a position coding module, adding with the fusion enhanced feature vector to carry out global feature analysis, and obtaining a coding enhanced feature vector; constructing a target query set through the density enhanced feature vector, the target query quantity and the guide vector; a decoding module performs step-by-step decoding on the coding enhancement feature vector according to the target query set based on a multi-layer interactive attention mechanism to obtain a target detection result; and the classification performance of the decoding module is optimized through the classification loss function module. According to the invention, the long-tail target can be detected under a complex background and noise interference.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, and in particular relates to a method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement. Background Technology

[0002] Unmanned aerial vehicles (UAVs) are flight platforms that are either manually controlled or autonomously navigated, offering advantages such as low cost, high maneuverability, and flexible payload. In recent years, with the development of artificial intelligence, sensor technology, and integrated chips, the intelligence level of UAVs has continued to improve, demonstrating wide application value in fields such as military reconnaissance, environmental monitoring, agricultural operations, power line inspection, and emergency rescue.

[0003] In drone-based visual perception tasks, targets of interest are identified and located by analyzing images or videos captured by the drone, and the corresponding category and location are output. Compared to traditional ground-based monitoring, drones offer advantages such as flexibility, efficiency, and wide coverage, significantly improving operational efficiency and reducing labor costs.

[0004] However, drone aerial images often suffer from significant class imbalance, exhibiting a long-tailed distribution, which severely impacts the performance of drone target detection, especially affecting the recognition accuracy and robustness of long-tailed targets.

[0005] To address the aforementioned issues, existing technologies generally employ the following methods: resampling methods, which balance data distribution by oversampling tail classes or downsampling head classes, increasing the weight of tail class samples during training. However, this method may lead to overfitting of tail class samples or reduce the model's generalization ability by discarding head class samples; reweighting methods, which assign greater weights to tail class samples or hard samples in the loss function, can improve the classifier's learning ability but struggle to address the problem of insufficient representation of tail class targets in the feature embedding space; multi-stage learning methods, which gradually strengthen the learning of tail class samples through staged training, can alleviate insufficient tail class learning, but training is complex and inefficient; transfer learning methods, which alleviate the problem of data imbalance by transferring knowledge learned from the majority class to the tail class, but their effectiveness depends on the model structure and mechanism, making implementation difficult; and ensemble methods, which are based on ensemble strategies that aggregate multiple expert models, but their computational cost is high and they still have limitations in addressing the problem of insufficient tail class feature representation.

[0006] In summary, existing methods for long-tail target detection suffer from several drawbacks. The scarcity of long-tail category samples leads to insufficient feature extraction and dispersed feature representation, resulting in weak recognition capabilities and low detection accuracy. Furthermore, under complex backgrounds and noise interference, feature distribution is easily discretized, causing long-tail targets to overlap with similar categories in the feature space, increasing the risk of confusion and false detection. Existing loss functions tend to focus on head-class target detection and lack constraints on the boundaries between similar categories, further impacting the accuracy of long-tail target detection. Summary of the Invention

[0007] To address some or all of the technical problems existing in the prior art, this invention provides a method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement.

[0008] The technical solution of the present invention is as follows: A method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement is provided, including: Acquire drone aerial images; The feature extraction module performs multi-level, multi-scale modeling of drone aerial images to obtain feature maps at different levels. The multimodal feature fusion module extracts semantic feature vectors from the feature map through a text encoder and optimizes them to obtain semantic feature optimized vectors; it also extracts visual feature vectors from the feature map and optimizes them to obtain visual feature optimized vectors; and then performs weighted fusion and feature enhancement on the semantic feature optimized vectors and visual feature optimized vectors to obtain fused enhanced feature vectors. The location encoding module encodes the position of the pixel in the feature map and adds it to the fused enhanced feature vector to obtain the location enhanced feature vector; Global feature analysis is performed on the location-enhanced feature vector based on a multi-layer self-attention mechanism and a feedforward network to obtain the encoded enhanced feature vector. The encoded enhanced feature vector is inversely transformed to generate a density image, and the number of target queries is adaptively set according to the density regions in the density image. Align the density image with the encoded enhanced feature vector, and perform feature enhancement based on the attention mechanism to obtain the density-enhanced feature vector; Sample images from the training dataset are acquired, features are extracted from the sample images, and the directional similarity with similar vectors is analyzed. Gaussian noise is introduced and feature fusion is performed to obtain perturbation information. The perturbation information is added to the mean of the visual vector of the tail sample to obtain the updated visual vector of the tail sample. The feature perturbation network is trained based on the sample images and the updated visual vector of the tail sample. The feature perturbation network enhances the feature vector according to the density and generates the corresponding guiding vector. The dynamic query module generates query content and query location information based on density-enhanced feature vectors, target query quantity, and guiding vectors, and constructs a target query set. The decoding module uses a multi-layer interactive attention mechanism to progressively decode the encoded enhanced feature vector based on the target query set to obtain the target detection result. The classification performance of the decoding module is optimized by constructing a classification loss function based on a category instance adaptation mechanism and a semantic visual boundary constraint mechanism.

[0009] Further, in some embodiments, the steps of: extracting semantic feature vectors from the feature map using a text encoder based on a multimodal feature fusion module and optimizing them to obtain semantic feature optimized vectors; extracting visual feature vectors from the feature map and optimizing them to obtain visual feature optimized vectors; and performing weighted fusion and feature enhancement on the semantic feature optimized vectors and visual feature optimized vectors to obtain fused enhanced feature vectors; including: Based on the text encoder, semantic features are extracted from the feature map to obtain multiple sets of semantic feature vectors. Then, the multiple sets of semantic feature vectors are averaged, normalized, and linearly projected to obtain the semantic feature optimized vector. Visual features are extracted from the feature map to obtain visual feature vectors. The visual feature vectors are then processed by average pooling, multilayer perceptual projection, and normalization to obtain optimized visual feature vectors. A gating weighting mechanism is used to dynamically allocate the weights of the semantic feature optimization vector and the visual feature optimization vector for feature fusion, thereby obtaining a fused feature vector; By adjusting the perturbation amplitude of the features through a scaling factor, the fused feature vector is enhanced to obtain the fused enhanced feature vector.

[0010] Further, in some embodiments, the step of: encoding the position of pixels in the feature map using a position encoding module and adding it to the fused enhanced feature vector to obtain a position enhanced feature vector; includes: The position of pixels in the feature map is encoded by the position encoding module to obtain the position encoding vector; The location encoding vector is added to the fused enhanced feature vector to obtain the location enhanced feature vector.

[0011] Further, in some embodiments, the step of: performing an inverse transform on the encoded enhanced feature vector to generate a density image, and adaptively setting the number of target queries based on the density regions in the density image; includes: The encoded enhanced feature vector is restored to a two-dimensional feature map by deflating, and the two-dimensional feature map is input into the density generation network for prediction to obtain a density image. The target query count is adaptively set based on the density regions in the density image.

[0012] Further, in some embodiments, the step of aligning the density image with the encoded enhanced feature vector and performing feature enhancement based on an attention mechanism to obtain a density-enhanced feature vector includes: Align the density image with the encoded augmented feature vector to obtain the target distribution in the encoded augmented feature vector; Based on the spatial attention mechanism, feature enhancement processing is performed on the dense target region in the encoded enhanced feature vector to obtain the spatial attention vector; Based on the channel attention mechanism, feature enhancement processing is performed on the target-related region in the encoded enhanced feature vector to obtain the channel attention vector; Density-enhanced feature vectors are obtained by weighted fusion of spatial attention vectors and channel attention vectors on the encoded enhanced feature vectors.

[0013] Further, in some embodiments, the steps of: acquiring sample images from the training dataset, extracting features from the sample images, analyzing the directional similarity with similar vectors, introducing Gaussian noise and performing feature fusion to obtain perturbation information, adding the perturbation information to the mean of the tail sample visual vector to obtain an updated tail sample visual vector, training the feature perturbation network based on the sample images and the updated tail sample visual vector; and generating corresponding guiding vectors through the feature perturbation network based on density-enhanced feature vectors; including: Obtain sample images from the training dataset and statistically analyze the annotation information corresponding to the sample images; Based on the annotation information, the sample images are divided into head sample images, mid-frequency sample images, and tail sample images; Extract the semantic vector of the tail sample from the tail sample image using a text editor; Obtain the semantic vector of the comparison sample that is similar to the semantic vector of the tail sample, and calculate the semantic direction vector of the tail sample; ; in, This is the semantic direction vector for the tail. This is the semantic vector of the tail sample. The semantic vectors of the comparison samples are similar to those of the tail samples. This indicates normalization processing; Extract the visual vector of the tail sample from the tail sample image; Obtain visual vectors of contrasting samples that are similar to the visual vectors of the tail samples, and calculate the visual direction vector of the tail. ; in, The visual direction vector for the tail. The mean of the visual vectors of the tail samples. The mean visual vector of the comparison sample that is similar to the visual vector of the tail sample; Gaussian noise is projected onto the tail semantic direction vector and the tail visual direction vector respectively to generate the noise tail semantic vector and the noise tail visual vector. ; ; in, It is Gaussian noise. , This is the semantic vector of the noise tail. The visual vector for the noise tail; The tail semantic direction vector, tail visual direction vector, noisy tail semantic vector, and noisy tail visual vector are weighted and fused to obtain perturbation information. ; in, For perturbation information, As a balance factor, This is the semantic perturbation intensity factor. The visual disturbance intensity factor. These are the semantic feature weight parameters. For visual feature weight parameters; The perturbation information is injected into the mean of the visual vector of the tail sample to obtain the updated visual vector of the tail sample, which is used as the guiding vector. ; in, To update the visual vector of the tail sample.

[0014] Furthermore, in some implementations, the step of constructing a classification loss function based on a category instance adaptation mechanism and a semantic visual boundary constraint mechanism to optimize the classification performance of the decoding module includes: Construct category discrimination boundaries; ; in, For the goal Class and Target Class category discrimination boundary, For the goal Class and Target The semantic similarity matrix of classes For the goal Class and Target Visual similarity matrix of classes For semantic similarity weights, For visual similarity weights; Adaptively adjust the category discrimination boundary based on the instance to obtain the category instance boundary; ; in, For the goal Class and Target Class instance boundaries, This is the instance difficulty factor; Boundary correction is performed on the detection results of samples that are not true categories to obtain corrected sample detection results; ; in, To correct the sample test results, For sample test results; The cross-entropy loss function is used to adjust the class instance boundaries based on the corrected sample detection results, so as to optimize the classification performance of the decoding module.

[0015] The main advantages of the technical solution of this invention are as follows: This invention provides a method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement. Through a feature extraction module and a multimodal feature fusion module, it achieves multi-feature representation of long-tailed targets, solving the problems of scarce long-tail category samples and insufficient feature extraction in traditional technologies. Furthermore, by constructing a target query set, it improves the detection performance of the decoding module for long-tailed targets. The classification loss function constructed based on the category instance adaptation mechanism and the semantic visual boundary constraint mechanism reduces the risk of recognition confusion and false detection by expanding the category boundary between the feature space where the long-tailed target is located and similar categories. This further improves the detection accuracy and robustness of long-tailed targets under complex backgrounds and noise interference, achieving end-to-end detection of long-tailed targets. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and constitute a part of this invention, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement, as provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0018] The technical solutions provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0019] refer to Figure 1 This invention provides a method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement, including: Acquire drone aerial images; The feature extraction module performs multi-level, multi-scale modeling of drone aerial images to obtain feature maps at different levels. The multimodal feature fusion module extracts semantic feature vectors from the feature map through a text encoder and optimizes them to obtain semantic feature optimized vectors; it also extracts visual feature vectors from the feature map and optimizes them to obtain visual feature optimized vectors; and then performs weighted fusion and feature enhancement on the semantic feature optimized vectors and visual feature optimized vectors to obtain fused enhanced feature vectors. The location encoding module encodes the position of the pixel in the feature map and adds it to the fused enhanced feature vector to obtain the location enhanced feature vector; Global feature analysis is performed on the location-enhanced feature vector based on a multi-layer self-attention mechanism and a feedforward network to obtain the encoded enhanced feature vector. The encoded enhanced feature vector is inversely transformed to generate a density image, and the number of target queries is adaptively set according to the density regions in the density image. Align the density image with the encoded enhanced feature vector, and perform feature enhancement based on the attention mechanism to obtain the density-enhanced feature vector; Sample images from the training dataset are acquired, features are extracted from the sample images, and the directional similarity with similar vectors is analyzed. Gaussian noise is introduced and feature fusion is performed to obtain perturbation information. The perturbation information is added to the mean of the visual vector of the tail sample to obtain the updated visual vector of the tail sample. The feature perturbation network is trained based on the sample images and the updated visual vector of the tail sample. The feature perturbation network enhances the feature vector according to the density and generates the corresponding guiding vector. The dynamic query module generates query content and query location information based on density-enhanced feature vectors, target query quantity, and guiding vectors, and constructs a target query set. The decoding module uses a multi-layer interactive attention mechanism to progressively decode the encoded enhanced feature vector based on the target query set to obtain the target detection result. The classification performance of the decoding module is optimized by constructing a classification loss function based on a category instance adaptation mechanism and a semantic visual boundary constraint mechanism.

[0020] In this embodiment of the invention, a feature extraction module performs multi-scale feature extraction on drone aerial images acquired by the drone to obtain a feature map. A multimodal feature fusion module extracts corresponding feature vectors based on semantic and visual dimensions, performs optimization processing, and performs weighted fusion and feature enhancement to obtain a fused enhanced feature vector. A position encoding module encodes the position of each pixel in the feature map and adds it to the fused enhanced feature vector to obtain a position enhanced feature vector. A multi-layer self-attention mechanism and a feedforward network are used to perform global feature analysis on the position enhanced feature vector to obtain an encoded enhanced feature vector.

[0021] By performing an inverse transform on the encoded enhanced feature vector to restore the spatial structure and generate a density image, the number of target queries is adaptively set according to the density regions in the density image. The density image is aligned with the encoded enhanced feature vector, and feature enhancement is performed based on spatial attention and channel attention mechanisms to obtain the density enhanced feature vector. The feature perturbation network generates the corresponding guiding vector based on the density enhanced feature vector. The dynamic query module generates query content and query location information based on density-enhanced feature vectors, target query quantity, and guiding vectors, and constructs a target query set. The decoding module uses a multi-layer interactive attention mechanism to progressively decode the encoded enhanced feature vector based on the target query set to obtain the target detection result.

[0022] In this embodiment of the invention, a classification loss function is constructed based on a category instance adaptation mechanism and a semantic visual boundary constraint mechanism to optimize the classification performance of the decoding module.

[0023] In this embodiment of the invention, the multimodal feature fusion module performs weighted fusion and feature enhancement based on semantic and visual features to obtain a fused enhanced feature vector, which enriches the feature expression of long-tailed targets, reduces the impact of insufficient features of long-tailed targets, and improves the distinguishability and detection accuracy of long-tailed targets in complex backgrounds.

[0024] In this embodiment of the invention, feature extraction is performed on sample images based on semantic and visual features, and the directional similarity with similar vectors is analyzed. Gaussian noise is introduced and feature fusion is performed to obtain perturbation information. The perturbation information is added to the mean of the visual vector of the tail sample to obtain the updated visual vector of the tail sample. The feature perturbation network is trained using the sample images and the updated visual vector of the tail sample to improve the network's recognition performance of sample images under complex interference. The feature perturbation network enhances the feature vector according to the density and generates corresponding guiding vectors, which improves the diversity and aggregation of long-tail target features, reduces the adverse effects of noise on feature learning, and thus enhances the decoding module's ability to judge long-tail targets and its robustness.

[0025] In this embodiment of the invention, feature extraction is performed by a feature extraction module, and feature extraction and fusion based on semantic and visual dimensions are performed by a multimodal feature fusion module to achieve rich expression of long-tail target features. Spatial location information is added based on a location encoding module, and global contextual feature perception is achieved through a multi-layer self-attention mechanism and a feedforward network to obtain the encoded enhanced feature vector. By adaptively setting the number of target queries, constructing density-enhanced feature vectors, and generating guiding vectors based on perturbation information, a target query set is obtained. The decoding module decodes the encoded enhanced feature vectors layer by layer according to the target query set to obtain the target detection result. Compared with traditional technologies, In comparison, the above method, through a feature extraction module and a multimodal feature fusion module, realizes multi-feature representation of long-tail targets, solving the problems of scarce long-tail category samples and insufficient feature extraction in traditional technologies. Furthermore, by constructing a target query set, it improves the detection performance of the decoding module for long-tail targets. The classification loss function constructed based on the category instance adaptation mechanism and the semantic visual boundary constraint mechanism reduces the risk of recognition confusion and false detection by expanding the category boundary between the feature space where the long-tail target is located and similar categories. It further improves the detection accuracy and robustness of long-tail targets under complex backgrounds and noise interference, realizing end-to-end detection of long-tail targets and has broad application prospects.

[0026] In this embodiment of the invention, the feature extraction module performs multi-level and multi-scale modeling of UAV aerial images to obtain feature maps at different levels; among them, the low-level feature maps can retain rich edge textures and spatial details, while the high-level feature maps extract more abstract semantic information, thereby taking into account both local detail representation and global context understanding.

[0027] In this embodiment of the invention, the feature extraction module is built based on convolutional neural networks, residual networks, and Transformer models.

[0028] In this embodiment of the invention, the feature extraction module can use a ResNet50 network, which can efficiently capture target features at different resolutions and output multi-scale feature maps by stacking convolutional layers and residual connection structures.

[0029] In this embodiment of the invention, a Transformer encoder based on a multi-layer self-attention mechanism and a feedforward network is used to perform global feature analysis on the position enhancement feature vector, establish the dependency relationship between different position features, analyze the correlation between image regions, and obtain the encoded enhancement feature vector of global feature expression.

[0030] In this embodiment of the invention, a deformable attention mechanism can also be used to perform feature analysis on the position enhancement feature vector, so as to adaptively select key regions for modeling among features of different scales, effectively reducing computational overhead, while enhancing the perception ability of small and sparse targets.

[0031] In this embodiment of the invention, the dynamic query module generates query content and query location information based on the density-enhanced feature vector, the number of target queries, and the guiding vector, and constructs a target query set to reduce redundant queries in sparse scenarios and ensure detection coverage in dense scenarios, thereby improving detection efficiency and accuracy.

[0032] In this embodiment of the invention, the decoding module is built based on the Transformer decoder structure. It performs stepwise decoding of the encoded enhanced feature vector according to the target query set to obtain the target detection result. The target detection result includes the spatial location, category label and confidence data of the long-tail target.

[0033] In this embodiment of the invention, the steps are as follows: Based on the multimodal feature fusion module, the semantic feature vector of the feature map is extracted by the text encoder and optimized to obtain a semantic feature optimization vector; the visual feature vector of the feature map is extracted and optimized to obtain a visual feature optimization vector; the semantic feature optimization vector and the visual feature optimization vector are weighted and fused together for feature enhancement to obtain a fused and enhanced feature vector; including: Based on the text encoder, semantic features are extracted from the feature map to obtain multiple sets of semantic feature vectors. Then, the multiple sets of semantic feature vectors are averaged, normalized, and linearly projected to obtain the semantic feature optimized vector. Visual features are extracted from the feature map to obtain visual feature vectors. The visual feature vectors are then processed by average pooling, multilayer perceptual projection, and normalization to obtain optimized visual feature vectors. A gating weighting mechanism is used to dynamically allocate the weights of the semantic feature optimization vector and the visual feature optimization vector for feature fusion, thereby obtaining a fused feature vector; By adjusting the perturbation amplitude of the features through a scaling factor, the fused feature vector is enhanced to obtain the fused enhanced feature vector.

[0034] In this embodiment of the invention, the text encoder is implemented as a CLIP encoder, which extracts the semantic features of the feature map, performs averaging, normalization and linear projection operations to obtain the semantic feature optimization vector, so as to alleviate the semantic bias caused by a single expression.

[0035] In this embodiment of the invention, the text encoder obtains multiple sets of semantic feature vectors based on multiple prompt templates. For example, for bicycle, it generates the following sentences: "a photo of a {bicycle}", "a drone view of a {bicycle}", "a photo of a small {bicycle}", "a photo of a large {bicycle}", "a clear photo of a {bicycle}", "a blurry photo of a {bicycle}".

[0036] In this embodiment of the invention, visual feature vectors are extracted from the feature map and subjected to average pooling, multi-layer perceptual projection, and normalization to obtain optimized visual feature vectors. A gating weighting mechanism is used to dynamically allocate the weights of the optimized semantic feature vectors and the optimized visual feature vectors for feature fusion to obtain fused feature vectors. This enriches the feature representation of long-tail targets in the feature map, and the fused feature vectors are enhanced to reduce the confusion between long-tail targets and similar targets, thereby improving the detection accuracy of the method for long-tail targets.

[0037] In this embodiment of the invention, the step is as follows: the position of the pixel in the feature map is encoded by the position encoding module and added to the fused enhanced feature vector to obtain the position enhanced feature vector; including: The position of pixels in the feature map is encoded by the position encoding module to obtain the position encoding vector; The location encoding vector is added to the fused enhanced feature vector to obtain the location enhanced feature vector.

[0038] In this embodiment of the invention, the position encoding module is used to generate spatial position information based on the feature map, that is, to encode the position of the pixel in the feature map to obtain the position encoding vector, and to add the position encoding vector to the fusion enhancement feature vector to obtain the position enhancement feature vector. The position enhancement feature vector contains representations of semantic features, visual features and spatial position information, so that the method pays attention to the relationship between target category and space in the target detection process, maintains spatial structure constraints, and improves the detection accuracy of long-tail targets in complex backgrounds.

[0039] In this embodiment of the invention, the steps include: performing an inverse transform on the encoded enhanced feature vector to generate a density image, and adaptively setting the number of target queries based on the density regions in the density image; including: The encoded enhanced feature vector is restored to a two-dimensional feature map by deflating, and the two-dimensional feature map is input into the density generation network for prediction to obtain a density image. The target query count is adaptively set based on the density regions in the density image.

[0040] In this embodiment of the invention, the encoded enhanced feature vector is restored to a two-dimensional feature map through the Unflatten operation, and the density generation network is input to generate a density image; the target query quantity is obtained by performing a global pooling operation on the density image.

[0041] In this embodiment of the invention, 300 queries are set for low-density regions in the density image; 500 queries are set for low-to-medium density regions in the density image; 900 queries are set for medium-to-high density regions in the density image; and 1500 queries are set for high-density regions in the density image.

[0042] In this embodiment of the invention, the density generation network consists of convolutional layers, dilated convolutional layers, and nonlinear activation units.

[0043] In this embodiment of the invention, the number of target queries is dynamically set according to the sparsity or density of long-tail targets, which effectively reduces redundant calculations while ensuring detection accuracy.

[0044] In this embodiment of the invention, the steps include: aligning the density image with the encoded enhanced feature vector, and performing feature enhancement based on an attention mechanism to obtain the density-enhanced feature vector; including: Align the density image with the encoded augmented feature vector to obtain the target distribution in the encoded augmented feature vector; Based on the spatial attention mechanism, feature enhancement processing is performed on the dense target region in the encoded enhanced feature vector to obtain the spatial attention vector; Based on the channel attention mechanism, feature enhancement processing is performed on the target-related region in the encoded enhanced feature vector to obtain the channel attention vector; Density-enhanced feature vectors are obtained by weighted fusion of spatial attention vectors and channel attention vectors on the encoded enhanced feature vectors.

[0045] In this embodiment of the invention, the density image is aligned with the encoded enhancement feature vector to obtain the target distribution in the encoded enhancement feature vector. Feature enhancement is performed on the target-dense region and the target-related region based on the spatial attention mechanism and the channel attention mechanism, respectively, and weighted fusion is performed to obtain the density enhancement feature vector.

[0046] In this embodiment of the invention, attention is allocated to the concentrated area of ​​long-tailed targets based on the spatial attention mechanism, weakening the interference information of the background area; the feature channels that are highly related to the target category are enhanced based on the channel attention mechanism, while irrelevant or redundant channels are suppressed, thereby highlighting the semantic representation of long-tailed targets, and feature fusion is performed to obtain density-enhanced feature vectors, so as to improve the detection stability and robustness of small targets and long-tailed targets.

[0047] In this embodiment of the invention, the steps are as follows: acquiring sample images from the training dataset, extracting features from the sample images and analyzing their directional similarity with similar vectors, introducing Gaussian noise and then performing feature fusion to obtain perturbation information, adding the perturbation information to the mean of the tail sample visual vector to obtain an updated tail sample visual vector, training the feature perturbation network based on the sample images and the updated tail sample visual vector; and generating corresponding guiding vectors by using the feature perturbation network to enhance the feature vectors according to density. This includes: Obtain sample images from the training dataset and statistically analyze the annotation information corresponding to the sample images; Based on the annotation information, the sample images are divided into head sample images, mid-frequency sample images, and tail sample images; Extract the semantic vector of the tail sample from the tail sample image using a text editor; Obtain the semantic vector of the comparison sample that is similar to the semantic vector of the tail sample, and calculate the semantic direction vector of the tail sample; ; in, This is the semantic direction vector for the tail. This is the semantic vector of the tail sample. The semantic vectors of the comparison samples are similar to those of the tail samples. This indicates normalization processing; Extract the visual vector of the tail sample from the tail sample image; Obtain visual vectors of contrasting samples that are similar to the visual vectors of the tail samples, and calculate the visual direction vector of the tail. ; in, The visual direction vector for the tail. The mean of the visual vectors of the tail samples. The mean visual vector of the comparison sample that is similar to the visual vector of the tail sample; Gaussian noise is projected onto the tail semantic direction vector and the tail visual direction vector respectively to generate the noise tail semantic vector and the noise tail visual vector. ; ; in, It is Gaussian noise. , This is the semantic vector of the noise tail. The visual vector for the noise tail; The tail semantic direction vector, tail visual direction vector, noisy tail semantic vector, and noisy tail visual vector are weighted and fused to obtain perturbation information. ; in, For perturbation information, As a balance factor, This is the semantic perturbation intensity factor. The visual disturbance intensity factor. These are the semantic feature weight parameters. For visual feature weight parameters; The perturbation information is injected into the mean of the visual vector of the tail sample to obtain the updated visual vector of the tail sample, which is used as the guiding vector. ; in, To update the visual vector of the tail sample.

[0048] In this embodiment of the invention, during the training phase of the feature perturbation network, tail semantic direction vectors and tail visual direction vectors are generated based on semantic and visual feature differences between categories, respectively. Gaussian noise is projected onto the tail semantic direction vectors and tail visual direction vectors, respectively. The tail semantic direction vectors, tail visual direction vectors, noisy tail semantic vectors, and noisy tail visual vectors are weighted and fused to obtain perturbation information. The perturbation information is injected into the mean of the tail sample visual vectors to obtain updated tail sample visual vectors. The feature perturbation network is trained using sample images and updated tail sample visual vectors to improve the recognition of tail target images under complex interference.

[0049] In this embodiment of the invention, during the inference stage based on the feature perturbation network, the interference noise information contained in the input density-enhanced feature vector is identified, and a corresponding guiding vector is generated.

[0050] In this embodiment of the invention, by introducing directional perturbation information, an updated visual vector for tail samples is constructed. This expands the discrimination boundary between tail samples and similar categories under the condition of scarce tail samples, thereby improving the recognition performance of the feature perturbation network for tail samples. Furthermore, the input density-enhanced feature vector is processed to generate a guiding vector, which significantly reduces the false detection and false negative problems caused by semantic or visual similarity, thereby improving the detection accuracy and robustness under long-tail distribution.

[0051] In this embodiment of the invention, the step of constructing a classification loss function based on a category instance adaptation mechanism and a semantic visual boundary constraint mechanism to optimize the classification performance of the decoding module includes: Construct category discrimination boundaries; ; in, For the goal Class and Target Class category discrimination boundary, For the goal Class and Target The semantic similarity matrix of classes For the goal Class and Target Visual similarity matrix of classes For semantic similarity weights, For visual similarity weights; Adaptively adjust the category discrimination boundary based on the instance to obtain the category instance boundary; ; in, For the goal Class and Target Class instance boundaries, This is the instance difficulty factor; Boundary correction is performed on the detection results of samples that are not true categories to obtain corrected sample detection results; ; in, To correct the sample test results, For sample test results; The cross-entropy loss function is used to adjust the class instance boundaries based on the corrected sample detection results, so as to optimize the classification performance of the decoding module.

[0052] In this embodiment of the invention, class instance boundaries are adaptively constructed based on instances, in the target... Class and Target When the level of class confusion is high, enlarge the boundaries of class instances to ensure clear boundaries between long-tail targets and similar categories. Class and Target When classes are not easily confused, reduce the boundaries of class instances to avoid excessive penalties.

[0053] In this embodiment of the invention, boundary correction is performed on the detection results of samples that are not true categories to ensure the accuracy of classification during training.

[0054] In this embodiment of the invention, the classification performance of the decoding module is optimized by using a classification loss function module, thereby improving the classification accuracy of the method for long-tail targets and similar targets, effectively alleviating the category confusion problem in tail class detection, significantly improving the recognition accuracy of tail class and small sample targets, and thus enhancing the robustness and generalization ability of target detection.

[0055] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Additionally, the terms "front," "back," "left," "right," "upper," and "lower" in this document refer to the placement shown in the accompanying drawings.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement, characterized in that, include: Acquire drone aerial images; The feature extraction module performs multi-level, multi-scale modeling of drone aerial images to obtain feature maps at different levels. The multimodal feature fusion module extracts semantic feature vectors from the feature map through a text encoder and optimizes them to obtain semantic feature optimized vectors; it also extracts visual feature vectors from the feature map and optimizes them to obtain visual feature optimized vectors; and then performs weighted fusion and feature enhancement on the semantic feature optimized vectors and visual feature optimized vectors to obtain fused enhanced feature vectors. The location encoding module encodes the position of the pixel in the feature map and adds it to the fused enhanced feature vector to obtain the location enhanced feature vector; Global feature analysis is performed on the location-enhanced feature vector based on a multi-layer self-attention mechanism and a feedforward network to obtain the encoded enhanced feature vector. The encoded enhanced feature vector is inversely transformed to generate a density image, and the number of target queries is adaptively set according to the density regions in the density image. Align the density image with the encoded enhanced feature vector, and perform feature enhancement based on the attention mechanism to obtain the density-enhanced feature vector; Sample images from the training dataset are acquired, features are extracted from the sample images, and the directional similarity with similar vectors is analyzed. Gaussian noise is introduced and feature fusion is performed to obtain perturbation information. The perturbation information is added to the mean of the visual vector of the tail sample to obtain the updated visual vector of the tail sample. The feature perturbation network is trained based on the sample images and the updated visual vector of the tail sample. The feature perturbation network enhances the feature vector according to the density and generates the corresponding guiding vector. The dynamic query module generates query content and query location information based on density-enhanced feature vectors, target query quantity, and guiding vectors, and constructs a target query set. The decoding module uses a multi-layer interactive attention mechanism to progressively decode the encoded enhanced feature vector based on the target query set to obtain the target detection result. The classification performance of the decoding module is optimized by constructing a classification loss function based on a category instance adaptation mechanism and a semantic visual boundary constraint mechanism.

2. The method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement according to claim 1, characterized in that, The steps include: First, based on the multimodal feature fusion module, the semantic feature vector of the feature map is extracted by the text encoder and optimized to obtain an optimized semantic feature vector. Second, the visual feature vector of the feature map is extracted and optimized to obtain an optimized visual feature vector. Third, the optimized semantic feature vector and the optimized visual feature vector are weighted and fused together for feature enhancement to obtain a fused enhanced feature vector. This includes: Based on the text encoder, semantic features are extracted from the feature map to obtain multiple sets of semantic feature vectors. Then, the multiple sets of semantic feature vectors are averaged, normalized, and linearly projected to obtain the semantic feature optimized vector. Visual features are extracted from the feature map to obtain visual feature vectors. The visual feature vectors are then processed by average pooling, multilayer perceptual projection, and normalization to obtain optimized visual feature vectors. A gating weighting mechanism is used to dynamically allocate the weights of the semantic feature optimization vector and the visual feature optimization vector for feature fusion, thereby obtaining a fused feature vector; By adjusting the perturbation amplitude of the features through a scaling factor, the fused feature vector is enhanced to obtain the fused enhanced feature vector.

3. The method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement according to claim 1, characterized in that, The steps include: encoding the position of pixels in the feature map using a position encoding module, and adding it to the fused enhanced feature vector to obtain the position enhanced feature vector; including: The position of pixels in the feature map is encoded by the position encoding module to obtain the position encoding vector; The location encoding vector is added to the fused enhanced feature vector to obtain the location enhanced feature vector.

4. The method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement according to claim 1, characterized in that, The steps include: performing an inverse transform on the encoded enhanced feature vector to generate a density image, and adaptively setting the number of target queries based on the density regions in the density image; including: The encoded enhanced feature vector is restored to a two-dimensional feature map by deflating, and the two-dimensional feature map is input into the density generation network for prediction to obtain a density image. The target query count is adaptively set based on the density regions in the density image.

5. The method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement according to claim 1, characterized in that, The steps include: aligning the density image with the encoded enhanced feature vector, and performing feature enhancement based on an attention mechanism to obtain the density-enhanced feature vector; including: Align the density image with the encoded augmented feature vector to obtain the target distribution in the encoded augmented feature vector; Based on the spatial attention mechanism, feature enhancement processing is performed on the dense target region in the encoded enhanced feature vector to obtain the spatial attention vector; Based on the channel attention mechanism, feature enhancement processing is performed on the target-related region in the encoded enhanced feature vector to obtain the channel attention vector; Density-enhanced feature vectors are obtained by weighted fusion of spatial attention vectors and channel attention vectors on the encoded enhanced feature vectors.

6. The method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement according to claim 1, characterized in that, The steps are as follows: obtain sample images from the training dataset, extract features from the sample images, analyze the directional similarity with similar vectors, introduce Gaussian noise and perform feature fusion to obtain perturbation information, add the perturbation information to the mean of the tail sample visual vector to obtain the updated tail sample visual vector, and train the feature perturbation network based on the sample images and the updated tail sample visual vector. The feature perturbation network generates corresponding guiding vectors based on density-enhanced feature vectors; including: Obtain sample images from the training dataset and statistically analyze the annotation information corresponding to the sample images; Based on the annotation information, the sample images are divided into head sample images, mid-frequency sample images, and tail sample images; Extract the semantic vector of the tail sample from the tail sample image using a text editor; Obtain the semantic vector of the comparison sample that is similar to the semantic vector of the tail sample, and calculate the semantic direction vector of the tail sample; ； in, This is the semantic direction vector for the tail. This is the semantic vector of the tail sample. The semantic vectors of the comparison samples are similar to those of the tail samples. This indicates normalization processing; Extract the visual vector of the tail sample from the tail sample image; Obtain visual vectors of contrasting samples that are similar to the visual vectors of the tail samples, and calculate the visual direction vector of the tail. ； in, This is the visual direction vector for the tail. The mean of the visual vectors of the tail samples. The mean visual vector of the comparison sample that is similar to the visual vector of the tail sample; Gaussian noise is projected onto the tail semantic direction vector and the tail visual direction vector respectively to generate the noise tail semantic vector and the noise tail visual vector. ； ； in, It is Gaussian noise. , This is the semantic vector of the noise tail. The visual vector for the noise tail; The tail semantic direction vector, tail visual direction vector, noisy tail semantic vector, and noisy tail visual vector are weighted and fused to obtain perturbation information. ； in, For perturbation information, As a balance factor, This is the semantic perturbation intensity factor. The visual disturbance intensity factor. These are the semantic feature weight parameters. These are the visual feature weight parameters; The perturbation information is injected into the mean of the visual vector of the tail sample to obtain the updated visual vector of the tail sample, which is used as the guiding vector. ； in, To update the visual vector of the tail sample.

7. The method for detecting long-tailed targets in UAV aerial images based on multimodal feature fusion and feature enhancement according to claim 1, characterized in that, The steps include: constructing a classification loss function based on a category instance adaptation mechanism and a semantic visual boundary constraint mechanism to optimize the classification performance of the decoding module; including: Construct category discrimination boundaries; ； in, For the goal Class and Target Class category discrimination boundary, For the goal Class and Target The semantic similarity matrix of classes For the goal Class and Target Visual similarity matrix of classes For semantic similarity weights, For visual similarity weights; Adaptively adjust the category discrimination boundary based on the instance to obtain the category instance boundary; ； in, For the goal Class and Target Class instance boundaries, This is the instance difficulty factor; Boundary correction is performed on the detection results of samples that are not true categories to obtain corrected sample detection results; ； in, To correct the sample test results, For sample test results; The cross-entropy loss function is used to adjust the class instance boundaries based on the corrected sample detection results, so as to optimize the classification performance of the decoding module.

Images