A multi-modal adaptive fine-grained feature fusion method based on a drone visual angle

Abstract

The application discloses a multi-modal adaptive fine-grained feature fusion method based on a UAV visual angle, which comprises the following steps: visible light images and infrared images collected by a UAV are subjected to feature extraction through single-modal feature extraction branches respectively, so that multi-scale feature maps of two modes are obtained; based on the multi-scale feature maps of the two modes, the weight is adjusted by calculating the local region structure similarity of the modal features, and an adaptive fusion weight matrix is constructed; the multi-scale feature maps of the two modes are subjected to fine-grained feature weighted fusion by using the adaptive fusion weight matrix, in the fusion process, the weight matrix and the feature map space size are matched through an interpolation method first, and then a weighted operation is performed, so that a fused feature map is obtained; based on the fused feature map, the multi-scale features are aggregated through a detector neck first, and then target classification and a bounding box regression are performed through a detector head, so that a classification result and a bounding box regression result of the target are outputted. The application can adapt to a complex detection scene of the UAV visual angle.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, and in particular relates to a multimodal adaptive fine-grained feature fusion method based on the perspective of a UAV. Background Technology

[0002] In recent years, deep learning technology has made significant progress in the field of target detection and has been widely applied to aerial target recognition tasks from the perspective of UAVs. Single visible light images have inherent limitations in complex environments such as low light, fog, and strong occlusion, easily leading to blurred target features and poor distinguishability, resulting in a significant decline in UAV target recognition performance. Although some studies have attempted to improve model robustness through data augmentation and other methods, relying solely on information from a single visible light modality is still insufficient to meet the high-precision target recognition requirements of UAVs in complex environments. To alleviate the problem of insufficient visible light modal information, multimodal fusion methods have gradually gained attention. Researchers have introduced infrared, lidar, and other modalities to complement and fuse with visible light images to improve the stability of detection under complex conditions. Existing research has confirmed that the combined application of visible light and infrared modalities can significantly improve detection performance in low-light scenarios. As research progresses, various deep learning-based fusion strategies are emerging, including attention mechanisms and feature-level fusion. For example, some methods combine target features from infrared images with texture details from visible light images to enhance detection accuracy. Some methods achieve multimodal detection deployable by UAVs through feature correlation learning. Other studies improve downsampling, loss functions, and clustering strategies to adapt to the specific perspective requirements of UAVs. Some works also introduce a dual cross-attention mechanism at the feature map level to achieve more refined adaptive fusion.

[0003] Most existing multimodal fusion methods still focus on global feature fusion, failing to fully utilize the fine-grained complementary information between modalities in local regions. From a UAV perspective, scenarios with frequent lighting changes, widespread target occlusion, and densely distributed small targets are common. Global fusion strategies cannot adaptively adjust to modal information differences in local image regions, resulting in insufficient detection accuracy in these complex scenarios. Furthermore, existing methods do not retain enough modality-specific information during training, easily leading to the loss of key modal information in the fused features, further affecting the accuracy and stability of target detection and failing to meet the high-performance requirements of UAV multimodal target detection in practical applications. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a multimodal adaptive fine-grained feature fusion method based on the perspective of an unmanned aerial vehicle (UAV) to solve the problems existing in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a multimodal adaptive fine-grained feature fusion method based on the perspective of an unmanned aerial vehicle (UAV), comprising:

[0006] S1: The visible light images and infrared images collected by the UAV are extracted using a single-modal feature extraction branch to obtain multi-scale feature maps for each of the two modalities;

[0007] S2: Based on the multi-scale feature maps of the two modalities obtained in step S1, the weights are adjusted by calculating the local regional structural similarity of the modal features, and an adaptive fusion weight matrix is ​​constructed.

[0008] S3: Using the adaptive fusion weight matrix obtained in step S2, fine-grained feature weighting fusion is performed on the multi-scale feature maps of the two modalities. During the fusion process, the weight matrix and the feature map space size are first matched by interpolation, and then the weighting operation is performed to obtain the fused feature map.

[0009] S4: Based on the fused feature map obtained in step S3, first aggregate multi-scale features through the detector neck, then perform target classification and bounding box regression through the detector head, and output the target classification result and bounding box regression result;

[0010] S5: During the model training phase, calculate the network loss value, which includes location loss, target loss, classification loss, and feature contrast loss. The feature contrast loss is obtained by maximizing the mutual information between the fused feature map and the original feature maps of the two modalities. Based on the network loss value, optimize the model parameters through gradient feedback until the preset training requirements are met.

[0011] Preferably, the single-modality feature extraction branch in step S1 includes a series of convolutional layers. The convolutional layers gradually extract features from low-level spatial details to high-level semantic abstractions. After obtaining the multi-scale feature maps of the two modalities, a portion of the top-level feature map is selected for subsequent fine-grained feature weighting fusion.

[0012] Preferably, the process of calculating the local region structural similarity of modal features in step S2 includes:

[0013] Sigmoid normalization is performed on the feature maps of the two modalities, and channel-level maximum value operation is performed on the normalized feature maps to obtain spatial activation information;

[0014] A spatial activation mask is generated based on the spatial activation information. Connected regions of the spatial activation mask are extracted and the region overlap rate is calculated. The region overlap rate is used to characterize the structural similarity of local regions.

[0015] Preferably, in step S2, when constructing the adaptive fusion weight matrix, if the overlap rate of the local region exceeds the threshold, the weights of the two modes are evenly distributed; if the overlap rate does not exceed the threshold, the weight ratio of the richer mode is enhanced based on the difference between the threshold and the current overlap rate, and finally an adaptive fusion weight matrix adapted to the local region features is formed.

[0016] Preferably, the interpolation method used in step S3 is the nearest neighbor interpolation method. The size of the adaptive fusion weight matrix is ​​adjusted by the nearest neighbor interpolation method so that the adjusted weight matrix is ​​completely matched with the multi-scale feature maps of the two modalities in the height and width dimensions.

[0017] Preferably, in the fine-grained feature weighted fusion process in step S3, cross-modal feature enhancement processing is also performed on the feature maps of the two modalities. The cross-modal feature enhancement processing is achieved by flattening the feature maps and adding learnable positions to generate tokens, and then encoding cross-modal similarity information through a dual-branch attention mechanism.

[0018] Preferably, after cross-modal feature enhancement processing, iterative cross-modal feature enhancement is performed. The enhanced features from the previous round are used as the input for the current round using an iterative learning strategy, and the complementarity of cross-modal features is gradually optimized through parameter sharing.

[0019] Preferably, in step S4, the detector neck enhances the semantic representation by aggregating fused feature maps of different scales.

[0020] Preferably, in step S5, when calculating the feature contrast loss, the mutual information of the feature maps of the multiple layers of the feature pyramid is calculated separately, and then the loss term is constructed through an exponential function.

[0021] In a second aspect, the present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0022] Compared with the prior art, the present invention has the following advantages and technical effects:

[0023] To address the technical problem that "existing multimodal fusion methods mostly adopt global fusion strategies, which are difficult to cope with changes in lighting, target occlusion, and dense small objects under the perspective of UAVs, resulting in insufficient utilization of effective information in local areas," this invention breaks through the limitations of global fusion by "calculating the local regional structural similarity of modal features to adjust weights and constructing an adaptive fusion weight matrix" in step S2. It can dynamically allocate weights for different local regions of the image, so that the effective local information of visible light and infrared modalities under changes in lighting can be accurately utilized, while alleviating the interference of occlusion and dense small objects on detection, and adapting to the complex detection scenarios under the perspective of UAVs.

[0024] This invention first matches the weight matrix with the spatial dimensions of the feature map using interpolation, and then performs a weighting operation to ensure that the adaptive fusion weight matrix accurately corresponds to the multi-scale feature map, avoiding fusion deviations caused by spatial size mismatch. This allows the complementary features of the two modalities to be effectively aggregated, and the generated fusion feature map has both the texture details of the visible light modality and the target contour information of the infrared modality. This invention further mines the multi-scale semantic information of the fusion features by combining "multi-scale feature aggregation in the detector neck + classification and regression in the detector head", improving the detection adaptability to targets of different sizes, and ultimately significantly improving the accuracy of target classification and bounding box localization.

[0025] This invention introduces a feature contrast loss technique that maximizes the mutual information between the fused feature map and the original feature maps of the two modalities. This technique, along with the location loss, target loss, and classification loss, constitutes the network loss. During training, it guides the model to retain the specific information of the two modalities, avoiding the loss or redundancy of modal information. At the same time, the gradient feedback optimization mechanism continuously adjusts the model parameters, enabling the model to stably learn effective features even in complex scenarios (such as low light and strong occlusion), enhancing the overall detection robustness of the model, and ensuring that the trained model maintains stable performance in actual UAV detection tasks. Attached Figure Description

[0026] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0027] Figure 1 This is a schematic diagram of a multimodal adaptive fine-grained feature fusion network according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the fusion module and the iterative cross-modal feature enhancement module in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the cross-modal fine-grained feature weighting module structure according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the cross-modal feature enhancement module structure according to an embodiment of the present invention;

[0031] Figure 5 The flowcharts for network model training and test sample detection are shown in this embodiment of the invention. Detailed Implementation

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0034] Example 1

[0035] To address the issue that existing multimodal fusion methods primarily rely on global feature fusion and struggle to fully utilize fine-grained complementary information between modalities, this embodiment designs a higher-performance and more accurate multimodal adaptive fine-grained feature fusion network (as shown in Figure 1). By innovatively introducing a fine-grained adaptive fusion weighting module into the architecture, fusion weights can be adaptively allocated in local regions, more fully mining fine-grained complementary information between modalities. A carefully designed feature contrast loss guides iterative model training, further encouraging the model to retain the unique features of different modalities and improving the discriminativeness of the fused representation. The combination of these two approaches not only effectively overcomes the performance limitations of existing methods in complex scenarios but also significantly improves the accuracy of UAV multimodal target detection.

[0036] This embodiment provides a multimodal adaptive fine-grained feature fusion method based on UAV perspective, including:

[0037] S1: The visible light images and infrared images collected by the UAV are extracted using a single-modal feature extraction branch to obtain multi-scale feature maps for each of the two modalities;

[0038] Furthermore, the single-modality feature extraction branch in step S1 includes a series of convolutional layers. Through these convolutional layers, features from low-level spatial details to high-level semantic abstractions are extracted step by step. Finally, after obtaining the multi-scale feature maps of the two modalities, a portion of the top-level feature map is selected for subsequent fine-grained feature weighted fusion.

[0039] S2: Based on the multi-scale feature maps of the two modalities obtained in step S1, the weights are adjusted by calculating the local regional structural similarity of the modal features, and an adaptive fusion weight matrix is ​​constructed.

[0040] Furthermore, the process of calculating the local region structural similarity of modal features in step S2 includes:

[0041] Sigmoid normalization is performed on the feature maps of the two modalities, and channel-level maximum value operation is performed on the normalized feature maps to obtain spatial activation information;

[0042] A spatial activation mask is generated based on the spatial activation information. Connected regions of the spatial activation mask are extracted and the region overlap rate is calculated. The region overlap rate is used to characterize the structural similarity of local regions.

[0043] Furthermore, in step S2, when constructing the adaptive fusion weight matrix, if the overlap rate of the local region exceeds the threshold, the weights of the two modes are evenly distributed; if the overlap rate does not exceed the threshold, the weight ratio of the richer information mode is enhanced based on the difference between the threshold and the current overlap rate, thus forming an adaptive fusion weight matrix that adapts to the local region features.

[0044] S3: Using the adaptive fusion weight matrix obtained in step S2, fine-grained feature weighting fusion is performed on the multi-scale feature maps of the two modalities. During the fusion process, the weight matrix and the feature map space size are first matched by interpolation, and then the weighting operation is performed to obtain the fused feature map.

[0045] Furthermore, the interpolation method used in step S3 is the nearest neighbor interpolation method. The size of the adaptive fusion weight matrix is ​​adjusted by the nearest neighbor interpolation method so that the adjusted weight matrix is ​​completely matched with the multi-scale feature maps of the two modalities in the height and width dimensions.

[0046] Furthermore, in the fine-grained feature weighted fusion process in step S3, cross-modal feature enhancement processing is also performed on the feature maps of the two modalities. The cross-modal feature enhancement processing is achieved by flattening the feature maps and adding learnable positions to generate tokens, and then encoding cross-modal similarity information through a dual-branch attention mechanism.

[0047] Furthermore, after cross-modal feature enhancement processing, iterative cross-modal feature enhancement is performed. Using an iterative learning strategy, the enhanced features from the previous round are used as the input for the current round, and the complementarity of cross-modal features is gradually optimized through parameter sharing.

[0048] Specifically, Figure 2 further illustrates the construction method of the fusion module. The fine-grained adaptive weighted fusion module comprises four main sub-modules: a cross-modal fine-grained feature weighting module, a spatial feature compression module, a feature enhancement module, and a fine-grained fusion guidance module. These sub-modules are specifically designed to accurately model the fine-grained correspondences between modalities and to promote efficient and effective feature integration. This embodiment will describe the design of each sub-module in detail in the following sections.

[0049] Cross-modal fine-grained feature weighting module: Unlike traditional global feature-level fusion strategies, the cross-modal fine-grained feature weighting module in this embodiment introduces a fine-grained modal fusion mechanism specifically designed to capture the local feature interactions between visible and infrared modalities. As shown in Figure 3, it adaptively calculates fusion weights for feature maps specific to the two modalities and dynamically adjusts the fusion intensity in different fine-grained spatial regions. This adjustment process is guided by the structural alignment of the connected regions between the two modalities, which is quantified through the region overlap relationship.

[0050] Let the input feature maps be respectively and ,in and Representing the first Layer (corresponding to in the structure diagram) The visible light modal feature map and infrared modal feature map of ). In the cross-modal fine-grained feature weighting module, The value is set to 3. First, in this embodiment, both feature maps are normalized using Sigmoid. This step constrains the feature values ​​within a bounded range, thus creating favorable conditions for subsequent fine-grained fusion.

[0051] To extract the most significant responses among the channels, this embodiment performs channel-level maximum value operations on the normalized feature map of each mode, resulting in two spatially compressed maps. and These two compressed maps contain the main spatial activation information used for subsequent weight calculations. Then, adaptive thresholds are calculated for each modality based on the corresponding compressed maps. and The compressed image is then binarized using these thresholds, and the relevant calculation formulas are as follows:

[0052] (1)

[0053] (2)

[0054] Among them, the generated binary image and These represent spatial activation masks for the visible light and infrared modes, respectively. Each element in these masks indicates whether the corresponding location exhibits sufficiently strong mode-specific activation, a determination achieved through an adaptive threshold. Here, This indicates an indicator function whose output is either 0 or 1.

[0055] Next, from the binary graph and Extract connected components from the given components, denoted as follows: and For each connected region in the visible light mode. The corresponding region in the infrared mode Defined as occupying and A set of active pixels at the same spatial location. Therefore, the overlap rate calculated based on the visible light modal connectivity region can be defined as:

[0056] (3)

[0057] Similarly, for each connected region in the infrared mode In the visible light mode, the active regions at the same pixel location are denoted as... The formula for calculating its overlap rate is as follows:

[0058] (4)

[0059] Since the number of activation regions in a binary graph can initially reflect the amount of information contained in the mode, the initial weights... and The calculation method is as follows:

[0060] (5)

[0061] At a fine-grained level, since the overlap rate is calculated based on the connected regions anchored to the current mode, a high overlap rate indicates that the information provided by the two modes is consistent and effective, and the weights should be evenly distributed in this case. Conversely, a low overlap rate indicates that the other mode provides less effective information in the corresponding region, while the current mode contains more useful information, thus requiring an increase in the weight proportion of the current mode. This weight adjustment strategy based on connected regions and overlap rate enables fine-grained adjustment of the fused information, improving the model's adaptability to UAV target detection tasks by highlighting spatially relevant and information-rich regions. Based on this idea, this embodiment can... Calculate the adjusted weights and The formula is as follows:

[0062] (6)

[0063] (7)

[0064] Using a similar method, it can be calculated that and :

[0065] (8)

[0066] (9)

[0067] The final weights can be obtained by averaging the adjusted weights. and :

[0068] (10)

[0069] weight matrix The size is The first and second channels store respectively and The specific expression is as follows:

[0070] (11)

[0071] By using fine-grained spatial adaptive weighting based on local structural consistency, the cross-modal fine-grained feature weighting module enables the model to focus more accurately and meticulously on regions relevant to the target, rather than spreading attention to broad regions with little information.

[0072] Spatial Feature Compression Module: Considering the efficiency requirements of UAV applications, this embodiment introduces a spatial feature compression module to better adapt to processor resource constraints. This module employs an adaptive fusion of average pooling and max pooling, dynamically balancing the contributions of both through a learnable parameter, thereby simultaneously capturing global contextual information and local salient details. Specifically, it will compress the original feature to a size of... Feature maps compressed to The size is adjusted to reduce the subsequent computational complexity. The relevant calculation formula is as follows:

[0073] (12)

[0074] in, The input feature map represents the input from the visible light or infrared mode; This is the space reduction factor; and These represent the compressed feature maps obtained through average pooling and max pooling, respectively. It is a learnable parameter used to control the fusion ratio of the two pooling methods in the final output.

[0075] Feature Enhancement Module: This module includes the Iterative Cross-Modal Feature Enhancement Module and its sub-module, the Cross-Modal Feature Enhancement Module. The structure of these two modules will be described in detail below.

[0076] (1) Cross-modal feature enhancement module: To achieve more effective complementary information fusion between the two modalities, this embodiment introduces a cross-modal feature enhancement module (as shown in Figure 4). Let the first... The input feature map of the layer is and This module first flattens out these features and adds a size of [size missing]. Learnable position embeddings to obtain tokens and These tokens will serve as input to the cross-modal feature enhancement module. To mitigate the misalignment between visible light and infrared images, this embodiment employs a dual cross-modal feature enhancement module to extract information from both modalities, achieving cross-modal feature enhancement. This process can be represented as:

[0077] (13)

[0078] in, and These represent the visible light token and the infrared token input to the cross-modal feature enhancement module, respectively. and This represents the output token after modal enhancement; and These represent the cross-modal feature enhancement modules corresponding to the visible light branch and the infrared branch, respectively.

[0079] The following uses the infrared branch as an example to illustrate the feature extraction process of the cross-modal feature enhancement module (subscripts are omitted for simplicity). Infrared token It will be projected into two matrices and These serve as the value and key in the attention mechanism, respectively; while visible light tokens... It will be projected as a matrix This serves as the query in the attention mechanism. The relevant formula is as follows:

[0080] (14)

[0081] in, , and All are learnable weight matrices. To obtain the similarity between visible light and infrared features, this embodiment first... and Perform a dot product operation, followed by softmax normalization. Then, compare the resulting similarity matrix with... Multiplying them together yields a vector encoding cross-modal similarity information. The formula is as follows:

[0082] (15)

[0083] Then, through the projection matrix Will Projected back to the original space and connected to the original token via residual connection. The features are then combined. Next, a standard Transformer feedforward network is applied to optimize the fused modal features. The final output is... The calculation process is as follows:

[0084] (16)

[0085] (17)

[0086] in, , , A learnable parameter with an initial value of 1 allows the model to adaptively select branches with higher feature quality during training.

[0087] (2) Iterative Cross-Modal Feature Enhancement Module: The iterative cross-modal feature enhancement module adopts an iterative learning strategy to enhance cross-modal information interaction and intra-modal feature complementarity. Each iteration uses the output of the previous iteration as input to achieve gradual feature optimization (as shown in Figure 2). This strategy achieves optimized and complementary extraction of cross-modal features through parameter sharing in each iteration without increasing the number of parameters. Assume the number of iterations is... The iterative process can then be represented as:

[0088] (18)

[0089] in, Indicates the process Output tokens for the visible light and infrared branches after round iteration; This is the input token for this module; This represents an iterative cross-modal feature enhancement module, which includes cross-modal feature enhancement modules for the visible and infrared branches. and After converting back to the feature map, bilinear interpolation needs to be applied to ensure spatial alignment between the two modes.

[0090] Fine-grained fusion guidance module: Known multimodal feature maps and The fine-grained fusion guidance module introduces the corresponding guidance weight matrix. This guides the feature fusion process. Before the fusion operation, nearest neighbor interpolation must be used to adjust... and The dimensions, so that they are respectively with and Spatial size matching. Subsequently, the feature maps of different modalities are weighted and fused to generate a fine-grained fused feature map. The fusion process is defined as follows:

[0091] (19)

[0092] in, This represents element-wise multiplication. In summary, the fine-grained fusion guidance module dynamically adjusts the contribution ratio of each modality in space by guiding weights, achieving more targeted and discriminative fine-grained feature fusion. Ultimately, the fused features exhibit superior representational capabilities in UAV target detection tasks.

[0093] S4: Based on the fused feature map obtained in step S3, first aggregate multi-scale features through the detector neck, then perform target classification and bounding box regression through the detector head, and output the target classification result and bounding box regression result;

[0094] Furthermore, in step S4, the detector neck enhances the semantic representation by aggregating fused feature maps of different scales to improve the detection adaptability to targets of different sizes.

[0095] S5: During the model training phase, calculate the network loss value, which includes location loss, target loss, classification loss, and feature contrast loss. The feature contrast loss is obtained by maximizing the mutual information between the fused feature map and the original feature maps of the two modalities. Based on the network loss value, optimize the model parameters through gradient feedback until the preset training requirements are met.

[0096] Furthermore, in step S5, when calculating the feature contrast loss, the mutual information of the feature maps of the multiple layers of the feature pyramid is calculated separately, and then the loss term is constructed through an exponential function to avoid the gradient vanishing problem.

[0097] Specifically, in order to effectively integrate the unique features of visible light and infrared modes and obtain high-quality fused features, this embodiment designs a mutual information-guided feature contrast loss for multimodal fusion tasks.

[0098] (1) Mutual Information-Guided Feature Contrast Loss: Feature contrast loss aims to enhance the correlation between fused features and original modal features, thereby improving the discriminative power and complementarity of fused features. To ensure that fused features retain key information of each modality at both the spatial and semantic levels, feature contrast loss achieves this goal by maximizing the mutual information between fused features and original modal features. Specifically, the approximate calculation method of mutual information is as follows:

[0099] (20)

[0100] in, Representation of feature map and Covariance between and Representing feature maps respectively and The variance of the mutual information is calculated. To ensure that mutual information is inversely proportional to the loss value and to comprehensively evaluate the fusion quality, the feature contrastive loss is calculated using feature maps at three different levels, as shown in the following formula:

[0101] (twenty one)

[0102] Feature contrastive loss effectively preserves multimodal information and enhances feature complementarity by maximizing the mutual information between the fused feature and each individual modality feature. Furthermore, the exponential function-based design avoids the vanishing gradient problem, improving training stability and sensitivity to small changes in mutual information.

[0103] (2) Final loss function: The final loss function is constructed by fusing multiple independent loss terms, specifically including location loss. Targeted loss Classification loss The feature contrast loss mentioned above is formulated as follows:

[0104] (twenty two)

[0105] like Figure 5 As shown, the flowcharts for network model training and test sample detection are presented.

[0106] The training phase includes the following steps:

[0107] Step 1: Modal Feature Extraction. This stage includes two single-modal feature extraction branches. In each branch, the model progressively extracts hierarchical features through a series of convolutional layers, ultimately obtaining five feature maps at different scales. These feature maps are used to capture multi-scale feature representations ranging from low-level spatial details to high-level semantic abstractions. To improve the generalization ability while maintaining the effectiveness of the fused feature representation, this embodiment selects the top three feature maps for fine-grained fusion in the subsequent cross-modal feature interaction and fusion stage. The single-modal feature extraction process for visible light and infrared inputs can be represented as follows:

[0108] (twenty three)

[0109] in, and These represent visible light images and infrared images, respectively. and Let H, W, and C represent the feature maps extracted by the visible light branch and the infrared branch at the i-th layer, respectively, where H, W, and C represent the height, width, and number of channels of the feature map, respectively. This indicates a feature extraction operation. and These represent the learnable parameters for the visible light branch and the infrared branch, respectively.

[0110] Step 2: Constructing an adaptive fusion weight matrix and fine-grained weighted fusion. Based on the single-modal feature maps of visible light and infrared extracted in the previous step, the feature weighted fusion matrix is ​​adaptively calculated as follows:

[0111] (twenty four)

[0112] Cross-modal fine-grained feature fusion is performed at layers P3 to P5 of the feature pyramid (as shown in Figure 1). The fusion process can be defined as follows:

[0113] (25)

[0114] in, and These represent the visible light branch and the infrared branch at the 1st... Feature map of the layer; Indicates the generation of a feature-weighted fusion matrix The module, this matrix, is used to guide the fine-grained fusion process; Indicates the first Layer fusion feature representation; This represents the fusion function.

[0115] Step 3: Target Prediction. The detector neck enhances semantic representation by aggregating features at different scales, thereby improving the network's ability to detect targets of different sizes. Subsequently, the detector head performs bounding box regression, target classification, and target estimation tasks. (The resulting multi-scale feature map is shown.) The formula used to predict the final detection result is as follows:

[0116] (26)

[0117] (27)

[0118] in, Used for fusing and enhancing multi-scale features output by the backbone network; and These represent the learnable parameters of the detector's neck and head, respectively. and These represent the classification results and the bounding box regression results, respectively.

[0119] Step 4: Network Loss Calculation. Based on the loss calculation formulas (21) and (22) given above, the final network loss value is calculated. .

[0120] Step 5: Gradient Feedback Optimization. Based on the network loss value obtained in the previous step... The gradient descent optimization algorithm is used to optimize the parameters of all network models once.

[0121] Step 6: Training Cycle Determination. When the preset training cycle is reached, stop training the network model and output the network model parameters; otherwise, jump to Step 1 and continue execution.

[0122] The steps in the testing phase specifically include:

[0123] Step 1: Modal Feature Extraction. Features of the infrared and visible light modes are extracted using a single-modal feature extraction branch. and , .

[0124] Step 2: Construct an adaptive fusion weight matrix and fine-grained weighted fusion. Based on the... The feature map of the layer is used to calculate the current fusion weight matrix. To guide fine-grained fusion and obtain fusion features , .

[0125] Step 3: Target Result Prediction. The fused multi-scale feature map. Used to predict the final detection result, obtaining classification results and bounding box regression results. and .

[0126] Step 4: Output the recognition results. The corresponding prediction results are presented to the user intuitively in a combination of text and visualization.

[0127] This embodiment proposes a multimodal adaptive fine-grained feature fusion method based on the UAV perspective. Based on a local feature consistency fusion mechanism, it adaptively allocates fusion weights according to the cross-modal structural similarity within different local regions of the image, thereby enhancing the effective aggregation of target-related features. Simultaneously, a loss calculation guided by mutual information feature comparison is introduced to instruct the model to retain modality-specific information during the training phase, resulting in higher-quality fused features.

[0128] This embodiment significantly reduces the adverse effects of occlusion and modal degradation from the UAV's perspective. Experiments on three publicly available datasets have verified that it offers higher detection accuracy and applicability compared to existing multimodal target detection technologies. This embodiment not only has outstanding practical value in traffic control and vehicle recognition but also demonstrates broad application prospects in fields such as security, agriculture, rescue, and energy inspection.

[0129] Example 2

[0130] This embodiment also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0131] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal adaptive fine-grained feature fusion method based on UAV perspective, characterized in that, Includes the following steps: S1: The visible light images and infrared images collected by the UAV are extracted using a single-modal feature extraction branch to obtain multi-scale feature maps for each of the two modalities; S2: Based on the multi-scale feature maps of the two modalities obtained in step S1, the weights are adjusted by calculating the local regional structural similarity of the modal features, and an adaptive fusion weight matrix is ​​constructed. The process of calculating the local region structural similarity of modal features in step S2 includes: normalizing the feature maps of the two modalities using Sigmoid, performing channel-level maximum value operation on the normalized feature maps to obtain spatial activation information; generating a spatial activation mask based on the spatial activation information, extracting the connected regions of the spatial activation mask and calculating the region overlap rate, and using the region overlap rate to characterize the local region structural similarity. In step S2, when constructing the adaptive fusion weight matrix, if the overlap rate of the local region exceeds the threshold, the weights of the two modes are evenly distributed. If the overlap rate does not exceed the threshold, the weight ratio of the information-rich mode is enhanced based on the difference between the threshold and the current overlap rate, and finally an adaptive fusion weight matrix that adapts to the local region features is formed. S3: Using the adaptive fusion weight matrix obtained in step S2, fine-grained feature weighting fusion is performed on the multi-scale feature maps of the two modalities. During the fusion process, the weight matrix and the feature map space size are first matched by interpolation, and then the weighting operation is performed to obtain the fused feature map. S4: Based on the fused feature map obtained in step S3, first aggregate multi-scale features through the detector neck, then perform target classification and bounding box regression through the detector head, and output the target classification result and bounding box regression result; S5: During the model training phase, calculate the network loss value, which includes location loss, target loss, classification loss, and feature contrast loss. The feature contrast loss is obtained by maximizing the mutual information between the fused feature map and the original feature maps of the two modalities. Based on the network loss value, optimize the model parameters through gradient feedback until the preset training requirements are met.

2. The method according to claim 1, characterized in that, The single-modality feature extraction branch in step S1 includes a series of convolutional layers. Through these convolutional layers, features from low-level spatial details to high-level semantic abstractions are extracted step by step. Finally, after obtaining the multi-scale feature maps of the two modalities, a portion of the top-level feature map is selected for subsequent fine-grained feature weighting and fusion.

3. The method according to claim 1, characterized in that, The interpolation method used in step S3 is the nearest neighbor interpolation method. The size of the adaptive fusion weight matrix is ​​adjusted by the nearest neighbor interpolation method so that the adjusted weight matrix is ​​completely matched with the multi-scale feature maps of the two modalities in the height and width dimensions.

4. The method according to claim 1, characterized in that, In step S3, during the fine-grained feature weighted fusion process, cross-modal feature enhancement processing is also performed on the feature maps of the two modalities. The cross-modal feature enhancement processing is achieved by flattening the feature maps and adding learnable positions to generate tokens, and then encoding cross-modal similarity information through a dual-branch attention mechanism.

5. The method according to claim 4, characterized in that, After cross-modal feature enhancement, iterative cross-modal feature enhancement is performed. The enhanced features from the previous round are used as the input for the current round using an iterative learning strategy. The complementarity of cross-modal features is gradually optimized through parameter sharing.

6. The method according to claim 1, characterized in that, In step S4, the detector neck enhances semantic representation by aggregating fused feature maps of different scales.

7. The method according to claim 1, characterized in that, In step S5, when calculating the feature contrast loss, mutual information is calculated for each of the multi-level feature maps of the feature pyramid, and then the loss term is constructed using an exponential function.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-7.

Images