Weak alignment multi-modal target detection method and system based on bidirectional balance alignment network
By employing an adaptive cross-modal association, symmetric offset generation, and progressive fusion strategy using a bidirectional balanced alignment network, the problem of cross-modal misalignment in multimodal UAV target detection is solved, improving detection accuracy and robustness, especially in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-25
- Publication Date
- 2026-03-13
AI Technical Summary
Existing multimodal UAV target detection methods struggle to achieve effective cross-modal interaction and information fusion in complex environments, especially when there is weak alignment between RGB and infrared images, resulting in insufficient detection accuracy and robustness.
A bidirectional balanced alignment network-based approach is adopted to explicitly mitigate cross-modal misalignment, enhance feature interaction, and achieve balanced fusion through an adaptive cross-modal association module, a symmetric offset generator, and a progressive fusion strategy.
It significantly improves the stability and accuracy of multimodal target detection in complex environments, and enhances the robustness and semantic consistency of the model, especially under adverse conditions such as changes in lighting and fog.
Smart Images
Figure CN121661541A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology and relates to a weakly aligned multimodal target detection method and system based on a bidirectional balanced alignment network. Background Technology
[0002] Unmanned aerial vehicle (UAV) target detection aims to achieve precise location and classification of objects in aerial images, and is currently widely used in many fields such as precision crop monitoring, infrastructure inspection, and wildlife conservation monitoring. However, most existing methods rely solely on RGB images, making their detection performance highly sensitive to imaging conditions. Although RGB data can provide rich texture and structural information under good lighting conditions, its data quality often degrades significantly in complex environments such as nighttime or foggy weather.
[0003] To overcome this limitation, researchers have proposed a multimodal detection framework that integrates visible light and infrared information, aiming to leverage the complementary properties between different modalities to improve the robustness and generalization ability of the model. Nevertheless, achieving effective alignment and fusion of heterogeneous features between visible light and infrared modalities remains a significant challenge in this field. Existing multimodal fusion strategies can be broadly categorized into feature-level fusion and decision-level fusion. Feature-level fusion integrates multimodal information in the intermediate layers of the network to achieve joint representation learning; decision-level fusion combines independent detection results from different modalities. However, both methods still face significant challenges in achieving effective cross-modal interaction and fully utilizing complementary information, thus potentially hindering optimal performance in complex or dynamic scenarios.
[0004] In practical applications, due to factors such as differences in sensor installation location and viewing angle, the acquired RGB and infrared images often exhibit spatial misalignment, i.e., weak alignment. This weak alignment not only manifests as spatial offset but also introduces semantic bias at the feature level. Traditional alignment methods often address this issue from a single dimension, making it difficult to simultaneously guarantee semantic consistency and spatial correspondence. This often leads to one modality dominating during the fusion process, thereby suppressing complementary information in the other modality and ultimately affecting detection accuracy.
[0005] Therefore, there is an urgent need for a new method that can explicitly mitigate cross-modal misalignment, enhance feature interaction, and achieve balanced fusion to improve the accuracy and robustness of multimodal UAV target detection under weak alignment conditions. This invention is proposed against this technological backdrop. Summary of the Invention
[0006] In view of this, the purpose of the present invention is to provide a weakly aligned multimodal target detection method and system based on a bidirectional balanced alignment network.
[0007] To achieve the above objectives, the present invention provides the following technical solution: A weakly aligned multimodal target detection method based on a bidirectional balanced alignment network includes the following steps: S1, acquire multimodal image pairs including RGB and infrared images; S2, use a dual-path backbone network to extract the first feature of the RGB image and the second feature of the infrared image respectively; S3, the first feature and the second feature are processed by the Weak Alignment Module (WAM) to obtain the output feature, wherein the Weak Alignment Module (WAM) includes an Adaptive Cross-Modal Correlation Module (ACMCM), a Symmetric Offset Generator (SOG), and a Progressive Fusion Strategy (PFS); S4, Target detection is performed based on the output features, and the detection result is output.
[0008] Furthermore, the Adaptive Cross-Modal Association Module (ACMCM) constructs a bidirectional semantic correspondence through the following steps: First, the concatenated first and second features are processed using Global Average Pooling (GAP) to generate global dependency weights. , ,in This indicates the first feature. This indicates the second feature. This indicates a channel splicing operation. For 1×1 convolution weights, Use the Sigmoid activation function; Next, the difference between the first feature and the second feature is calculated. and association ,in , , , , This represents an element-wise multiplication operation, and generates local similarity weights based on mean pooling and max pooling. and ,in ,in This indicates the mean pooling operation. This indicates a max pooling operation. This indicates a lightweight convolution operation; Finally, through learnable balance vectors The global dependency weight and the local similarity weight By merging the data, we can obtain the bidirectional correlation weights. and ,in, , . This represents a channel dimension expansion operation, applied to feature enhancement to obtain semantically enhanced features. and ,in , , This is the preset scaling factor.
[0009] Furthermore, the symmetric offset generator SOG employs a coarse-to-fine strategy to generate symmetric offsets, including: First, by sharing convolutional blocks Processing semantic enhancement features after concatenation and Generate shared features ,in ; Secondly, coarse offset is predicted through coarse-grained branching. ,in Predicting fine offsets through fine-grained branching ,in ; Then, the coarse offset is fused. and the fine offset Get the base offset ,in , A preset scaling factor is used; subsequently, modulation and normalization operations are applied to generate the final offset. ,in ,in This indicates a modulation convolution operation. It is the Sigmoid activation function. This indicates the standard deviation calculation operation. This represents an element-wise multiplication operation; Finally, deformable convolution is used based on the final offset. Spatially align the semantically enhanced features to generate spatially aligned features. and ,in , , This represents a deformable convolution operator.
[0010] Furthermore, the Progressive Fusion Strategy (PFS) integrates features through the following steps: The Progressive Fusion Strategy (PFS) integrates features through the following steps: First, the original features and Features after spatial alignment and Perform stitching and use projected convolutional blocks Generate re-fusion features ,in ; Secondly, through learnable weight vectors Adaptive fusion of the original feature stream, aligned feature stream, and refused feature stream yields the output feature. ,in ,in Normalized using the Softmax function.
[0011] Furthermore, the preset scaling factor The value is 0.5.
[0012] Furthermore, the dual-path backbone network extracts multi-scale features and outputs features including stage 3 features P3, stage 4 features P4, and stage 5 features P5. The weak alignment module WAM is applied to the multi-scale features.
[0013] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.
[0014] A weakly aligned multimodal target detection system based on a bidirectional balanced alignment network includes: The image acquisition module is used to acquire multimodal image pairs, including RGB images and infrared images; The feature extraction module includes a dual-path backbone network for extracting the first feature of the RGB image and the second feature of the infrared image, respectively. The Weak Alignment Module (WAM) includes an Adaptive Cross-Modal Correlation Module (ACMCM), a Symmetric Offset Generator (SOG), and a Progressive Fusion Strategy (PFS), which are used to process the first feature and the second feature to obtain the output feature. The target detection module is used to perform target detection based on the output features and output the detection results.
[0015] Furthermore, the Adaptive Cross-Modal Association Module (ACMCM) is configured to construct a bidirectional semantic correspondence through global dependency weights and local similarity weights, the Symmetric Offset Generator (SOG) is configured to generate symmetric offsets through a coarse-to-fine strategy, and the Progressive Fusion Strategy (PFS) is configured to integrate multi-stream features through learnable weight vectors.
[0016] The beneficial effects of this invention are as follows: (1) This invention effectively solves the problem of spatial misalignment caused by sensor misalignment and viewpoint differences in multimodal remote sensing images by introducing a weak alignment module. This module can explicitly model the semantic association and spatial correspondence between cross-modal features, which significantly improves the stability of target detection under complex imaging conditions.
[0017] (2) Through a bidirectional interaction mechanism that combines global dependency modeling and local similarity calculation of the adaptive cross-modal association module, this invention achieves balanced alignment of semantic representations between modalities. This design effectively avoids the single-modal dominance phenomenon common in traditional methods, makes full use of the complementary characteristics of visible light and infrared modes, and enhances the semantic consistency of the model under harsh environments such as changes in illumination and fog.
[0018] (3) By adopting a coarse-to-fine symmetrical offset generation strategy, this invention can generate stable and accurate spatial offsets. This mechanism can correct large-scale displacements and optimize local minor misalignments. Robust spatial alignment is achieved through deformable convolution, which significantly improves the detection performance of small and distant targets.
[0019] (4) The progressive fusion strategy proposed in this invention integrates the original features, aligned features and refused features through learnable weights. This hierarchical fusion method not only preserves the unique information of each modality, but also strengthens cross-modal interaction, and finally generates a balanced representation with both discriminative power and robustness, effectively improving the integrity of multimodal feature fusion.
[0020] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is an overall framework diagram of the multimodal UAV remote sensing target detection network model proposed in this invention; Figure 2The present invention includes the Adaptive Cross-Modal Association Module (ACMCM) and the Symmetric Offset Generator (SOG). Figure 3 This invention proposes a progressive fusion strategy, PFS. Figure 4 The results show the detection comparison with other detectors on the DRONEVEHICLE dataset; Figure 5 This is a comparison of detection results with other detectors on the VEDAI dataset. Detailed Implementation
[0022] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0023] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0024] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.
[0025] This invention provides a weakly aligned multimodal target detection method based on a bidirectional balanced alignment network.
[0026] 1. Construct a multimodal UAV remote sensing image data training set to facilitate effective model learning: The training image datasets used were the DRONEVHICLE and VEDAI multimodal datasets, with a training batch size of 150 per iteration. Multi-scale training was performed on both datasets, including training images ranging from 640 pixels. To enhance the model's representational power with a limited number of instances, a combination of fusion, label smoothing, and randomized affine transformations was employed. The image size for testing was 640 pixels.
[0027] 2. Construct a bidirectional path multimodal UAV remote sensing small target detection network model: The convolutional neural network designed in this invention is as follows: Figure 1 As shown, the input image first extracts multi-scale features through the backbone network. The outputs of stages 3, 4, and 5 in this network are represented as P3, P4, and P5, respectively, and these outputs serve as the basis for subsequent feature fusion. The left side represents the dual-branch backbone network, and the right side represents the Weak Alignment Module (WAM). WAM consists of three collaborative components: First, the Adaptive Cross-Modal Association Module (ACMCM), which comprises global and local modeling, is used to establish cross-modal semantic relevance; second, the Symmetric Offset Generator (SOG) employs a coarse-to-fine strategy combined with modulation operations to generate stable and symmetrical offsets, achieving accurate spatial alignment; finally, the Progressive Fusion Strategy (PFS) adaptively fuses the original features and aligned features using learnable weights. WAM simultaneously constructs symmetrical semantic and spatial correspondences and achieves progressive feature integration. Furthermore, WAM can explicitly alleviate cross-modal misalignment problems and enhance cross-modal feature interaction capabilities.
[0028] 3. Design a weakly aligned module (WAM) in the backbone network to alleviate cross-modal misalignment and enhance feature interaction: In real-world multimodal UAV imagery, RGB and IR images typically exhibit weak alignment characteristics, manifesting not only as spatial positional offsets but also as semantic biases introduced at the feature level. Such biases often lead to a single modality dominating the fusion process, thereby weakening potential complementary information from the other modality. Existing methods often address alignment issues from a single dimension, making it difficult to simultaneously guarantee semantic consistency and spatial correspondence. To overcome these limitations, this method proposes a novel Weak Alignment Module (WAM). This module consists of three specially designed sub-components: an Adaptive Cross-Modal Association Module (ACMCM), a Symmetric Offset Generator (SOG), and a Progressive Fusion Strategy (PFS). WAM explicitly enhances bidirectional information transfer across modalities, constructs symmetrical semantic and spatial correspondences, and achieves stepwise feature fusion, thus obtaining stable and efficient multimodal fusion results without requiring strict image pair registration.
[0029] In WAM, the feature maps of RGB and infrared modes are denoted as follows: and WAM first estimates the bidirectional association weights using ACMCM. and These weights are used to jointly capture global dependencies and local variations, thereby strengthening regions with semantic consistency across modalities. Subsequently, these weights are applied to the original features to generate semantically enhanced feature representations. and This significantly improves cross-modal semantic alignment. Based on this, SOG predicts symmetric coarse-to-fine offsets through modulation and normalization mechanisms. These offsets are used to guide deformable convolutions to compensate for spatial misalignment between modalities, thereby generating spatially aligned features. and .
[0030] Features after re-fusion By progressively integrating the original and aligned features with learnable balanced weights, the network preserves modality-specific information while ensuring spatial consistency and semantic integrity. The final output features are then obtained. .
[0031] In summary, WAM provides an efficient solution for multimodal remote sensing target detection under weak alignment conditions. This module explicitly alleviates the problem of asymmetric utilization of cross-modal information by introducing balanced correlation modeling, stable offset learning, and adaptive feature fusion within a unified framework.
[0032] 4. The Adaptive Cross-Modal Association Module (ACMCM) constructs bidirectional semantic correspondences by jointly modeling global dependencies and local similarities: In weakly aligned multimodal scenarios, simple feature concatenation or unidirectional correlation often leads to semantic bias, where one modality dominates the fusion process. Traditional cross-modal correlation methods primarily rely on local feature similarity, ignoring the asymmetry in reliability between RGB and IR modalities. Although some studies have introduced global attention to enhance cross-modal interaction, they typically utilize only a single factor from global correlation or local spatial similarity, still resulting in unbalanced and semantically inconsistent representations between the two modalities. To mitigate this asymmetry, this invention constructs an Adaptive Cross-Modal Correlation Module (ACMCM), which simultaneously models global dependencies and local spatial information in a unified correlation estimator. This design achieves balanced and bidirectional information interaction between modalities, thereby improving semantic consistency. Simultaneously, it suppresses redundant modality-specific responses and provides stable and reliable correlation priors for subsequent alignment processes. Figure 2 The structure of ACMCM is shown.
[0033] The relevance weights of ACMCM are calculated in three steps. First, global average pooling (GAP) is applied by concatenating features from two modalities to aggregate global dependencies across modalities. This process can be represented as:
[0034] in, Indicates global average pooling. Indicates channel splicing. Two-story Convolutional structure This is the Sigmoid activation function. The channel weights generated by this operation... It can highlight semantic features shared across modalities.
[0035] Subsequently, to supplement the global perspective, we further extract bidirectional local similarities and differences, which can be represented as:
[0036]
[0037] in, Used to characterize cross-modal differences, while This is used to capture the correlation activation between two modes; This indicates element-wise multiplication. Furthermore, in... and The mean pooling and max pooling operations are applied to the above, and the results are concatenated, as shown below:
[0038] in, It is used to highlight areas of spatial consistency across modalities while suppressing noise interference. This represents a lightweight convolution operation consisting of two convolutional layers with non-linear activations. Furthermore, it simultaneously computes... and To achieve two-way spatial relationship modeling.
[0039] Finally, through a learnable balance vector with softmax normalization By merging global branches and spatial branches, adaptive weighted integration of the two types of related information is achieved.
[0040]
[0041]
[0042] in, This represents expanding the spatial feature map along the channel dimension. This adaptive weighting mechanism allows the network to dynamically emphasize global or local information based on the specific nature of the input features. The final relevance weight calculation formula is:
[0043]
[0044] After completing the bidirectional modal association calculation, the semantically aligned feature tensor and Spatial alignment conditions are already met, among which parameters This ensures a moderate level of feature enhancement. In summary, the ACMCM framework establishes a more balanced semantic foundation for subsequent spatial alignment and feature fusion by balancing global dependencies and local mutations.
[0045] 5. The Symmetric Offset Generator (SOG) employs a coarse-to-fine strategy to generate stable and symmetrical offsets, achieving accurate and robust spatial alignment: Although ACMCM alleviates semantic inconsistency, spatial differences still exist in cross-modal features due to factors such as sensor misalignment, parallax, or changes in viewpoint. These spatial differences lead to inconsistencies in receptive fields and degradation of correspondences, which is particularly evident in small or distant targets. Traditional offset prediction methods are usually based on deformation estimation in only one direction or at one scale, which easily leads to alignment instability and the dominance of one mode. To address this, this invention proposes a Symmetric Offset Generator (SOG), whose core includes a coarse-to-fine offset estimation strategy and a fine-tuning stage: the coarse-grained branch handles large-scale displacements, while the fine-grained branch corrects local misalignments. This symmetric multi-scale design restores positional correspondences through a collaborative working mechanism, effectively reducing modal bias and achieving more balanced spatial alignment. The SOG structure is as follows: Figure 2 As shown.
[0046] First, the SOG module concatenates modality-specific features with their corresponding associated priors, and then performs unified processing on the fused input through shared convolutional blocks:
[0047] in, Indicates by Convolution (responsible for channel projection) and Convolutional blocks consisting of convolutions (combined with ReLU activation to capture local cross-modal context); the final output feature map Joint cross-modal representations were encoded.
[0048] The module then generates two complementary offset branches to achieve a coarse-to-fine alignment process. The first branch passes through a convolutional predictor. Generate coarse-grained offset This is primarily used to compensate for large-scale spatial displacements caused by sensor misalignment or changes in viewing angle. However, such offsets mainly provide global correction and are difficult to eliminate subtle structural inconsistencies. Therefore, the sub-branch employs a lightweight convolutional predictor. Generate fine-grained offset This is specifically optimized for regions with local deformation and fine structures. The final offset is obtained by fusing global displacement correction and local refinement results, and its mathematical expression is as follows:
[0049] in, and These represent the features from the shared feature map. The predicted coarse and fine-grained offsets. The final offset after aggregation is calculated by the following formula:
[0050] Wherein, scaling factor Based on experience, a value of 0.5 is set to balance the contribution of the local refinement process. Subsequently, the stability of the offset estimation is improved by a unified processing step that combines modulation and normalization.
[0051] Specifically, firstly, by using a Sigmoid activation function Convolutional predictor Based on splicing features Predictive modulation operation. This operation applies to the fundamental offset element-wise. This effectively suppresses spatially unstable regions. Furthermore, to avoid excessive deformation, the modulated offset is normalized and scaled using a hyperbolic tangent function. The final refined offset is calculated using the following formula:
[0052] in, This represents the standard deviation calculated along each sample spatial dimension. Finally, a bidirectional deformable convolution guided by a refined offset is used to generate spatially aligned feature representations by deforming the enhanced feature map.
[0053]
[0054] in, This represents a deformable convolution operator. This step ultimately generates spatially aligned RGB features. infrared features By explicitly coupling offset prediction and deformable alignment mechanisms, SOG achieves cross-modal symmetric spatial alignment, thus providing reliable input for subsequent fusion stages.
[0055] Therefore, SOG achieves robust multimodal alignment by jointly estimating symmetric coarse-fine offsets: this mechanism both ensures the stability of deformable convolution and alleviates the asymmetric dependence on a single mode. This design can more accurately correct spatial misalignment problems while maintaining balanced cross-modal information flow.
[0056] 6. Progressive fusion strategy (PFS) integrates multi-stream features through learnable weights, improving spatial consistency and semantic complementarity: Despite the achievement of semantic-geometric alignment, cross-modal fusion still faces challenges due to the inherent reliability differences between RGB and infrared modalities. Differences in illumination conditions, texture features, and thermal response characteristics can easily lead to over-reliance on a particular modality, resulting in asymmetric utilization where the dominant modality suppresses complementary information. Existing fusion methods either directly replace the original features, causing a loss of modality-specific details, or simply splice features, introducing redundancy and noise. To address these limitations, this invention proposes a Progressive Fusion Selection (PFS) mechanism that uses an adaptive gating mechanism to perform hierarchical fusion of the original and aligned features. This progressive structure gradually balances modal contributions, improves spatial coherence and semantic integrity, and ultimately generates a balanced and highly discriminative multimodal detection representation under weak misalignment conditions.
[0057] like Figure 3 As shown, PFS employs a hierarchical architecture to integrate multiple feature streams. Specifically, it uses the original features... , and its alignment features , As input, the fusion process proceeds in two stages: the first stage constructs compact refused features through full feature stream concatenation and transform block projection. :
[0058] in, It consists of two cascaded convolutional layers, including batch normalization and ReLU activation operations. This step effectively reduces feature redundancy while enhancing cross-modal interaction.
[0059] In the second stage, the network learns weight vectors. The contribution of the three complementary feature streams is adaptively adjusted, and the vector is normalized using the Softmax function:
[0060] The first term preserves modality-specific information from the original feature stream, the second term enhances spatial consistency through alignment representation, and the third term introduces complementary information from the refused features. By progressively integrating these three components, the final output... To form an equilibrium representation that combines enhanced discriminative power with robustness.
[0061] In summary, this method employs an adaptive weighting mechanism to hierarchically integrate the original features, aligned features, and re-fused features, ensuring balanced participation of both modalities during the fusion process. This progressive design enhances cross-modal consistency while preserving modality-specific information, mitigating the problem of asymmetric information utilization and generating robust multimodal representations with strong generalization capabilities even under weak alignment conditions.
[0062] 7. Implement detection on a multimodal UAV remote sensing dataset: Based on this method, experiments were conducted on the challenging DRONEVHICLE and VEDAI datasets to evaluate the effectiveness of the proposed detection method and different multimodal detection networks on these datasets. As shown in Tables 1 and 2, the proposed method achieved excellent detection performance.
[0063] Table 1. Detection results of different detectors on the DRONEVEHICLE
[0064] Table 2. Detection results of different detectors on VEDAI
[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A weakly aligned multimodal target detection method based on a bidirectional balanced alignment network, characterized in that: Includes the following steps: S1, acquire multimodal image pairs including RGB and infrared images; S2, use a dual-path backbone network to extract the first feature of the RGB image and the second feature of the infrared image respectively; S3, the first feature and the second feature are processed by the weak alignment module WAM to obtain the output feature, wherein the weak alignment module WAM includes an adaptive cross-modal association module ACMCM, a symmetric offset generator SOG and a progressive fusion strategy PFS. S4, Target detection is performed based on the output features, and the detection result is output.
2. The weakly aligned multimodal target detection method based on a bidirectional balanced alignment network according to claim 1, characterized in that: The Adaptive Cross-Modal Association Module (ACMCM) constructs a bidirectional semantic correspondence through the following steps: First, the concatenated first and second features are processed using Global Average Pooling (GAP) to generate global dependency weights. , ,in This indicates the first feature. This indicates the second feature. This indicates a channel splicing operation. For 1×1 convolution weights, Use the Sigmoid activation function; Next, the difference between the first feature and the second feature is calculated. and association ,in , , , , This represents an element-wise multiplication operation, and generates local similarity weights based on mean pooling and max pooling. and ,in ,in This indicates the mean pooling operation. This indicates a max pooling operation. This indicates a lightweight convolution operation; Finally, through learnable balance vectors The global dependency weight and the local similarity weight By merging the data, we can obtain the bidirectional correlation weights. and ,in, , . This represents a channel dimension expansion operation, applied to feature enhancement to obtain semantically enhanced features. and ,in , , This is the preset scaling factor.
3. The weakly aligned multimodal target detection method based on a bidirectional balanced alignment network according to claim 1, characterized in that: The symmetric offset generator SOG employs a coarse-to-fine strategy to generate symmetric offsets, including: First, by sharing convolutional blocks Processing semantic enhancement features after concatenation and Generate shared features ,in ; Secondly, coarse offset is predicted through coarse-grained branching. ,in Predicting fine offsets through fine-grained branching ,in ; Then, the coarse offset is fused. and the fine offset Get the base offset ,in , A preset scaling factor is used; subsequently, modulation and normalization operations are applied to generate the final offset. ,in ,in This indicates a modulation convolution operation. It is the Sigmoid activation function. This indicates the standard deviation calculation operation. This represents an element-wise multiplication operation; Finally, deformable convolution is used based on the final offset. Spatially align the semantically enhanced features to generate spatially aligned features. and ,in , , This represents a deformable convolution operator.
4. The weakly aligned multimodal target detection method based on a bidirectional balanced alignment network according to claim 1, characterized in that: The progressive fusion strategy PFS integrates features through the following steps: First, the original features and Features after spatial alignment and Perform stitching and use projected convolutional blocks Generate re-fusion features ,in ; Secondly, through learnable weight vectors Adaptive fusion of the original feature stream, aligned feature stream, and refused feature stream yields the output feature. ,in ,in Normalized using the Softmax function.
5. The weakly aligned multimodal target detection method based on a bidirectional balanced alignment network according to claim 2 or 3, characterized in that: The preset scaling factor The value is 0.
5.
6. The weakly aligned multimodal target detection method based on a bidirectional balanced alignment network according to claim 1, characterized in that: The dual-path backbone network extracts multi-scale features and outputs features P3 (stage 3), P4 (stage 4), and P5 (stage 5). The weak alignment module WAM is applied to these multi-scale features.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.
8. A weakly aligned multimodal target detection system based on a bidirectional balanced alignment network, characterized in that: include: The image acquisition module is used to acquire multimodal image pairs, including RGB images and infrared images; The feature extraction module includes a dual-path backbone network for extracting the first feature of the RGB image and the second feature of the infrared image, respectively. The weak alignment module (WAM) includes an adaptive cross-modal association module (ACMCM), a symmetric offset generator (SOG), and a progressive fusion strategy (PFS), which are used to process the first feature and the second feature to obtain the output feature. The target detection module is used to perform target detection based on the output features and output the detection results.
9. The weakly aligned multimodal target detection system based on a bidirectional balanced alignment network according to claim 8, characterized in that: The Adaptive Cross-Modal Association Module (ACMCM) is configured to construct a bidirectional semantic correspondence through global dependency weights and local similarity weights. The Symmetric Offset Generator (SOG) is configured to generate symmetric offsets through a coarse-to-fine strategy. The Progressive Fusion Strategy (PFS) is configured to integrate multi-stream features through learnable weight vectors.