Multi-modal fusion object detection method and system based on global semantic concentration perception and local optimal transmission alignment

Abstract

The application discloses a kind of multi-modal fusion target detection method and system based on global semantic condensation perception and local optimal transmission alignment, propose global semantic condensation perception fusion mechanism, by spatial condensation and serialization modeling to cross-modal feature, realize efficient, low-redundancy global semantic fusion;Introduce local optimal transmission alignment module, formalize cross-modal local feature alignment problem as optimal transport problem with constraint, improve the geometric consistency and semantic discriminability of feature alignment;Build the two-stage collaborative fusion architecture of "global fusion+local alignment", give consideration to high-level semantic complementarity and local structure accuracy, effectively alleviate the fusion misalignment and information loss problem in existing method, under the premise of ensuring high reasoning speed and low computational complexity, significantly improve the discriminability and robustness of fusion feature, more suitable for real-time multi-modal target detection task in complex industrial scene.

Description

Technical Field

[0001] This invention relates to the field of visual recognition technology, and in particular to a multimodal fusion target detection method and system based on global semantic condensation perception and local optimal transmission alignment. Background Technology

[0002] Against the backdrop of rapid development in artificial intelligence, visual perception technology has become a core means of enabling intelligent systems to perceive and understand the objective world. As a fundamental research topic in computer vision, object detection aims to achieve category recognition and spatial localization of targets of interest in images or video sequences. Currently, this technology is widely used in key production fields such as autonomous driving, industrial inspection, smart security, and drone remote sensing, serving as a crucial cornerstone for achieving system automation and intelligence. However, in complex real-world application scenarios, the uncontrollability of the environment poses a severe challenge to the robustness of perception algorithms.

[0003] Traditional visual perception solutions often rely on single-modal sensors, but these suffer from significant limitations in extreme conditions. For example, visible light cameras experience a sharp decline in image quality in low light or nighttime environments, making it difficult to capture effective targets; event cameras, while possessing high dynamic range, lack sufficient ability to detect stationary or slow-moving targets; and lidar is susceptible to increased measurement errors due to refraction and scattering interference in adverse weather conditions such as rain and snow. Therefore, developing multi-sensor fusion perception solutions is of significant strategic importance in achieving universal global perception and realizing all-weather, all-time, and all-domain sensing capabilities. Infrared and visible light modalities exhibit strong complementary characteristics: visible light images are rich in color and texture information, while infrared images overcome the limitations of lighting conditions by detecting thermal radiation, enabling stable imaging in all weather conditions. Effectively combining information from these two sensor modalities can significantly improve the detection accuracy and robustness of the perception system in complex environments.

[0004] However, the introduction of multi-sensor data significantly increases the computational burden on the model, making it difficult to implement on embedded or edge devices with limited computing power. How to effectively suppress cross-modal redundant information, achieve efficient fusion of complementary features, and ensure that inference speed meets real-time requirements—that is, to achieve the optimal balance between accuracy and speed—has become a core issue of concern for both academia and industry.

[0005] Currently, research in the field of infrared and visible light target detection is mainly divided into three categories: early fusion, mid-term fusion, and late-term fusion. Early fusion, also known as data-level fusion, is based on the idea of ​​aligning and merging multimodal raw data at the pixel level before feature extraction. This highlights the salient pixel features of infrared targets while preserving high-fidelity environmental details in the visible light environment. This increases the data domain difference between the target foreground and the environmental background, thus assisting downstream tasks such as target detection and instance segmentation.

[0006] Non-patent literature (Ma J, Yu W, Liang P, et al. FusionGAN: A generative adversarial network for infrared and visible image fusion[J]. Informationfusion, 2019, 48: 11-26.) uses a generative adversarial network to fuse two images into a single high-quality synthetic image with rich visible light background texture and significant infrared light targets; intermediate fusion, also known as feature-level fusion, refers to the use of a fusion module to interact and enhance modal features at different depths during the feature extraction of high-level abstract features of each modality in the backbone network.

[0007] Non-patent literature (Qingyun F, Dapeng H, Zhaokui W. Cross-modality fusion transformer for multispectral object detection[J]. arXiv preprint arXiv:2111.00273, 2021.) first utilized the Transformer to capture long-range dependencies across modalities. By simultaneously calculating intra- and extra-domain similarity matrices, it guides the network to generate high-quality fusion global discriminative features, thereby improving the model's detection performance. Late-stage fusion is a decision-level fusion scheme, which allows each modal branch to output prediction results independently, and finally performs weighted integration at the detection box or confidence level.

[0008] Non-patent literature (Chen YT, Shi J, Ye Z, et al. Multimodal object detection via probabilistic ensembling[C] / / European Conference on Computer Vision.Cham: Springer Nature Switzerland, 2022: 139-158.) improves the fault tolerance of the final decision by performing Bayesian probability-based post-processing fusion on multimodal detection results.

[0009] The disadvantages of the existing technology are: The synergistic guiding role of global perception and local refinement in enhancing modal information was ignored; The positive effect of high-level semantic information on modality fusion was ignored; Ignoring the theoretical guidance of mathematical modeling for modal feature alignment; The effect of computing resource allocation on the balance between detection accuracy and inference speed was ignored.

[0010] This invention application is filed to address the shortcomings of existing technologies. Summary of the Invention

[0011] This invention aims to address the core bottlenecks in existing infrared-visible light fusion detection schemes, such as low cross-modal fusion efficiency, inaccurate discriminant feature alignment, and an imbalance between detection accuracy and inference speed. To this end, this invention proposes a multimodal target detection scheme based on global fusion and local alignment. Through key technologies such as global semantic condensation perception fusion and local optimal transmission alignment, it significantly improves detection accuracy and system robustness while ensuring high real-time performance of the algorithm, better meeting the high scalability requirements of complex industrial production scenarios.

[0012] To achieve the above-mentioned objectives, the technical solution provided by this invention is: a multimodal fusion target detection method based on global semantic condensation perception and local optimal transmission alignment, comprising the following steps: Step S1: Construct the overall network structure, including: a pre-trained visible light image feature extraction network M1; a pre-trained infrared image feature extraction network M2; a fusion network M3 for global modal features; a local feature alignment network M4; and a detection network M5 for outputting the final detection result. The detection network M5 contains two fully connected layers of different dimensions, which are used to predict the target's location information and category information, respectively. Step S2: Pre-train the feature extraction models of visible light image feature extraction network M1 and infrared image feature extraction network M2 to obtain the single-modal features F1 of the visible light image and the single-modal features F2 of the infrared image, respectively. Step S3: The global semantic condensation and perception fusion module performs fusion of features from different modalities at the global level; specifically: Step S30: The single-modal feature F1 of the visible light image output by the feature extraction model of the visible light image feature extraction network M1 and the single-modal feature F2 of the infrared image output by the feature extraction model of the infrared image feature extraction network M2 are spatially compressed by downsampling or pooling operations. Step S31: Expand into a spatial sequence and inject learnable positional codes, then concatenate to form a joint semantic sequence; Step S32: Obtain query vector Q, key vector K, and value vector V through three transformation matrices of equal dimensions; Step S33: Apply Softmax normalization to query vector Q and key vector K in the row and column dimensions of the space, respectively. Step S34: Calculate the result of each key vector K and value vector V using dot product to obtain the global condensed semantic correlation matrix, which is used to dynamically modulate the semantic response intensity of each vector channel of Q. Step S35: The preliminary fusion feature F3 is output through the multilayer perceptron and residual connection; Step S36: Iterate through the process, repeating steps S30-S35 to facilitate deep interaction and aggregation of cross-modal semantics; Step S4: After separating the preliminary fusion feature F3 obtained in step S3 from the spatial dimension and performing dimensional transformation, based on the correspondence between the feature element coordinates and the single-mode feature F1 of the visible light image and the single-mode feature F2 of the infrared image before stitching, the visible light dominant mode feature F4 and the infrared dominant mode feature F5 are obtained respectively. Step S5, feature alignment; After fusion in step S4, the features and the original features are inconsistent in terms of feature dimension and semantic expression strength. Therefore, features with two different distributions are fed into a locally optimal transmission alignment module for feature alignment. Specifically: In step S50, in the local optimal transmission alignment module, the single-mode feature F1 and the dominant visible light mode feature F4 of the visible light image, and the single-mode feature F2 and the dominant infrared mode feature F5 of the infrared image are expanded into feature sequences in the spatial plane, and the feature sequences are defined as two discrete probability distributions respectively. Step S51: Define a custom adaptive transportation cost function and construct an element-wise transportation loss matrix; Step S52 introduces relaxed convex constraints and solves the feasible transportation matrix that minimizes the total transportation cost. The feasible transportation matrix is ​​used to guide the redistribution and alignment of the original modal features and fused features in the local space, thereby effectively improving the spatial consistency and semantic fidelity of discriminative features. In step S53, the finally aligned features are fed back to the backbone network to generate higher-level semantic features, and then added and fused together before being sent to the detection network M5. Step S6: Detect the detection results output by network M5; The detection network M5 consists of multiple convolutional layers and fully connected layers, and outputs predicted category, location, and prediction confidence information to the fused feature map.

[0013] The preferred technical solution provided by this invention is as follows: In step S1, both the visible light image feature extraction network M1 and the infrared image feature extraction network M2 contain convolutional layers, global average pooling layers, and residual structure components of different sizes, and adopt one of the VGG, ResNet, and CSPDarknet network models.

[0014] A further preferred technical solution provided by the present invention is as follows: In step S2, the image size of both the visible light image feature extraction network M1 and the infrared image feature extraction network M2 will be uniformly adjusted to 640 during training. The images are normalized, with pixel values ​​scaled to the range of 0 to 1. During the initialization of the pre-trained model, the backbone network weights are retained, while the fully connected layer parameters related to the original classification task are discarded to adapt to the downstream detection task.

[0015] A further preferred technical solution provided by the present invention is as follows: In step S6, the detection network M5 further includes a pre-training step, specifically: The detection network M5 uses the binary cross-entropy loss function to measure the model's confidence in the existence of the target and its category prediction, and uses the EIOU loss function to calculate the difference between the model's predicted box position and the ground truth box. Furthermore, the detection network M5 uses random flipping as a data augmentation method to improve the network training effect.

[0016] This invention also provides a multimodal fusion target detection system based on global semantic condensation perception and local optimal transmission alignment, comprising: The feature extraction module includes feature extraction of visible light image features, used to obtain the single-modal feature F1 of the visible light image, and feature extraction of infrared image features, used to obtain the single-modal feature F2 of the infrared image. The global semantic condensation perception fusion module realizes the fusion of different modal features at the global level to obtain the preliminary fused feature F3; The local optimal transport alignment module aligns the fused features with the original features using two different feature distributions. The feature sequences are defined as two discrete probability distributions, a custom adaptive transport cost function is defined, an element-level transport loss matrix is ​​constructed, and a relaxed convex constraint is introduced to solve for the feasible transport matrix that minimizes the total transport cost. The feasible transport matrix is ​​used to guide the redistribution and alignment of the original modal features and the fused features in the local space. The detection output module detects the detection results output by the M5 network. The detection network M5 consists of multiple convolutional layers and fully connected layers, and outputs predicted category, location, and prediction confidence information to the fused feature map.

[0017] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the methods described in the above embodiments.

[0018] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described in the above embodiments.

[0019] Beneficial effects: This invention subdivides the feature fusion scheme into two interaction scales: global fusion and local alignment. In the global fusion stage, this invention designs a global semantic condensation and perception fusion module. This module achieves efficient cross-modal information fusion by modeling the bimodal long-range semantic correlation of the condensed modalities at the spatial scale. In the local alignment stage, this invention designs a locally optimal transmission alignment module. By defining feature alignment as an operations research-based optimal transmission problem, abstracting the two features into probability distributions, defining a custom transport function, establishing relaxed convex constraints, and obtaining feasible solutions to construct the transport matrix, this invention achieves efficient semantic aggregation and precise spatial alignment of cross-modal features through the dual mechanisms of "global condensation and perception fusion" and "local optimal transmission alignment." This effectively enhances the quality of discriminable features, avoiding information redundancy and semantic confusion caused by traditional element-by-element fusion, and overcoming the limitations of simple attention mechanisms in local structural alignment. Thus, while maintaining low computational complexity and high inference speed, it significantly improves the accuracy and robustness of target detection in complex environments, demonstrating good engineering application prospects and industrial promotion value. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart of the multimodal fusion target detection method based on global semantic condensation perception and local optimal transmission alignment of the present invention. Figure 2 This is a block diagram of the global semantic condensation perception fusion module method in this invention; Figure 3 This is a block diagram of the local optimal transmission alignment module method in this invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0022] Figure 1 This is a flowchart illustrating the overall process of the multimodal fusion target detection method based on global semantic condensation perception and local optimal transmission alignment of the present invention. Figure 1 As shown, the present invention provides a multimodal fusion target detection method based on global semantic condensation perception and local optimal transmission alignment, comprising the following steps: Step S1: Construct the overall network structure, including: a pre-trained visible light image feature extraction network M1; a pre-trained infrared image feature extraction network M2; a fusion network M3 for global modal features; a local feature alignment network M4; and a detection network M5 for outputting the final detection result. The detection network M5 contains two fully connected layers of different dimensions, which are used to predict the target's location information and category information, respectively. The aforementioned network includes convolutional layers of different sizes, global average pooling layers, residual structures, and other components. In this invention, the pre-trained feature extraction networks M1 and M2 can use any classic network, such as VGG, ResNet, CSPDarkNet, and other network models.

[0023] In the specific implementation process, image datasets with category labels are used. (in The total number of samples, For the first Zhang input image, For The backbone feature extraction network is pre-trained using the corresponding label vectors. The network structure includes fully convolutional layers, global average pooling layers, and two fully connected classification heads, ultimately outputting the predicted probability distribution for each target category. During training, data augmentation strategies such as multi-scale scaling, random cropping, and horizontal flipping are applied to the input image to improve the model's generalization ability. In the subsequent feature extraction stage, only the backbone network parameters are loaded, the fully connected layers are discarded, and multi-scale deep semantic features are output for cross-modal fusion.

[0024] Step S2: Pre-train the feature extraction models of visible light image feature extraction network M1 and infrared image feature extraction network M2 to obtain the single-modal features F1 of the visible light image and the single-modal features F2 of the infrared image, respectively. In step S2, the image size of both the visible light image feature extraction network M1 and the infrared image feature extraction network M2 will be uniformly adjusted to 640 during training. The images are normalized, with pixel values ​​scaled to the range of 0 to 1. During the initialization of the pre-trained model, the backbone network weights are retained, while the fully connected layer parameters related to the original classification task are discarded to adapt to the downstream detection task.

[0025] Step S3: The global semantic condensation and perception fusion module performs fusion of features from different modalities at the global level; specifically: Step S30: The single-modal feature F1 of the visible light image output by the feature extraction model of the visible light image feature extraction network M1 and the single-modal feature F2 of the infrared image output by the feature extraction model of the infrared image feature extraction network M2 are spatially compressed by downsampling or pooling operations. Step S31: Expand into a spatial sequence and inject learnable positional codes, then concatenate to form a joint semantic sequence; Step S32: Obtain query vector Q, key vector K, and value vector V through three transformation matrices of equal dimensions; Step S33: Apply Softmax normalization to query vector Q and key vector K in the row and column dimensions of the space, respectively. Step S34: Calculate the result of each key vector K and value vector V using dot product to obtain the global condensed semantic correlation matrix, which is used to dynamically modulate the semantic response intensity of each vector channel of Q. Step S35: The preliminary fusion feature F3 is output through the multilayer perceptron and residual connection; Step S36: Iterate through the process, repeating steps S30-S35 to facilitate deep interaction and aggregation of cross-modal semantics; Step S4: After separating the preliminary fusion feature F3 obtained in step S3 from the spatial dimension and performing dimensional transformation, based on the correspondence between the feature element coordinates and the single-mode feature F1 of the visible light image and the single-mode feature F2 of the infrared image before stitching, the visible light dominant mode feature F4 and the infrared dominant mode feature F5 are obtained respectively. Figure 2 This is a block diagram of the global semantic condensation perception fusion module method in this invention, combined with... Figure 2 As shown, this invention uses a semantic condensation-based perceptual attention mechanism to efficiently aggregate high-level semantic information between infrared and visible light modalities. Infrared and visible light image features are denoted as... (Where H is the feature map height, W is the feature map width, and C is the number of feature map channels). To reduce redundancy and improve computational efficiency, semantic condensation is first performed using dynamic global average pooling and spatial concatenation operations.

[0026] in This represents the scaling factor for the feature map. This indicates dynamic adaptive global average pooling. This represents the splicing of different features in a spatial dimension. This represents the scaled, expanded, and concatenated modal feature sequence (where n is the length of the feature sequence). Then, the query vector Q, key vector K, and value vector V are obtained through three linear transformation matrices with shared weights. Specifically:

[0027] in This represents the learnable parameter matrix. Next, we use dot product to calculate the result of each key vector K-value vector V, and obtain the global semantic association matrix. :

[0028] in This represents the softmax normalized activation function applied along the column dimension of the space, where i represents the index of the attention partition head. This matrix is ​​used to dynamically modulate the query vector. The channel response intensity is used to generate preliminary fusion features. As shown below:

[0029] in Π represents the softmax normalized activation function applied in the row dimension, h represents the channel-dimensional stacking of the results from different attention heads, and h represents the number of heads. Finally, the final fused features are output through two layers of linear transformation with ReLU activation.

[0030] This module effectively achieves long-range dependency modeling and efficient condensation fusion of cross-modal semantics, significantly reducing redundant computations while strengthening complementary semantic expression.

[0031] Step S5, feature alignment; After fusion in step S4, the features and the original features are inconsistent in terms of feature dimension and semantic expression strength. Therefore, features with two different distributions are fed into a locally optimal transmission alignment module for feature alignment. Specifically: In step S50, in the local optimal transmission alignment module, the single-mode feature F1 and the dominant visible light mode feature F4 of the visible light image, and the single-mode feature F2 and the dominant infrared mode feature F5 of the infrared image are expanded into feature sequences in the spatial plane, and the feature sequences are defined as two discrete probability distributions respectively. Step S51: Define a custom adaptive transportation cost function and construct an element-wise transportation loss matrix; Step S52 introduces relaxed convex constraints and solves the feasible transportation matrix that minimizes the total transportation cost. The feasible transportation matrix is ​​used to guide the redistribution and alignment of the original modal features and fused features in the local space, thereby effectively improving the spatial consistency and semantic fidelity of discriminative features. In step S53, the finally aligned features are fed back to the backbone network to generate higher-level semantic features, and then added and fused together before being sent to the detection network M5. Figure 3 This is a block diagram of the local optimal transmission alignment module method in this invention, combined with... Figure 3 As shown, this invention innovatively formalizes the local feature alignment problem into an optimal transport optimization problem with relaxed constraints, thereby improving the spatial consistency and discriminativeness of cross-modal features. Let the fused and original features be denoted as . and ,in , Represents the length of the feature sequence. The channel dimension represents the feature sequence. In the optimal transmission problem, the optimization objective is defined as follows:

[0032] in Represents the transmission weight matrix. The matrix represents the pointwise loss matrix, and the product of these matrices represents the overall loss of transmission alignment. Normalized cosine distance is used to preserve numerical boundedness:

[0033] To reduce computational complexity and ensure real-time inference, relaxed convex constraints are introduced:

[0034] Where i is the row index and j is the column index. Under this constraint, the optimization problem is a convex programming problem with a unique global optimum, which can be expressed as:

[0035] Finally, the fused features are redistributed using the transport matrix to obtain the aligned features:

[0036] This module achieves precise and interpretable local structure alignment through mathematical modeling, effectively suppressing feature misalignment caused by modal differences, thereby enhancing the robustness and accuracy of downstream detection tasks.

[0037] Step S6: Detect the detection results output by network M5; The detection network M5 consists of multiple convolutional layers and fully connected layers, and outputs predicted category, location, and prediction confidence information to the fused feature map.

[0038] In step S6, the detection network M5 further includes a pre-training step, specifically: The detection network M5 uses the binary cross-entropy loss function to measure the model's confidence in the existence of the target and its category prediction, and uses the EIOU loss function to calculate the difference between the model's predicted box position and the ground truth box. Furthermore, the detection network M5 uses random flipping as a data augmentation method to improve the network training effect.

[0039] This invention decomposes the cross-modal feature fusion process into two collaborative stages: global semantic aggregation and local structural alignment. These stages achieve efficient and accurate feature interaction from the perspectives of high-level semantic complementarity and local spatial consistency, respectively. Specifically, the proposed global semantic condensation perception fusion mechanism, through spatial compression and dual-dimensional normalized attention modeling, significantly reduces computational redundancy while strengthening cross-modal semantic association. The designed local optimal transport alignment module formalizes the feature matching problem into a constrained optimal transport problem, achieving mathematically interpretable and structurally accurate local alignment. The solution provided by this invention effectively improves detection accuracy, generalization ability, and system robustness in complex environments while ensuring high inference speed and low computational overhead.

[0040] This invention proposes a global semantic condensation and perception fusion mechanism, which achieves efficient and low-redundancy global semantic fusion by spatially condensing and serializing cross-modal features. A local optimal transport alignment module is introduced to formalize the cross-modal local feature alignment problem into a constrained optimal transport problem, thereby improving the geometric consistency and semantic discriminability of feature alignment. We construct a two-stage collaborative fusion architecture of "global fusion + local alignment" to balance the complementarity of high-level semantics and the accuracy of local structure, effectively alleviating the problems of fusion inaccuracy and information loss in existing methods. The key difference of this invention lies in two aspects: First, it explicitly divides the cross-modal fusion process into two complementary stages: global semantic condensation perception and local optimal transport alignment. These stages achieve efficient and accurate feature interaction at both the overall semantic level and the local structural level, avoiding semantic confusion or loss of detail caused by coarse-grained fusion at a single scale in traditional fusion methods. Second, this invention introduces a semantic condensation perception attention mechanism in the global stage, effectively modeling long-range semantic dependencies across modalities while significantly reducing feature dimensionality and computational overhead. In the local stage, feature alignment is formalized as an optimal transport problem with relaxed constraints, achieving mathematically interpretable and structurally aligned feature matching by solving the transport matrix. Therefore, the solution provided by this invention significantly improves the discriminativity and robustness of fused features while ensuring high inference speed and low computational complexity. It effectively overcomes the core problems of low efficiency in cross-modal fusion, inaccurate alignment of key features, and difficulty in balancing accuracy and speed in existing technologies, making it more suitable for real-time multimodal target detection tasks in complex industrial scenarios.

[0041] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0042] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0043] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause an electronic device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0044] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal fusion target detection method based on global semantic condensation perception and local optimal transmission alignment, characterized in that, Includes the following steps: Step S1: Construct the overall network structure. This includes a pre-trained visible light image feature extraction network M1 and a pre-trained infrared image feature extraction network M2. M3 is a fusion network for global modality features; M4 is a local feature alignment network. The detection network M5 is used to output the final detection results. The detection network M5 contains two fully connected layers with different dimensions, which are used to predict the target's location information and category information, respectively. Step S2: Pre-train the feature extraction models of visible light image feature extraction network M1 and infrared image feature extraction network M2 to obtain the single-modal features F1 of the visible light image and the single-modal features F2 of the infrared image, respectively. Step S3: The global semantic condensation perception fusion module realizes the fusion of different modal features at the global level; Specifically: Step S30: The single-modal feature F1 of the visible light image output by the feature extraction model of the visible light image feature extraction network M1 and the single-modal feature F2 of the infrared image output by the feature extraction model of the infrared image feature extraction network M2 are spatially compressed by downsampling or pooling operations. Step S31: Expand into a spatial sequence and inject learnable positional codes, then concatenate to form a joint semantic sequence; Step S32: Obtain query vector Q, key vector K, and value vector V through three transformation matrices of equal dimensions; Step S33: Apply Softmax normalization to query vector Q and key vector K in the row and column dimensions of the space, respectively. Step S34: Calculate the result of each key vector K and value vector V using dot product to obtain the global condensed semantic correlation matrix, which is used to dynamically modulate the semantic response intensity of each vector channel of Q. Step S35: The preliminary fusion feature F3 is output through the multilayer perceptron and residual connection; Step S36: Iterate through the process, repeating steps S30-S35 to facilitate deep interaction and aggregation of cross-modal semantics; Step S4: After separating the preliminary fusion feature F3 obtained in step S3 from the spatial dimension and performing dimensional transformation, based on the correspondence between the feature element coordinates and the single-mode feature F1 of the visible light image and the single-mode feature F2 of the infrared image before stitching, the visible light dominant mode feature F4 and the infrared dominant mode feature F5 are obtained respectively. Step S5, feature alignment; After fusion in step S4, the features and the original features are inconsistent in terms of feature dimension and semantic expression strength. Therefore, features with two different distributions are fed into a locally optimal transmission alignment module for feature alignment. Specifically: In step S50, in the local optimal transmission alignment module, the single-mode feature F1 and the dominant visible light mode feature F4 of the visible light image, and the single-mode feature F2 and the dominant infrared mode feature F5 of the infrared image are expanded into feature sequences in the spatial plane, and the feature sequences are defined as two discrete probability distributions respectively. Step S51: Define a custom adaptive transportation cost function and construct an element-wise transportation loss matrix; Step S52 introduces relaxed convex constraints and solves the feasible transportation matrix that minimizes the total transportation cost. The feasible transportation matrix is ​​used to guide the redistribution and alignment of the original modal features and fused features in the local space, thereby effectively improving the spatial consistency and semantic fidelity of discriminative features. In step S53, the finally aligned features are fed back to the backbone network to generate higher-level semantic features, and then added and fused together before being sent to the detection network M5. Step S6: Detect the detection results output by network M5; The detection network M5 consists of multiple convolutional layers and fully connected layers, and outputs predicted category, location, and prediction confidence information to the fused feature map.

2. The multimodal fusion target detection method based on global semantic condensation perception and local optimal transmission alignment according to claim 1, characterized in that, In step S1, both the visible light image feature extraction network M1 and the infrared image feature extraction network M2 contain convolutional layers, global average pooling layers, and residual structure components of different sizes, and adopt one of the VGG, ResNet, and CSPDarknet network models.

3. The multimodal fusion target detection method based on global semantic condensation perception and local optimal transmission alignment according to claim 1, characterized in that, In step S2, the image size of both the visible light image feature extraction network M1 and the infrared image feature extraction network M2 will be uniformly adjusted to 640 during training. The images are normalized, with pixel values ​​scaled to the range of 0 to 1. During the initialization of the pre-trained model, the backbone network weights are retained, while the fully connected layer parameters related to the original classification task are discarded to adapt to the downstream detection task.

4. The multimodal fusion target detection method based on global semantic condensation perception and local optimal transmission alignment according to claim 1, characterized in that, In step S6, the detection network M5 further includes a pre-training step, specifically: The detection network M5 uses the binary cross-entropy loss function to measure the model's confidence in the existence of the target and its category prediction, and uses the EIOU loss function to calculate the difference between the model's predicted box position and the ground truth box. Furthermore, the detection network M5 uses random flipping as a data augmentation method to improve the network training effect.

5. A multimodal fusion target detection system based on global semantic condensation perception and local optimal transmission alignment as described in any one of claims 1-4, characterized in that, include: The feature extraction module includes feature extraction of visible light image features, used to obtain the single-modal feature F1 of the visible light image, and feature extraction of infrared image features, used to obtain the single-modal feature F2 of the infrared image. The global semantic condensation perception fusion module realizes the fusion of different modal features at the global level to obtain the preliminary fused feature F3; The local optimal transport alignment module aligns the fused features with the original features using two different feature distributions. The feature sequences are defined as two discrete probability distributions, a custom adaptive transport cost function is defined, an element-level transport loss matrix is ​​constructed, and a relaxed convex constraint is introduced to solve for the feasible transport matrix that minimizes the total transport cost. The feasible transport matrix is ​​used to guide the redistribution and alignment of the original modal features and the fused features in the local space. The detection output module detects the detection results output by the M5 network. The detection network M5 consists of multiple convolutional layers and fully connected layers, and outputs predicted category, location, and prediction confidence information to the fused feature map.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method described in claims 1 to 5.

Images