Multi-modal image fusion identification method based on comparative learning

Abstract

The invention discloses a multi-modal image fusion identification method based on comparative learning, and belongs to the technical field of artificial intelligence and machine learning, and the method comprises the steps: obtaining a first modal image and a second modal image; processing the first modal image and the second modal image through a preset first feature extraction network and a preset second feature extraction network, generating a first feature graph and a second feature graph, generating a cross-modal relation graph with correlation strength among feature points, and determining a positive sample pair and a negative sample pair containing feature point combinations; adjusting the parameters of the network by using a contrast learning target, so that the distance between the feature points in the positive sample pair is reduced, and the distance between the feature points in the negative sample pair is increased; and based on the cross-modal relation graph, carrying out weighted fusion on the first feature graph and the second feature graph to generate a fused feature graph, and inputting the fused feature graph into a preset identification network to obtain an identification result. According to the method, a dynamic cross-modal relation graph guided comparative learning and feature weighted fusion mechanism is adopted, end-to-end adaptive feature alignment can be realized, and the recognition precision in a complex scene is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and machine learning technology, and in particular to a multimodal image fusion and recognition method based on contrastive learning. Background Technology

[0002] Currently, multimodal image fusion recognition technology aims to integrate complementary information from different imaging modalities such as visible light, infrared, and radar to enhance the perception capabilities of computer vision systems in complex scenes. This type of technology falls under the category of intelligent computing systems based on machine learning models and is widely used in fields such as autonomous driving, security monitoring, and medical diagnosis.

[0003] Current mainstream methods mainly employ two approaches: one is an early fusion strategy based on feature concatenation or weighted averaging, which directly merges features extracted from different modalities at the input or intermediate layers; the other is to extract features using a pre-trained single-modal model and then fuse them through attention mechanisms or simple similarity calculations. Some approaches introduce contrastive learning, but this is only applied to single-modal pre-training or as an auxiliary loss function.

[0004] The above-mentioned approach has significant limitations. Direct feature merging easily dilutes key information and introduces noise, leading to a decrease in the discriminative power of the fused features; hand-designed fusion rules are difficult to adapt to modal differences in complex scenes; single-modal contrastive learning cannot uncover cross-modal semantic associations; existing cross-modal contrastive methods rely on fixed sample pairs to define strategies and lack the ability to model dynamic association strength. These problems limit the fusion effect, especially under challenging conditions such as low light and high noise, where recognition performance degrades sharply. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a multimodal image fusion and recognition method based on contrastive learning. It employs a contrastive learning and feature weighted fusion mechanism guided by a dynamic cross-modal relationship graph, which enables end-to-end adaptive feature alignment, noise suppression, and complementary information enhancement, significantly improving recognition accuracy and robustness in complex scenarios.

[0006] The above objectives can be achieved through the following approach: A multimodal image fusion and recognition method based on contrastive learning includes: acquiring a first modal image and a second modal image; processing the first modal image through a preset first feature extraction network to generate a first feature map, and processing the second modal image through a preset second feature extraction network to generate a second feature map; dynamically generating a cross-modal relationship map representing the correlation strength between feature points based on the first and second feature maps; determining positive sample pairs containing feature point combinations with correlation strength higher than a preset first threshold and negative sample pairs containing feature point combinations with correlation strength lower than a preset second threshold between the first and second feature maps based on the cross-modal relationship map; adjusting the parameters of the first and second feature extraction networks using a contrastive learning objective to reduce the distance between feature points in the positive sample pairs and increase the distance between feature points in the negative sample pairs; weightedly fusing the first and second feature maps based on the cross-modal relationship map to generate a fused feature map; and inputting the fused feature map into a preset recognition network to obtain a recognition result.

[0007] Optionally, the step of processing the first modality image through a preset first feature extraction network to generate a first feature map, and processing the second modality image through a preset second feature extraction network to generate a second feature map includes: obtaining a first multi-scale convolutional kernel of the first feature extraction network and a second multi-scale convolutional kernel of the second feature extraction network; extracting multi-level spatial features of the first modality image based on the first multi-scale convolutional kernel to generate a first primary feature map, and extracting multi-level spectral features of the second modality image based on the second multi-scale convolutional kernel to generate a second primary feature map; performing feature enhancement on the first primary feature map to generate an enhanced first primary feature map, and performing feature optimization on the second primary feature map to generate an optimized second primary feature map; performing adaptive max pooling sampling on the enhanced first primary feature map to generate a first feature map, and performing adaptive average pooling sampling on the optimized second primary feature map to generate a second feature map.

[0008] Optionally, the dynamic generation of a cross-modal relationship graph characterizing the correlation strength between each feature point includes: calculating the similarity between each feature point in the first feature graph and all feature points in the second feature graph; generating a relationship vector associated with the second feature graph for each feature point in the first feature graph based on the similarity; and combining the relationship vectors to dynamically generate a cross-modal relationship graph.

[0009] Optionally, determining positive and negative sample pairs containing feature point combinations between the first feature map and the second feature map includes: normalizing the correlation strength in the cross-modal relationship graph to obtain a standardized relationship matrix; selecting feature point combinations in the first and second feature maps with a correlation strength higher than a preset first threshold as positive sample pairs based on the standardized relationship matrix; and selecting feature point combinations in the first and second feature maps with a correlation strength lower than a preset second threshold as negative sample pairs based on the standardized relationship matrix. Optionally, adjusting the parameters of the first feature extraction network and the second feature extraction network using the contrastive learning objective includes: constructing a contrastive loss function based on the positive sample pair and the negative sample pair; optimizing the contrastive loss function using gradient descent to adjust the parameters of the first feature extraction network and the second feature extraction network; iteratively updating the network parameters until the distance between the feature points of the positive sample pair is lower than a preset convergence threshold, and the distance between the feature points of the negative sample pair is higher than a preset separation threshold.

[0010] Optionally, generating the fused feature map includes: using the values ​​representing the association strength in the cross-modal relationship graph as weights; obtaining the feature values ​​at corresponding positions in the first feature map and the second feature map respectively; using the weights to perform a weighted summation of the feature values ​​at corresponding positions in the first feature map and the second feature map to obtain a fused feature value; and performing spatial traversal on the first feature map and the second feature map based on the fused feature value to generate the fused feature map.

[0011] Optionally, the step of inputting the fused feature map into a preset recognition network to obtain a recognition result includes: inputting the fused feature map into a preset recognition network, extracting high-level semantic features through multi-layer nonlinear transformation of the recognition network; and generating a recognition result associated with the first modality image and the second modality image based on the high-level semantic features. Optionally, the method further includes: performing feature extraction based on the recognition result to obtain a recognition loss function; and combining the contrast loss function and the recognition loss function to jointly adjust the parameters of the first feature extraction network, the second feature extraction network, and the recognition network.

[0012] Optionally, the method further includes: performing weighted evaluation based on the contrast loss function and the identification loss function to generate a total loss function; and using backpropagation based on the total loss function to calculate and adjust the parameters of the network.

[0013] Based on the same inventive concept, this invention also provides a multimodal image fusion recognition system based on contrastive learning. The system includes: an image acquisition module for acquiring a first modal image and a second modal image; a feature extraction module for processing the first modal image through a preset first feature extraction network to generate a first feature map, and processing the second modal image through a preset second feature extraction network to generate a second feature map; a cross-modal relationship graph generation module for dynamically generating a cross-modal relationship graph representing the correlation strength between feature points based on the first feature map and the second feature map; and a sample pair determination module for determining samples based on the first feature map and the second feature map. A cross-modal relationship graph is used to identify positive and negative sample pairs containing feature point combinations between the first and second feature maps. A contrastive learning optimization module adjusts the parameters of the first and second feature extraction networks using a contrastive learning objective, reducing the distance between feature points in the positive sample pairs and increasing the distance between feature points in the negative sample pairs. A feature fusion module performs weighted fusion of the first and second feature maps based on the cross-modal relationship graph to generate a fused feature map. A recognition result output module inputs the fused feature map into a preset recognition network to obtain the recognition result.

[0014] Compared with the prior art, the present invention has the following advantages: 1. This invention achieves adaptive and precise alignment of cross-modal features. By dynamically generating cross-modal relationship graphs, it automatically quantifies the semantic association strength between features of different modalities, solving the alignment deviation problem caused by the reliance on fixed rules in traditional methods, and significantly improving the feature matching accuracy of heterogeneous images such as visible light and infrared.

[0015] 2. This invention enhances the discriminative power and anti-interference capability of fused features. Based on a contrastive learning target-driven network, it strengthens the similarity of positive samples to key targets and weakens the correlation of negative samples to noise or background, enabling the fusion process to selectively retain complementary information, effectively suppressing single-modal noise interference, and improving recognition robustness in complex environments.

[0016] 3. This invention forms an end-to-end collaborative optimization closed loop. Feature extraction, relationship graph construction, contrastive learning, and the recognition network are deeply coupled through a joint training mechanism. The dynamic relationship graph serves as both a supervisory basis for contrastive learning and a source of weights for feature fusion, achieving synergistic effects across optimization objectives and exceeding the performance limits of independently designed modules.

[0017] 4. This invention reduces reliance on manual prior knowledge and improves generalization. It abandons manually designed fusion rules or alignment templates, and instead uses data-driven adaptive learning of multimodal association patterns. This maintains stable performance in scenarios with varying lighting, occlusion, and sensor differences, significantly enhancing its generalization capabilities in fields such as security and healthcare.

[0018] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a multimodal image fusion and recognition method based on contrastive learning according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of multi-scale feature extraction according to an embodiment of the present invention.

[0022] Figure 3 This is a positive and negative sample pair screening diagram according to an embodiment of the present invention.

[0023] Figure 4 This is a feature fusion visualization diagram of an embodiment of the present invention.

[0024] Figure 5 This is a schematic diagram of the structure of a multimodal image fusion and recognition system based on contrastive learning according to an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Reference Figure 1 One embodiment of the present invention proposes a multimodal image fusion recognition method based on contrastive learning. It adopts a contrastive learning and feature weighted fusion mechanism guided by dynamic cross-modal relationship graph, which can achieve end-to-end adaptive feature alignment, noise suppression and complementary information enhancement, significantly improving the recognition accuracy and robustness in complex scenes.

[0027] The method described in this embodiment specifically includes: Acquire the first modality image and the second modality image; The first modality image is processed by a preset first feature extraction network to generate a first feature map; The second modality image is processed by a preset second feature extraction network to generate a second feature map; Based on the first feature map and the second feature map, a cross-modal relationship map that characterizes the correlation strength between each feature point is dynamically generated; Based on the cross-modal relationship graph, positive and negative sample pairs containing feature point combinations are determined between the first feature map and the second feature map; By using a contrastive learning objective to adjust the parameters of the first feature extraction network and the second feature extraction network, the distance between feature points in the positive sample pairs is reduced, and the distance between feature points in the negative sample pairs is increased. Based on the cross-modal relationship graph, the first feature map and the second feature map are weighted and fused to generate a fused feature map; The fused feature map is input into a preset recognition network to obtain the recognition result.

[0028] Specifically, the method first acquires a first modality image and a second modality image, which can be medical images from different imaging devices. Next, the first modality image is processed by a pre-defined first feature extraction network to generate a first feature map, which contains key feature information of the first modality image. Similarly, the second modality image is processed by a pre-defined second feature extraction network to generate a second feature map. Then, based on the first and second feature maps, a cross-modal relationship map is dynamically generated. This relationship map characterizes the correlation strength between feature points in the first and second feature maps. For example, the correlation strength can be calculated by the similarity between feature points; the higher the similarity, the stronger the correlation. Next, positive and negative sample pairs are determined according to the cross-modal relationship map. Positive sample pairs consist of feature points with high correlation strength, while negative sample pairs consist of feature points with low correlation strength. The parameters of the first and second feature extraction networks are adjusted using a contrastive learning objective, reducing the distance between feature points in positive sample pairs and increasing the distance between feature points in negative sample pairs, thereby enhancing the consistency of cross-modal features. Subsequently, the first and second feature maps are weighted and fused based on the cross-modal relationship graph to generate a fused feature map. During the fusion process, feature points with high correlation strength are assigned higher weights. Finally, the fused feature map is input into a pre-defined recognition network to obtain the final recognition result. This method effectively improves the accuracy and recognition performance of multimodal image fusion by dynamically generating cross-modal relationship graphs and contrastive learning. Through dynamic cross-modal relationship graphs and contrastive learning, the accuracy and robustness of information recognition can be significantly improved, while avoiding the subjectivity of manually designed features, making the fusion result more objective and reliable.

[0029] Optionally, the step of processing the first modal image through a preset first feature extraction network to generate a first feature map, and processing the second modal image through a preset second feature extraction network to generate a second feature map includes: Obtain the first multi-scale convolutional kernel of the first feature extraction network and the second multi-scale convolutional kernel of the second feature extraction network; Based on the first multi-scale convolution kernel, multi-level spatial features of the first modality image are extracted to generate a first primary feature map; based on the second multi-scale convolution kernel, multi-level spectral features of the second modality image are extracted to generate a second primary feature map. The first primary feature map is enhanced to generate an enhanced first primary feature map, and the second primary feature map is optimized to generate an optimized second primary feature map. Adaptive max pooling sampling is performed on the enhanced first primary feature map to generate a first feature map, and adaptive average pooling sampling is performed on the optimized second primary feature map to generate a second feature map.

[0030] Specifically, the first feature extraction network mainly includes a first multi-scale convolutional layer, which uses first multi-scale convolutional kernels of different sizes to extract multi-level features. The first multi-scale convolutional kernel refers to the use of convolutional kernels of different sizes, such as 3×3, 5×5, and 7×7, in a single convolutional layer. The feature enhancement module strengthens the features of important regions through an attention mechanism. Adaptive max pooling downsamples the enhanced feature map to retain the most significant features. The first backbone network, such as a residual network, uses residual connections to solve the gradient vanishing problem in deep networks. The second feature extraction network mainly consists of a second multi-scale convolutional layer, similar to the first network, but focusing on using second multi-scale convolutional kernels to extract spectral features. The second multi-scale convolutional kernel refers to using convolutional kernels with different channel numbers or spectral sensitivities to extract multi-level spectral features of the image. Commonly, wide-channel convolutional kernels are used to capture global spectral distribution (e.g., thermal radiation fields), while narrow-channel convolutional kernels are used to extract spectral characteristics of specific bands (e.g., high-temperature regions). A feature optimization module adjusts the feature distribution through normalization to improve intermodal consistency. Adaptive average pooling performs smooth downsampling on the optimized feature map. A second backbone network, such as a visual transformer, uses a global attention mechanism to model long-range dependencies. During initialization, the first and second feature networks predefine the structure of these multi-scale convolutional layers, including convolutional kernels of different sizes and their parameters. These kernels have different sizes to capture multi-level features of the image. Subsequently, the first multi-scale convolutional kernel is acquired and multi-level spatial features of the first modality image are extracted to generate a first primary feature map. The second multi-scale convolutional kernel is acquired and multi-level spectral features of the second modality image are extracted to generate a second primary feature map. For example... Figure 2 As shown, the feature response curves of multimodal images at different scales are illustrated. The three curves represent the following: scale 1.0 captures large-scale features, such as the overall outline of the target; scale 0.5 extracts medium-scale features, such as local structure; and scale 0.25 focuses on small-scale details, such as texture / edges. Feature enhancement operations are performed on the first primary feature map, such as strengthening features in important regions through an attention mechanism, to generate an enhanced first primary feature map. Feature optimization operations are performed on the second primary feature map, such as adjusting the feature distribution through normalization, to generate an optimized second primary feature map. Subsequently, adaptive max-pooling sampling is applied to the enhanced first primary feature map to retain the most salient features to generate the first feature map, while adaptive average pooling sampling is applied to the optimized second primary feature map to smooth the feature distribution and generate the second feature map. This method achieves effective feature representation of images with different modalities through multi-scale feature extraction and adaptive sampling.

[0031] For example, suppose the first modal image is a satellite optical image and the second modal image is an infrared thermal image. First, multi-scale convolutional kernels are used to extract spatial structural features from the optical image to generate a first primary feature map, while the spectral radiation features from the thermal image are extracted to generate a second primary feature map. Attention enhancement is applied to the optical feature map to highlight building area features, and normalization is applied to the thermal image feature map to optimize temperature distribution features. Then, adaptive max pooling is applied to the enhanced optical feature map to preserve building edge features and generate the first feature map, while adaptive average pooling is applied to the optimized thermal image feature map to smooth the temperature distribution and generate the second feature map. Multi-scale feature extraction and adaptive sampling effectively preserve the spatial details of the optical image and the spectral characteristics of the thermal image, providing more representative features for subsequent cross-modal fusion, while avoiding information loss caused by single-scale feature extraction and improving the robustness of feature representation. Optionally, the dynamic generation of a cross-modal relationship graph characterizing the correlation strength between each feature point includes: Calculate the similarity between each feature point in the first feature map and all feature points in the second feature map; Based on the similarity, a relation vector associated with the second feature map is generated for each feature point in the first feature map; By combining the aforementioned relation vectors, a cross-modal relation graph is dynamically generated.

[0032] Specifically, when generating the cross-modal relationship graph, this method first calculates the similarity between each feature point in the first feature graph and all feature points in the second feature graph. The similarity calculation can use the cosine similarity formula to calculate the similarity between two feature points. ,have: , in, In the first feature graph, the first... Feature vectors of 1 feature point; Indicates the second feature graph. Feature vectors of 1 feature point; This indicates performing a dot product operation on two vectors; The first step involves a modulo operation; the similarity score between the two features ranges from -1 to 1. Next, a relation vector is generated for each feature point in the first feature map, associated with the second feature map. Each element of this vector represents the similarity score of the corresponding feature point pair. Finally, all relation vectors are combined in the order of feature points to form a two-dimensional matrix, which is the dynamically generated cross-modal relation graph. This graph visually reflects the correlation strength between the two feature maps at different spatial locations, providing a basis for subsequent weighted fusion. Through this dynamic relation modeling, complex correspondences between cross-modal features can be adaptively captured.

[0033] For example, suppose the first feature map comes from an infrared sensor and contains 5 feature points, and the second feature map comes from a visible light sensor and contains 5 feature points. First, the cosine similarity between the first feature point in the infrared feature map and all 5 feature points in the visible light feature map is calculated, resulting in a 5-dimensional relationship vector. This process is repeated to generate a corresponding relationship vector for each feature point in the infrared feature map. Finally, the five 5-dimensional relationship vectors are combined into a 5x5 cross-modal relationship map. By establishing explicit cross-modal associations, the feature misalignment problem caused by the different imaging principles of infrared and visible light images can be effectively solved, significantly improving the accuracy and robustness of multi-source information fusion, especially in target matching and scene understanding tasks.

[0034] Optionally, determining the positive and negative sample pairs containing feature point combinations between the first feature map and the second feature map includes: The correlation strength in the cross-modal relationship graph is normalized to obtain a standardized relationship matrix; Based on the standardized relation matrix, feature point combinations in the first feature map and the second feature map whose correlation strength is higher than a preset first threshold are selected as positive sample pairs. Based on the standardized relation matrix, feature point combinations in the first feature map and the second feature map whose correlation strength is lower than a preset second threshold are selected as negative sample pairs. Specifically, this method first normalizes the association strengths in the cross-modal relationship graph to obtain a standardized relationship matrix. The normalization process uses a normalization exponential function to calculate the normalized standardized association strengths. ,have: , in, The first one in the original cross-modal relationship diagram The feature point and the first The correlation strength of each feature point This represents the total number of feature points in the second feature map. It is a loop variable used to iterate through the indices of all feature points in the second feature map, from 1 to... Next, based on the standardized relation matrix, feature point combinations with a correlation strength higher than a preset first threshold are selected as positive sample pairs, representing strong correlations between cross-modal features. Simultaneously, feature point combinations with a correlation strength lower than a preset second threshold are selected as negative sample pairs, representing weaker correlations between cross-modal features. Both thresholds must satisfy the condition that the first threshold is greater than the second threshold to ensure a clear distinction between positive and negative samples. Figure 3As shown in the figure, the curve represents the standardized association strength distribution of feature point pairs, and the dashed line is the threshold for judging positive and negative samples. Positive sample pairs are those with an association strength greater than 0.7, and negative sample pairs are those with an association strength less than 0.3. Through this sample selection mechanism based on standardized association strength, a discriminative contrastive learning sample set can be effectively constructed.

[0035] For example, suppose the correlation strengths between a feature point and other feature points in the standardized relation matrix are 0.6, 0.3, and 0.1, respectively. Set a first threshold of 0.5 and a second threshold of 0.2. Then, feature point pairs with a correlation strength of 0.6 are selected as positive sample pairs, and feature point pairs with a correlation strength of 0.1 are selected as negative sample pairs. Through standardization and threshold selection, the essential correlations between cross-modal features can be automatically identified and strengthened, while spurious correlations are weakened. This improves the effectiveness of subsequent comparative learning, enabling the model to better distinguish the commonalities and unique characteristics between different modalities and enhance the discriminative power of feature representation.

[0036] Optionally, adjusting the parameters of the first feature extraction network and the second feature extraction network using a contrastive learning objective includes: Based on the positive sample pairs and the negative sample pairs, a contrastive loss function is constructed; The contrast loss function is optimized using gradient descent, and the parameters of the first feature extraction network and the second feature extraction network are adjusted. The parameters of the network are iteratively updated until the distance between the feature points of the positive sample pairs is lower than a preset convergence threshold, and the distance between the feature points of the negative sample pairs is higher than a preset separation threshold.

[0037] Specifically, this method constructs a contrastive loss function using positive and negative sample pairs to calculate the overall contrastive loss value. ,have: , in, This represents the distance between feature points in a positive sample pair. This represents the distance between feature points in a negative sample pair. and These represent the number of positive and negative sample pairs, respectively. A temperature coefficient is used to adjust the smoothness of the distribution. Distance calculation uses the Euclidean distance formula. ,have: , The subscript 2 indicates that the L2 norm, also known as the Euclidean norm, is used. The loss function is optimized using gradient descent, while simultaneously adjusting the parameters of the first and second feature extraction networks. During optimization, the distance between positive sample pairs gradually decreases, while the distance between negative sample pairs gradually increases. The network parameters are iteratively updated until the distance between positive sample pairs falls below a preset convergence threshold, and the distance between negative sample pairs exceeds a preset separation threshold. A temperature coefficient, determined through experimental parameter tuning, is used to balance the learning difficulty between easy and difficult samples. This method explicitly constrains the spatial distribution of cross-modal features, causing relevant features to cluster and irrelevant features to separate.

[0038] For example, assuming the first feature map comes from a visible light image and the second feature map comes from an infrared image, 10 pairs of positive samples and 50 pairs of negative samples are selected to construct a contrastive loss. During the optimization process, the distance between pedestrian features in the visible light image and their corresponding pedestrian features in the infrared image gradually decreases, while the distance with background features gradually increases. Through contrastive learning optimization, the problem of inconsistent feature distribution caused by modal differences can be effectively eliminated, allowing similar targets under different modalities to cluster tightly in the feature space, significantly improving the accuracy of cross-modal retrieval and target matching, especially maintaining stable feature correspondences under conditions of lighting changes or occlusion.

[0039] Optionally, generating the fused feature map includes: The values ​​representing the correlation strength in the cross-modal relationship graph are used as weights; The feature values ​​at corresponding positions in the first feature map and the feature values ​​at corresponding positions in the second feature map are obtained respectively. The feature values ​​at corresponding positions in the first feature map and the feature values ​​at corresponding positions in the second feature map are weighted and summed to obtain the fused feature value. Based on the fused feature values, spatial traversal is performed on the first feature map and the second feature map to generate a fused feature map.

[0040] Specifically, firstly, the values ​​representing the correlation strength in the cross-modal relationship graph are used as weights. These weights reflect the correlation strength between corresponding feature points in the first and second feature maps. For each spatial location, the feature value of that location in the first feature map and the corresponding feature value in the second feature map are obtained respectively. These two feature values ​​are then weighted and summed to obtain the fused feature value. The calculation of the fused feature value... ,have: , in, These are the correlation strength values ​​at locations in the cross-modal relationship graph, which have been normalized to ensure... The feature vectors from the first and second feature maps at corresponding positions, with both having the same dimension to ensure dimensionality consistency, are weighted and summed position by position to ultimately generate a fused feature map. For example... Figure 4 As shown, the left image represents the structural features of the first feature map, the middle image represents the thermal radiation features of the second feature map, and the right image represents the fused feature map of the weighted fusion result. This process adaptively combines the advantages of the two feature maps, highlighting strongly correlated features and suppressing weakly correlated features, thereby improving the discriminative power of the fused result.

[0041] For example, suppose the first feature map is a feature map of a visible light image, and the second feature map is a feature map of an infrared image. At a certain location, the visible light feature value... for Infrared characteristic value for The weights given by the cross-modal relationship graph The value is 0.6. Therefore, the fused eigenvalue is... Calculated as After traversing all locations, the final fused feature map is generated. Through dynamic weight allocation, the detailed information of the visible light image and the thermal radiation information of the infrared image can be effectively integrated, so that the fused feature map can maintain high discriminative power in complex environments, and is especially suitable for target detection tasks at night or in low light conditions.

[0042] Optionally, inputting the fused feature map into a preset recognition network to obtain the recognition result includes: The fused feature map is input into a preset recognition network, and high-level semantic features are extracted through the multi-layer nonlinear transformation of the recognition network. Based on the advanced semantic features, a recognition result associated with the first modality image and the second modality image is generated; Specifically, this method integrates feature maps into a pre-defined recognition network, which then extracts high-level semantic features through multiple nonlinear transformations. The recognition network typically consists of stacked convolutional layers, activation function layers, and pooling layers. The output feature map of each layer is calculated, and the next layer's output feature map is calculated accordingly. ,have: , in, For the first The input feature map of the layer, For convolution kernel weights, This represents the convolution operation. For bias terms, The activation function is non-linear. Through layer-by-layer propagation, low-level visual features in the fused feature maps are progressively transformed into discriminative high-level semantic features. Finally, based on these high-level semantic features, the recognition network generates recognition results associated with the first and second modality images, such as target category labels or segmentation masks, through fully connected layers and a classifier. Throughout the process, the number of channels and spatial resolution of all feature maps meet the dimensionality consistency requirement, ensuring the effectiveness of information transmission.

[0043] For example, assume the fused feature map is composed of visible light and infrared feature maps, with a size of 512×512×64. The recognition network contains five convolutional layers, each extracting features progressively through 3×3 convolutions and linear rectified activation functions, ultimately outputting a 256-dimensional high-level semantic feature vector. Based on this feature vector, the classifier outputs the target category (e.g., pedestrian, vehicle). Through multi-layer nonlinear transformation, the cross-modal complementary information in the fused feature map can be fully exploited, transforming low-level texture features into high-level semantic expressions, significantly improving recognition accuracy in multimodal scenes, especially maintaining stable performance under varying lighting conditions or occlusion.

[0044] Optionally, the method further includes: Based on the recognition results, feature extraction is performed to obtain the recognition loss function; The parameters of the first feature extraction network, the second feature extraction network, and the recognition network are adjusted by combining the contrast loss function and the recognition loss function.

[0045] Specifically, after obtaining the recognition result, this method further calculates a recognition loss function based on the result to measure the difference between the predicted result and the true label. The recognition loss function typically uses cross-entropy loss, which is used to calculate the cross-entropy for classification tasks. ,have: , in, Indicates that the sample belongs to the category The true label (0 or 1). To identify the class probabilities predicted by the network, The total number of categories is represented by this loss function. The gradient of this loss function is calculated through backpropagation to optimize the parameters of the recognition network. Simultaneously, the recognition loss function is combined with the previously constructed contrastive loss function to form a joint loss function. This joint loss function uses gradient descent to jointly adjust the parameters of the first feature extraction network, the second feature extraction network, and the recognition network, enabling the model to improve the accuracy of the final recognition task while optimizing cross-modal feature alignment. The gradient calculations for both losses satisfy dimensional consistency, ensuring the effectiveness of parameter updates.

[0046] For example, in medical image analysis, the first modality is computed tomography (CT) images, and the second modality is magnetic resonance imaging (MRI) images. The model optimizes feature alignment between the two image types through contrastive loss, while ensuring accurate lesion classification through recognition loss. After joint training, the model maintains cross-modal consistency in the feature space and can accurately distinguish between lesions and normal tissues. Through the joint optimization strategy, the model can fully utilize the complementary information of multimodal data, while taking into account both feature-level alignment and task-level discriminability, significantly improving the reliability of cross-modal medical diagnosis, especially maintaining stable performance even under conditions of large modal differences.

[0047] Optionally, the method further includes: A total loss function is generated by weighted evaluation of the contrast loss function and the recognition loss function. The parameters of the network are calculated and adjusted using backpropagation based on the total loss function.

[0048] Specifically, this method combines the contrast loss function and the recognition loss function into a total loss function through weighted summation, thereby calculating the cross-modal feature alignment loss value. ,have: , in, and To balance the weighting coefficients of the two losses, and satisfy... The weight coefficients are determined through grid search or validation set performance to ensure a balance between the two losses. The total loss function's gradient is calculated using the backpropagation algorithm, and the chain rule is used to differentiate layer by layer, updating the parameters of the first feature extraction network, the second feature extraction network, and the recognition network. The update amount of the grid parameters is calculated accordingly. have: , in, The parameters of the network to be optimized include the weights and biases of the feature extraction network and the recognition network. For the total loss against the parameters gradient, The learning rate controls the step size of parameter updates to avoid oscillations or slow convergence. During backpropagation, the contrastive loss primarily affects the parameter updates of the feature extraction network, while the recognition loss primarily affects the parameter updates of the recognition network. All parameter updates use the same metric to ensure stable convergence during the optimization process.

[0049] For example, in an autonomous driving multi-sensor fusion scenario, the first modality is a camera image, and the second modality is a LiDAR point cloud. and The overall loss function simultaneously optimizes cross-modal feature matching and vehicle detection accuracy. During training, the feature spaces of the camera and LiDAR gradually align, while the detection network significantly improves its recognition rate for occluded vehicles. Through the joint optimization mechanism, the model can fully leverage the complementary advantages of different sensors to achieve more robust environmental perception in complex traffic environments, especially demonstrating stronger adaptability under extreme weather conditions.

[0050] Based on the same inventive concept, such as Figure 5 As shown, the present invention also provides a multimodal image fusion recognition system based on contrastive learning, the system comprising: The image acquisition module is used to acquire a first modality image and a second modality image; The feature extraction module is used to process the first modal image through a preset first feature extraction network to generate a first feature map, and to process the second modal image through a preset second feature extraction network to generate a second feature map; A cross-modal relationship graph generation module is used to dynamically generate a cross-modal relationship graph that characterizes the correlation strength between each feature point based on the first feature graph and the second feature graph. The sample pair determination module is used to determine positive and negative sample pairs containing feature point combinations between the first feature map and the second feature map based on the cross-modal relationship graph. The contrastive learning optimization module is used to adjust the parameters of the first feature extraction network and the second feature extraction network using the contrastive learning objective, so as to reduce the distance between feature points in the positive sample pair and increase the distance between feature points in the negative sample pair. The feature fusion module is used to perform weighted fusion of the first feature map and the second feature map based on the cross-modal relationship graph to generate a fused feature map; The recognition result output module is used to input the fused feature map into a preset recognition network to obtain the recognition result.

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] It should be noted that the formulas described above, through the principle of dimensional consistency and mathematical standardization methods (such as normalization, dimensionless parameter conversion, or unit system unification), can translate physical quantities with different properties into unitless standard values ​​or parameters that can be superimposed in the same dimension. This eliminates the interference of different dimensions on the computational logic, allowing the formulas to retain the original data distribution characteristics while possessing mathematical rationality and adaptability to objective laws. These are conventional technical methods and will not be elaborated further. The electrical connections between the various units described above do not necessarily represent direct or indirect connections; any indirect connection method is applicable to the embodiments of this invention as long as it achieves the purpose of this invention. The above descriptions are merely exemplary embodiments of this invention and should not be construed as limiting the scope of this invention.

[0053] All equivalent changes and modifications made in accordance with the teachings of this invention are still within the scope of this invention. Those skilled in the art will readily conceive of other embodiments of this invention upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this invention that follow the general principles of this invention and include common knowledge or conventional techniques in the art not described herein.

Claims

1. A multimodal image fusion and recognition method based on contrastive learning, characterized in that, The method includes: Acquire the first modality image and the second modality image; The first modal image is processed by a preset first feature extraction network to generate a first feature map, and the second modal image is processed by a preset second feature extraction network to generate a second feature map. Based on the first feature map and the second feature map, a cross-modal relationship map that characterizes the correlation strength between each feature point is dynamically generated; Based on the cross-modal relationship graph, positive and negative sample pairs containing feature point combinations are determined between the first feature map and the second feature map; By using a contrastive learning objective to adjust the parameters of the first feature extraction network and the second feature extraction network, the distance between feature points in the positive sample pairs is reduced, and the distance between feature points in the negative sample pairs is increased. Based on the cross-modal relationship graph, the first feature map and the second feature map are weighted and fused to generate a fused feature map; The fused feature map is input into a preset recognition network to obtain the recognition result.

2. The multimodal image fusion and recognition method based on contrastive learning according to claim 1, characterized in that, The step of processing the first modality image through a preset first feature extraction network to generate a first feature map, and processing the second modality image through a preset second feature extraction network to generate a second feature map includes: Obtain the first multi-scale convolutional kernel of the first feature extraction network and the second multi-scale convolutional kernel of the second feature extraction network; Based on the first multi-scale convolution kernel, multi-level spatial features of the first modality image are extracted to generate a first primary feature map; based on the second multi-scale convolution kernel, multi-level spectral features of the second modality image are extracted to generate a second primary feature map. The first primary feature map is augmented to generate an augmented first primary feature map. The second primary feature map is optimized to generate an optimized second primary feature map. Adaptive max pooling sampling is performed on the enhanced first primary feature map to generate a first feature map, and adaptive average pooling sampling is performed on the optimized second primary feature map to generate a second feature map.

3. The multimodal image fusion and recognition method based on contrastive learning according to claim 1, characterized in that, The dynamic generation of cross-modal relationship graphs characterizing the correlation strength between feature points includes: Calculate the similarity between each feature point in the first feature map and all feature points in the second feature map; Based on the similarity, a relation vector associated with the second feature map is generated for each feature point in the first feature map; By combining the aforementioned relation vectors, a cross-modal relation graph representing the correlation strength between each feature point is dynamically generated.

4. The multimodal image fusion and recognition method based on contrastive learning according to claim 1, characterized in that, The step of determining positive and negative sample pairs containing feature point combinations between the first feature map and the second feature map includes: The correlation strength in the cross-modal relationship graph is normalized to obtain a standardized relationship matrix; Based on the standardized relation matrix, feature point combinations in the first feature map and the second feature map whose correlation strength is higher than a preset first threshold are selected as positive sample pairs. Based on the standardized relation matrix, feature point combinations in the first and second feature maps with a correlation strength lower than a preset second threshold are selected as negative sample pairs.

5. The multimodal image fusion and recognition method based on contrastive learning according to claim 1, characterized in that, The step of adjusting the parameters of the first feature extraction network and the second feature extraction network using a contrastive learning objective includes: Based on the positive sample pairs and the negative sample pairs, a contrastive loss function is constructed; The contrast loss function is optimized using gradient descent, and the parameters of the first feature extraction network and the second feature extraction network are adjusted. The parameters of the network are iteratively updated until the distance between the feature points of the positive sample pairs is lower than a preset convergence threshold, and the distance between the feature points of the negative sample pairs is higher than a preset separation threshold.

6. The multimodal image fusion and recognition method based on contrastive learning according to claim 5, characterized in that, The generation of the fused feature map includes: The values ​​representing the correlation strength in the cross-modal relationship graph are used as weights; The feature values ​​at corresponding positions in the first feature map and the feature values ​​at corresponding positions in the second feature map are obtained respectively. The feature values ​​at corresponding positions in the first feature map and the feature values ​​at corresponding positions in the second feature map are weighted and summed to obtain the fused feature value. Based on the fused feature values, spatial traversal is performed on the first feature map and the second feature map to generate a fused feature map.

7. The multimodal image fusion and recognition method based on contrastive learning according to claim 6, characterized in that: The step of inputting the fused feature map into a preset recognition network to obtain the recognition result includes: The fused feature map is input into a preset recognition network, and high-level semantic features are extracted through the multi-layer nonlinear transformation of the recognition network. Based on the advanced semantic features, recognition results associated with the first modality image and the second modality image are generated.

8. The multimodal image fusion and recognition method based on contrastive learning according to claim 7, characterized in that, The method further includes: Based on the recognition results, feature extraction is performed to obtain the recognition loss function; The parameters of the first feature extraction network, the second feature extraction network, and the recognition network are adjusted by combining the contrast loss function and the recognition loss function.

9. The multimodal image fusion and recognition method based on contrastive learning according to claim 8, characterized in that, The method further includes: A total loss function is generated by weighted evaluation of the contrast loss function and the recognition loss function. The parameters of the network are calculated and adjusted using backpropagation based on the total loss function.

10. A multimodal image fusion recognition system based on contrastive learning, applied to a multimodal image fusion recognition method based on contrastive learning as described in any one of claims 1-9, characterized in that, The system includes: The image acquisition module is used to acquire a first modality image and a second modality image; The feature extraction module is used to process the first modal image through a preset first feature extraction network to generate a first feature map, and to process the second modal image through a preset second feature extraction network to generate a second feature map; A cross-modal relationship graph generation module is used to dynamically generate a cross-modal relationship graph that characterizes the correlation strength between each feature point based on the first feature graph and the second feature graph. The sample pair determination module is used to determine positive and negative sample pairs containing feature point combinations between the first feature map and the second feature map based on the cross-modal relationship graph. The contrastive learning optimization module is used to adjust the parameters of the first feature extraction network and the second feature extraction network using the contrastive learning objective, so as to reduce the distance between feature points in the positive sample pair and increase the distance between feature points in the negative sample pair. The feature fusion module is used to perform weighted fusion of the first feature map and the second feature map based on the cross-modal relationship graph to generate a fused feature map; The recognition result output module is used to input the fused feature map into a preset recognition network to obtain the recognition result.

Images