Multi-modal image registration and fusion method and system based on deep learning

Abstract

The invention discloses a multi-modal image registration and fusion method and system based on deep learning, and the method comprises the steps: generating a pseudo-infrared image through a cross-modal image generation network, so as to reduce the modal difference between source images, and precisely estimating a deformation field between the images through a multi-scale registration network; collaborative optimization of a registration network and a fusion network is realized by adopting a joint training framework, and semantic information of a fusion image is enhanced by applying a semantic-guided fusion framework. The technology is mainly used in the fields of automatic driving, video monitoring, medical imaging and the like, provides richer and more accurate scene description than a single sensor, and particularly has important application value in the aspects of day and night monitoring and target recognition.

Description

Technical Field

[0001] This invention belongs to the fields of image processing and computer vision, and particularly relates to a method and system for multimodal image registration and fusion based on deep learning. Background Technology

[0002] With the rapid development of computer science and information technology, multi-source image fusion technology is increasingly widely used in daily life and production. Multi-source image fusion technology can integrate the imaging advantages of multiple sensors, providing richer and more comprehensive information. In the field of multi-source image fusion, the fusion of infrared and visible light images is particularly typical. Visible light images have advantages such as intuitive clarity, rich color and texture details, and ease of understanding, but they are greatly affected by lighting conditions, resulting in extremely poor imaging performance in low light. Infrared images have strong penetrating power, are unaffected by lighting conditions, and can clearly observe targets even in harsh environments, but lack detailed descriptions of the scene. Therefore, by analyzing and integrating high-resolution scene texture information from visible light images and high-brightness target information from infrared images, more comprehensive imaging information can be obtained, effectively combining the advantages of both types of images, compensating for their respective shortcomings, and thus improving the overall imaging effect.

[0003] However, during data acquisition, data from different sensors may differ in angle, distance, etc., and cannot be directly processed in the same reference coordinate system. Therefore, registration is required before fusing images from different sensors. Image registration is the process of mapping one image onto another by finding an appropriate spatial transformation relationship between two images in different states. Image registration technology is an important prerequisite for fusion, and the quality of fusion is limited by the accuracy of registration. Traditional image registration methods have limitations when dealing with heterogeneous images with significant differences, such as infrared and visible light. For example, they are prone to ghosting when processing misaligned infrared and visible light images, and they do not meet the semantic information requirements of advanced vision tasks.

[0004] In recent years, the rise of deep learning technology has brought new opportunities for multimodal image registration and fusion. By fully utilizing the superior feature learning capabilities of deep learning methods, researchers have proposed a variety of high-precision and efficient registration and fusion methods. For example, methods based on convolutional neural networks (CNNs) improve the quality of fused images through multi-level feature extraction and fusion; methods based on generative adversarial networks (GANs) further enhance the quality and detail of fused images through adversarial game between the generator and discriminator. Although these methods have improved the performance of multimodal image registration and fusion to some extent, some shortcomings still exist. For example, most methods are highly dependent on the pre-registration of the input images, are sensitive to modal differences between images, and are prone to ghosting. In addition, most fusion algorithms focus on the visual quality of the images, ignoring the contribution of the fusion results to higher-level visual tasks, thus limiting the potential of fused images in subsequent applications.

[0005] Therefore, this invention aims to propose a deep learning-based multimodal image registration and fusion method and system. By jointly training a multi-scale registration network and a semantically guided fusion framework, it overcomes the shortcomings of existing methods, improves the accuracy of image registration and the quality of fused images, and provides support for advanced vision tasks. Summary of the Invention

[0006] The purpose of this invention is to address the problems existing in current infrared and visible light image registration and fusion technologies by proposing a multimodal image registration and fusion method and system based on deep learning.

[0007] This invention mainly addresses the problems existing in current infrared and visible light image registration and fusion technologies through the following methods:

[0008] To address the artifact problem that easily arises when processing unregistered source images, this invention proposes a multi-scale registration network (MSRN) based on joint training. First, a cross-modal image generation network (CIGN) is used to generate pseudo-infrared images with structures similar to real infrared images, reducing modal differences between source images. Then, the MSRN network employs a coarse-to-fine strategy to accurately estimate the deformation fields between images, progressively optimizing the deformation fields at different resolution levels to achieve more accurate image alignment. During the training phase, a joint training framework is designed, feeding the fusion loss back to the registration network, achieving collaborative optimization between the registration and fusion networks, further improving the accuracy of image registration and the quality of the fused image. Experimental results demonstrate that this method performs excellently in reducing artifacts and improving image registration accuracy.

[0009] To address the issue that existing image fusion methods neglect the semantic information requirements of subsequent advanced visual tasks, this invention proposes a semantically guided fusion framework. First, a dual-channel feature extraction network is used to extract features from both infrared and visible light images. Then, a semantic embedding module (SEM) effectively injects the rich semantic features extracted by the segmentation network into the fusion network, ensuring that the fused image not only maintains a high visual similarity to the source images but is also rich in semantic information.

[0010] The objective of this invention is achieved through the following technical solution:

[0011] This invention discloses a deep learning-based multimodal image registration and fusion method, which includes:

[0012] S1. Image Preprocessing: The visible light image to be registered is processed using a cross-modal image generation network (CIGN). vis Converted into a pseudo-infrared image with a structure similar to that of a real infrared image.

[0013] S2. Image Registration: The multi-scale registration network MSRN is used to accurately estimate the infrared image to be registered through multi-scale feature extraction and a coarse-to-fine strategy. With pseudo-infrared images The deformation field between them yields the registered infrared image.

[0014] S3. Image Fusion: Utilizing a semantically guided fusion network, combined with semantic features extracted by a segmentation network. seg The semantic information of the fused image is enhanced by the semantic embedding module SEM to generate the fused image I. fus ;

[0015] S4. Joint Training: Design a joint training framework to feed the fusion loss back to the registration network, optimize the registration effect and improve the quality of the fused image.

[0016] Furthermore, the cross-modal image generation network consists of two generators based on ResNet and two discriminators based on convolutional neural networks. The generators learn residual mappings through a series of downsampling, upsampling operations and ResNet Blocks (RES) to convert visible light images into pseudo-infrared images with structures similar to real infrared images. The discriminators process the input images through multi-layer convolution and normalization and perform binary classification of real and fake images.

[0017] Furthermore, the operation steps of the multi-scale registration network are as follows:

[0018] a) Multi-scale feature extraction: using a shared multi-level feature extractor F kk = 1, 2, ..., K, from the infrared image to be registered and pseudo-infrared images Extracting multi-level features, F k Refers to the feature extractor at the k-th scale;

[0019] b) Coarse Deformation Field Estimation: The coarse deformation field estimation module M is used. C Predict the coarse deformation field between two images First, the correlation map between the two features is calculated. Then, it is concatenated with the original features and processed through a six-layer convolutional neural network to obtain the coarse deformation field.

[0020] c) Refine the deformation field estimation: refine the coarse deformation field This is provided as input to the refined deformation field estimation module M. R First, the matching cost map is calculated, then processed through a six-layer convolutional neural network. Finally, the output of the convolutional neural network is residually connected to the coarse deformation field to obtain the refined deformation field.

[0021] d) Image resampling: When k = K, the final deformed field is obtained. The registered infrared image is reconstructed using a resampling layer to obtain the registered infrared image.

[0022] Furthermore, the semantically guided fusion network is designed using a cascade principle, including a backbone fusion network and a segmentation network. It extracts features from infrared and visible light images through a dual-channel feature extraction network, and embeds the semantic features extracted by the segmentation network into the backbone fusion network using a semantic embedding module.

[0023] Furthermore, the semantic embedding module consists of a semantic-guided cross-modal attention mechanism (SCAM) and a modality-guided cross-modal attention mechanism (MCAM). By calculating the global context representation, the features of different subspaces are complementary and interact to improve the semantic expressive power of modal features.

[0024] The semantically driven cross-attention mechanism utilizes semantic features f seg Generate query vector Q s Calculate the global context representation G ir and G vis , Where K ir and K visRepresenting modal features {f ir ,f vis The corresponding key vector, V ir and V vis Represents the corresponding value vector; by querying the vector Q s With global context representation G ir and G vis Multiply to obtain semantically dominant attention-weighted features S ir and S vis ;

[0025] The modality-dominated cross-attention mechanism introduces the modality query vector Q. ir Q vis , used to characterize modal features {f ir ,f vis The intrinsic modal properties of}, the key vector K s Sum vector V s From semantic features f seg Generate; through computing the global context of semantic features. Obtain modality-dominated cross-attention weighted features M ir and M vis M ir =Q ir ·G s M vis =Q vis ·G s .

[0026] Furthermore, the joint training method involves feeding the fusion loss back into the registration network, and training the registration network together with the registration loss. The training steps are as follows:

[0027] a) Initialization: Select a set of infrared and visible light images to construct a training set;

[0028] b) Iterative training: The maximum number of iterations is M. Each iteration includes three main steps: training the registration network, generating the registration image, and training the fusion network. The model performance is gradually improved through continuous iterative optimization.

[0029] ① Registration Network Parameter Update: In each iteration, p-step registration network parameter updates are performed first. The Adam optimizer is used to adjust the registration network parameters based on the joint loss function obtained by weighting the registration loss and fusion loss. The joint loss function comprehensively considers multiple factors involved in the registration process, aiming to enable the registration network to learn feature representations that are more conducive to fusion. Prioritized, the GradNorm method is introduced to dynamically update the joint loss weights. This dynamic weighting mechanism ensures that the contributions of each loss branch to network training are balanced, preventing any one branch from dominating the training process or becoming ineffective.

[0030] ② Generate registered images: Using the updated registration network parameters, generate registered infrared images from the training set. Input the distorted infrared images from the training set and the pseudo-infrared images obtained through the cross-modal image generation network into the registration network. The registration network transforms the distorted infrared images according to the learned deformation field to obtain high-quality registered images.

[0031] ③ Fusion Network Parameter Update: The registered visible-infrared image pairs generated by the registration network are input into the fusion network for training, and the fusion network parameters are updated q times. The Adam optimizer is still used, but this time a fusion loss function is employed, consisting of MS-SSIM loss, gradient loss, and saliency loss, which evaluate the quality of the fused image in terms of sharpness preservation, detail recovery, and salient object retention, respectively. By adjusting the fusion network parameters, it can better fuse the features of infrared and visible light images, generating a higher-quality fused image.

[0032] c) End of iteration: Repeat the above steps until all iterations are completed.

[0033] Furthermore, the cross-modal image generation loss function of the cross-modal image generation network consists of a perceptual style transfer loss and a cross-regulation loss; the perceptual style transfer loss is obtained by weighting the perceptual loss and the style loss; the perceptual loss is used to ensure that the generated pseudo-infrared image maintains the high-level features of the real infrared image at different levels of the VGG-19 network, while also considering the differences between the visible light image and the generated pseudo-infrared image at the same level; the style loss is used to maintain the style features of the generated image; the cross-regulation loss is obtained by weighting the content term and the edge term to ensure that the generated pseudo-infrared image is consistent with the real infrared image in terms of content.

[0034] Furthermore, the registration loss of the registration network consists of a bidirectional similarity loss and a smoothing loss. The bidirectional similarity loss includes forward and backward similarity losses, used to constrain the feature space similarity between the unregistered infrared image and the pseudo-infrared image. The forward similarity loss measures the similarity between the unregistered infrared image and the pseudo-infrared image in the feature space by calculating the difference between features at corresponding levels. The backward similarity loss calculates the difference between the pseudo-infrared image after inverse deformation and the original unregistered infrared image, ensuring that not only is the transformation from the unregistered infrared image to the pseudo-infrared image reasonable, but the similarity requirement is also met when returning from the pseudo-infrared image to the unregistered infrared image through inverse transformation. The smoothing loss is used to ensure the smoothness of the deformation field by calculating and summing the L1 norm of the deformation field gradient to penalize the drastically changing parts of the deformation field.

[0035] Furthermore, the fusion loss is obtained by weighting MS-SSIM loss, gradient loss, and saliency loss; the MS-SSIM loss is used to maintain a clear intensity distribution in the fused image; the gradient loss is used to encourage the recovery of texture details; and the saliency loss is used to preserve salient objects in the image.

[0036] Another aspect of the present invention discloses a deep learning-based multimodal image registration and fusion system, the system comprising:

[0037] Image preprocessing module: Utilizes a cross-modal image generation network to process the visible light image to be registered (I). vis Converted into a pseudo-infrared image with a structure similar to that of a real infrared image.

[0038] Image registration module: Employs a multi-scale registration network, accurately estimating the infrared image to be registered through multi-scale feature extraction and a coarse-to-fine strategy. With pseudo-infrared images The deformation field between them yields the registered infrared image.

[0039] Image fusion module: Employs a semantically guided fusion network, combining semantic features f extracted by the segmentation network. seg By enhancing the semantic information of the fused image through a semantic embedding module, a fused image I is generated. fus ;

[0040] Joint Training Module: Design a joint training framework to feed the fusion loss back to the registration network, optimize the registration effect and improve the quality of the fused image.

[0041] The beneficial effects of this invention are as follows:

[0042] 1. Reduced artifact generation: By introducing the cross-modal image generation network CIGN to generate pseudo-infrared images with structures similar to real infrared images, modal differences between source images are reduced, thereby effectively reducing artifact generation during the registration process.

[0043] 2. Improved image registration accuracy: The multi-scale registration network MSRN is adopted to accurately estimate the deformation field between images through a coarse-to-fine strategy. The deformation field is gradually optimized at different resolution levels, achieving more accurate image alignment and significantly improving the accuracy of image registration.

[0044] 3. Collaborative optimization of the registration and fusion networks was achieved: A joint training framework was designed, which feeds the fusion loss back to the registration network, thereby achieving collaborative optimization of the registration and fusion networks and further improving the accuracy of image registration and the quality of the fused images.

[0045] 4. Enhanced semantic information of fused images: The semantic embedding module (SEM) effectively injects the rich semantic features extracted by the segmentation network into the fusion network, ensuring that the fused image not only maintains a high degree of visual similarity to the source image, but is also rich in semantic information, thereby enhancing the application potential of fused images in advanced vision tasks.

[0046] 5. Improved quality of fused images: A dual-channel feature extraction network is used to extract features from infrared and visible light images respectively, and the semantic expression ability of modal features is further enhanced through a cross-attention mechanism, generating fused images rich in structural and semantic information, which significantly improves the quality of fused images.

[0047] 6. Verification of the effectiveness of the method: Experimental results show that the present invention exhibits excellent performance on multiple evaluation indicators, especially in reducing artifacts, improving image registration accuracy and generating high-quality fused images, thus verifying the effectiveness and reliability of the method.

[0048] 7. Expanded application areas: This invention not only has broad application prospects in fields such as autonomous driving, video surveillance, and medical imaging, but also provides a solid foundation for subsequent advanced vision tasks such as object detection and semantic segmentation, which has important practical significance and application value. Attached Figure Description

[0049] Figure 1 A general block diagram of the deep learning-based multimodal image registration and fusion method provided in the embodiments of the present invention;

[0050] Figure 2 This is a network block diagram of a cross-modal image generation network provided in an embodiment of the present invention;

[0051] Figure 3 This is a block diagram of a multi-scale registration network provided in an embodiment of the present invention;

[0052] Figure 4 This is a semantically guided fusion network block diagram provided in an embodiment of the present invention;

[0053] Figure 5 A block diagram of a semantic embedding module provided in an embodiment of the present invention;

[0054] Figure 6 A block diagram of the cross-attention module provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments disclosed herein.

[0056] For ease of explanation, numerous details are provided in the following description to provide a full understanding of the disclosed embodiments. However, the invention may also be implemented in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0057] like Figure 1 As shown in the figure, this embodiment provides a multimodal image registration and fusion method based on deep learning, which includes the following steps:

[0058] Step 1, Image Preprocessing: The visible light image to be registered is processed using the Cross-Modal Image Generation Network (CIGN). vis Converted into a pseudo-infrared image with a structure similar to that of a real infrared image. The aim is to reduce the differences between different modalities, in order to prepare for subsequent multi-scale registration and fusion.

[0059] like Figure 2 As shown, CIGN consists of two generators G based on the ResNet residual network architecture. A G B The system consists of two discriminators, D(ir) and D(vis), based on a convolutional neural network architecture. The generator learns residual mappings through a series of downsampling and upsampling operations and residual blocks, converting visible light images into pseudo-infrared images with structures similar to real infrared images, thus reducing modal differences between visible light and infrared images. The discriminator processes the input image through multi-layer convolution and normalization, outputting a single-channel prediction result for binary classification of real and fake images, assisting the generator in better mimicking the characteristics of real infrared images.

[0060] During training, registered visible light is used. —Infrared The image pairs are used to train the cross-modal image generation network. To ensure conversion quality, this invention designs a specific cross-modal image generation loss function, including perceptual style transfer loss and cross-regulation loss. These two losses work together to ensure that the generated pseudo-infrared images not only maintain consistency with real infrared images in high-level features but also are as similar as possible in visual style and edge information.

[0061] Perceptual style transfer loss L pst From perceived loss L pcp and style loss L sty The weighted average is obtained, and the calculation formula is as follows:

[0062] L pst =ω p L pcp +ω s Lsty

[0063] Where, ω p and ω s These are the perceived loss L pcp and style loss L sty The weights. Perceptual loss L pcp Defined as:

[0064]

[0065] Where, Φ j Let represent the j-th layer features of the VGG-19 network, where j∈[2,7,12,21,30]. This part aims to ensure that the generated pseudo-infrared images maintain consistency with the high-level features of the real infrared images at different layers of the VGG-19 network, while also considering the differences between the visible light images and the generated pseudo-infrared images at the same layers. Style loss L sty The calculation formula is as follows:

[0066]

[0067] The corresponding weights are: G is the operation for calculating the Gram matrix, used to preserve the stylistic features of the generated image. It is an effective tool for suppressing checkerboard artifacts in images.

[0068] Cross-regular loss L cross From content item L con and edge item L edge The weighted average ensures that the generated pseudo-infrared image is consistent with the real infrared image in content. The formula is as follows:

[0069]

[0070] L cross =α1L con +α2L edge

[0071] In the formula This refers to the Laplacian gradient operator, where α1 and α2 are the content terms L. con and edge item L edge The weights; using Charbonnier loss L char Calculate L edge ∈ is a small positive number designed to preserve edge information in the generated image and enhance image structural clarity. This loss function helps improve the quality of pseudo-infrared images, making them closer to the features of real infrared images, thus benefiting subsequent image registration and fusion processes.

[0072] Step 2, Image Registration: (e.g.) Figure 3 As shown, the multi-scale registration network MSRN is used to register the infrared image to be registered. With pseudo-infrared images Registration involves the following sub-steps:

[0073] 2.1 Multi-scale feature extraction: using a shared multi-level feature extractor F k (k = 1, 2, ..., K) from the infrared image to be registered and pseudo-infrared images Extract multi-level features from F. k This refers to the feature extractor at the k-th scale. The multi-level feature extractor is constructed by stacking convolutional neural networks. This step ensures that sufficient detail is available to support subsequent deformation field estimation.

[0074] 2.2 Coarse Deformation Field Estimation: The coarse deformation field estimation module M is used. C Predict the coarse deformation field between two images First, the correlation map between the two features is calculated. Then, it is concatenated with the original features and processed through a six-layer convolutional neural network to obtain the coarse deformation field. The calculation formula is:

[0075]

[0076] This step aims to quickly determine the approximate positional correspondence between two images, laying the foundation for more detailed matching.

[0077] 2.3 Refining the Deformation Field Estimation: This involves refining the coarse deformation field estimation... This is provided as input to the refined deformation field estimation module M. R First, the matching cost map is calculated, then processed through a six-layer convolutional neural network. Finally, the output of the convolutional network and the coarse deformation field are residually connected to obtain the refined deformation field. The calculation formula is:

[0078]

[0079] This method allows for gradual adjustment of the deformation field, making it more accurately reflect the actual differences between the two images.

[0080] 2.4 Image resampling: When k = K, the final deformed field is obtained. The registered infrared image is reconstructed using a resampling layer (Re-Sampler). The calculation formula is:

[0081]

[0082] 2.5 Registration Loss: To ensure the accuracy and smoothness of registration, a composite loss function is defined, consisting of bidirectional similarity loss and smoothing loss. Bidirectional similarity loss includes forward and backward similarity losses, used to constrain the feature space similarity between unregistered infrared images and pseudo-infrared images. Smoothing loss focuses on the spatial continuity of the deformation field.

[0083] Forward similarity loss measures the similarity between unregistered infrared and pseudo-infrared images in the feature space by calculating the differences between features at corresponding levels. It requires the network to learn feature representations that make the features of the two images as similar as possible at each level. If the feature differences are large, the forward similarity loss value will increase, prompting the network to adjust its parameters through backpropagation to reduce these differences. Forward similarity loss L for The calculation formula is:

[0084]

[0085] The backward similarity loss calculates the difference between the inversely transformed pseudo-infrared image and the original unregistered infrared image. The existence of this loss term ensures that not only is the transformation from the unregistered infrared image to the pseudo-infrared image reasonable, but also the transformation from the pseudo-infrared image to the pseudo-infrared image via inverse transformation is also reasonable. Even when reverting to the unregistered infrared image, high similarity is maintained, thus enhancing the robustness of registration. The backward similarity loss L... rev The calculation formula is:

[0086]

[0087] Bidirectional similarity loss L bi-sim The forward similarity loss and the backward similarity loss are weighted and summed:

[0088] L bi-sim =L for +λ rev L rev

[0089] The weights λ of the backward similarity loss rev It can be set to 0.2. This weighted summation method can more comprehensively constrain the similarity between unregistered infrared images and pseudo-infrared images in the feature space, allowing the network to consider both forward and reverse feature matching relationships during the registration process.

[0090] Smoothing loss is used to ensure the smoothness of the deformation field; the smoothing loss L... smooth The calculation formula is:

[0091]

[0092] Smoothing loss penalizes drastically changing parts of the deformable field by calculating and summing the L1 norm of the gradient. Abrupt changes in the deformable field can lead to unreasonable distortion or stretching in image registration. Minimizing smoothing loss makes the spatial changes of the deformable field smoother, avoiding the introduction of unnecessary noise or discontinuities during image registration, and ensuring that the registered image is more natural and accurate.

[0093] Registration loss L reg The calculation formula is:

[0094] L reg =L bi-sim +λ sm L smooth

[0095] Where, λ sm λ is the weighting parameter. This weight is used to balance the proportion of bidirectional similarity loss and smoothing loss in the overall registration loss. It is adjusted by... sm The value of can be optimized based on actual needs. If a greater emphasis is placed on feature similarity matching, it can be appropriately decreased; if greater emphasis is placed on the smoothness of the deformation field to avoid abnormal deformation during registration, it can be appropriately increased. The registration loss function, which combines bidirectional similarity loss and smoothness loss, enables the multi-scale registration network to ensure the smoothness of the deformation field while maintaining feature similarity between unregistered infrared and pseudo-infrared images during training, thereby achieving high-quality image registration.

[0096] The above steps work together to achieve high-precision registration between the infrared image to be registered and the pseudo-infrared image, enabling them to maintain a high degree of consistency at the feature level while maintaining a smooth transition of the deformation field, thereby improving the robustness and applicability of the entire system.

[0097] Step 3, Image Fusion: To generate a high-quality fused image, this invention employs a semantically guided fusion network, such as... Figure 4 As shown, this network improves the fusion effect by fully integrating semantic features. The detailed implementation steps are as follows:

[0098] 3.1 Feature Extraction and Semantic Acquisition: In this step, features of visible light and infrared light are first extracted simultaneously using a shallow convolutional neural network. Then, a dual-channel feature extraction network is used to extract features from the infrared image separately. and visible light image I vis Feature extraction is performed, and the semantic information f of the image is obtained using the classic semantic segmentation network SegFormer. seg .

[0099] a) Dual-channel feature extraction: Dual-channel feature extraction networks are specifically designed to efficiently extract features from infrared and visible light images, respectively. For example... Figure 4 As shown, this network consists of Residual Dense Connected Blocks (RDBs), which extract rich local features by densely connecting convolutional layers. Since infrared and visible light images have different modal characteristics, a dual-channel structure can specifically capture their unique information. The feature extraction process can be represented as follows:

[0100]

[0101] f vis =RDB(conv(I vis ))

[0102] In this way, the dual-channel feature extraction network extracts features f containing rich modal information from infrared and visible light images, respectively. ir and f vis .

[0103] b) Semantic Information Acquisition: The semantic information of the image is acquired using the classic semantic segmentation network SegFormer. The resulting semantic features f seg It contains semantic category information of different objects and scenes in the image, providing a foundation for subsequent semantic embedding.

[0104] 3.2 Semantic Embedding: The extracted semantic information is embedded into the backbone fusion network through the Semantic Embedding (SEM) module. For example... Figure 5 , Figure 6 As shown, the semantic embedding module consists of a semantically dominant cross-attention mechanism (SCAM) and a modality-dominant cross-attention mechanism (MCAM). These two mechanisms work together to significantly improve the semantic expressive power of modality features.

[0105] a) Semantic-Driven Cross-Attention Mechanism (SCAM): The goal of a semantically driven attention module is to provide more semantic attention to features from different modalities. SCAM utilizes f seg Generate query vector Q s This represents the inherent semantic information that needs enhancement. For each attention head, a global context representation, specifically G, is computed. ir and G vis The calculation formulas are as follows:

[0106]

[0107] Among them, K ir and K vis Representing modal features {f ir ,f vis The corresponding key vector, and V ir and V vis This represents the corresponding value vector. By querying the vector Q...s Multiplying these global context representations yields semantically dominant attention-weighted features S. ir and S vis :

[0108] S ir =Q s ·G ir

[0109] S vis =Q s ·G vis

[0110] b) Modality-Dominated Cross-Attention Mechanism (MCAM): This attention mechanism focuses more on modal information, fusing semantic information as supplementary information with modal features. Specifically, MCAM introduces a modal query vector Q. ir Q vis , used to characterize modal features {f ir ,f vis The inherent modal properties (such as target contour and image details) in}, key K s Sum V s Vectors are composed of semantic features f seg Generation. This is achieved by computing the global context G of semantic features. s :

[0111]

[0112] Further obtain modality-dominated cross-attention weighted features M ir and M vis

[0113] M ir =Q ir ·G s

[0114] M vis =Q vis ·G s

[0115] Finally, by integrating the feature set F vis ={S vis M vis} and F ir ={S ir M ir This yields two sets of parallel integrated features, providing rich semantic information and contextual content for subsequent image fusion tasks. Through this mechanism, SCAM and MCAM work together in the image fusion network, not only enhancing the semantic expressive power of modal features but also providing strong support for the performance of the fused image in subsequent high-level vision tasks.

[0116] 3.3 Fusion and Reconstruction: The integrated feature F ir ,F vis The input features are fed into the fusion module FM for final feature fusion. The fusion module FM first uses adaptive weights learned from a convolutional neural network to weight the input features to achieve effective fusion, and then uses a convolutional neural network to reconstruct and generate the final fused image I. fus Through this fusion and reconstruction process, the features and semantic information of infrared and visible light images are deeply fused to generate a fused image that contains rich modal information and clear semantic expression, providing high-quality image data for subsequent advanced vision tasks such as object detection and semantic segmentation.

[0117] 3.4 Fusion Loss: It consists of MS-SSIM loss, gradient loss, and significance loss. These loss functions will be described in detail below.

[0118] a) MS-SSIM loss: MS-SSIM loss L MS-SSIM The formula used to maintain a clear intensity distribution in the fused image is:

[0119]

[0120] MS-SSIM loss is a widely used metric for image quality assessment, comparing fused images at multiple scales. fus Registered infrared image and visible light image I vis The intensity distribution of the fused image is measured by its brightness, contrast, and structure. By calculating the structural similarity of these images at different scales, MS-SSIM can comprehensively evaluate the similarity between the fused image and the original image in terms of intensity distribution. The closer the intensity distribution of the fused image is to the original image, the closer the MS-SSIM value is to 1, where L... MS-SSIM The closer the value of L is to 0, the higher the quality of the fused image in terms of intensity distribution. Therefore, by minimizing L... MS-SSIM This can encourage the fusion network to generate a fused image with a clearer intensity distribution that is closer to the original image.

[0121] b) Gradient loss: Gradient loss L gradient The formula used to encourage the recovery of texture details is:

[0122]

[0123] In an image, the gradient reflects its texture and edge information. Gradient loss is calculated by comparing the fused images I... fus gradient and registered infrared image and visible light image I visThe maximum value of the gradient is used to measure the effectiveness of the fused image in restoring texture details. If the gradient of the fused image is as close as possible to the maximum value of the gradient of the original image, it means that the fused image retains more texture details. This is achieved by minimizing the gradient loss L. gradient This can guide the fusion network to focus on preserving and restoring texture details during the fusion process, making the texture of the fused image clearer and more in line with the needs of practical applications.

[0124] c) Significance loss: Significance loss L salienct The formula used to preserve salient objects in an image is:

[0125]

[0126] S vis (x,y)=||I μ-vis -I vis (x,y)||

[0127] Among them, S ir (x,y) and S vis (x,y) represents the saliency value at pixel position (x,y) in infrared and visible light, used to highlight important areas in the image. and I μ-vis This represents the average pixel value of infrared and visible light images. and I vis (x,y) represents the pixel value at position (x,y) in the infrared and visible light images. The saliency loss uses saliency values ​​to weight the difference between the fused image and the weighted original image. In image fusion, it is desirable for the fused image to retain salient targets from the original image, achieved by minimizing the saliency loss L. saliency This allows the fusion network to focus more on salient regions in the image, ensuring that the information in the fused image remains consistent with that in the original image, thereby improving the usability of the fused image in subsequent advanced vision tasks.

[0128] d) Overall fusion loss: Fusion loss L fusion =λ ssim L MS-SSIM +λ JG L gradient +λ svs L saliency , where λ ssim , λ JG , λ svsThese are weighting parameters. These weighting parameters are used to balance the contributions of different loss functions to the fusion loss. By adjusting these weights, the emphasis of the fused image in terms of intensity distribution, texture detail restoration, and salient object preservation can be adjusted according to the needs of the actual application. For example, if a certain application scenario requires high texture detail in the fused image, λ can be appropriately increased. JG The value of λ increases the proportion of gradient loss in the fusion loss, thus prompting the fusion network to focus more on restoring texture details; if more attention is paid to the intensity distribution of the fused image, λ can be increased. ssim The values ​​of these weights can be adjusted to better adapt the fusion loss function to different application scenarios, guiding the fusion network to generate high-quality fused images.

[0129] Through the above steps, deep fusion of infrared and visible light images is achieved, generating a high-quality fused image that contains rich modal information and clear semantic expression, suitable for advanced vision tasks such as object detection and semantic segmentation.

[0130] Step 4, Joint Training: This invention designs a joint training framework that optimizes the registration effect by feeding the fusion loss back to the registration network, thereby improving the performance of the entire image registration and fusion system. The joint training process is described in detail below.

[0131] 4.1 Initialization: Select a set of infrared and visible light images, which will serve as the foundational data for subsequent training. In practical applications, these images can originate from different acquisition devices and environments, covering various scenes and lighting conditions to enhance the model's generalization ability. For example, in the field of security monitoring, these images may come from surveillance cameras at different times and under different weather conditions, including infrared and visible light images taken during the day, at night, in sunny weather, and in rainy weather. These images constitute the training set, providing rich samples for model training, enabling the model to learn the characteristics and changing patterns of images under different conditions.

[0132] 4.2 Iterative Training: Perform a maximum of M iterations, with each iteration containing three main steps. Through continuous iterative optimization, the performance of the model is gradually improved.

[0133] a) Registration Network Parameter Update: First, p steps of registration network parameter update are performed. During this process, the joint loss function L... joint =λ reg L reg +λ fus L fusion The Adam optimizer is used to update the parameters of the registration network. Unlike the traditional method of fixing loss weights, this invention introduces the GradNorm method to implement joint loss weights λ. regand λ fus The method first calculates the gradient norm of each loss term with respect to the network's shared layer parameters w:

[0134]

[0135] Here, t is the time parameter during training, representing the current epoch in the training process. Then, the relative rate of decline r for each loss branch is recorded. j (t), which measures the convergence progress of each branch during training:

[0136]

[0137] Based on this, GradNorm calculates the ideal target gradient norm for each loss branch. Give more weight to branches that train more slowly:

[0138]

[0139] in, λ is the mean of all current gradient norms, and α is a hyperparameter controlling the balance strength. Finally, the weights λ are updated by minimizing the distance between the actual gradient norm and the target gradient norm. j :

[0140]

[0141] L grad It measures the actual gradient norm G. j (t) and the gradient norm of the ideal target The difference between them is a supervision weight λ j The loss function used to learn the weights λ j The updates ensure that the gradient norms of each task branch approach their ideal target values. This dynamic weighting mechanism guarantees a balanced contribution of each loss branch to network training, preventing any one branch from dominating the training process or becoming ineffective. The joint loss function comprehensively considers various factors in the registration process, aiming to enable the registration network to learn feature representations that are more conducive to fusion.

[0142] b) Generating the registered image: Next, using the parameters of the current registration network, a registered infrared image is generated from the training set. At this point, the registration network has undergone p steps of parameter updates and possesses a certain registration capability. The distorted infrared image from the training set and the pseudo-infrared image obtained through the cross-modal image generation network are input into the registration network. The registration network transforms the distorted infrared image according to the learned deformation field to obtain the registered infrared image.

[0143] c) Fusion Network Parameter Update: Finally, the registered visible-infrared image pairs generated by the registration network are input into the fusion network for training. A q-step parameter update is performed, using the Adam optimizer to update the network parameters according to the fusion loss function. The fusion loss function consists of MS-SSIM loss, gradient loss, and saliency loss, which evaluate the quality of the fused image from different perspectives. MS-SSIM loss maintains a clear intensity distribution in the fused image, gradient loss encourages the recovery of texture details, and saliency loss preserves salient objects in the image. The Adam optimizer updates the fusion network parameters based on the gradient calculated by the fusion loss function, enabling the fusion network to better fuse features from both infrared and visible light images, generating a higher-quality fused image.

[0144] 4.3 Iteration End: Repeat the above steps until all M iterations are completed. In each iteration, the registration network and the fusion network collaborate. Through a joint training framework, the registration network continuously receives loss information from the fusion network, adjusts its own parameters, improves registration accuracy, and provides the fusion network with higher-quality registered images. The fusion network trains based on the registered images, optimizes its own parameters, and generates fused images that better meet the requirements. As the number of iterations increases, the model gradually learns the inherent features and relationships between infrared and visible light images, and the registration and fusion effects gradually improve.

[0145] Another aspect of the present invention discloses a deep learning-based multimodal image registration and fusion system, the system comprising:

[0146] Image preprocessing module: This module utilizes the Cross-Modal Image Generation Network (CIGN) to convert the visible light image to be registered into a pseudo-infrared image with a structure similar to the real infrared image, reducing the modal differences between the two. Simultaneously, a specific loss function is calculated to optimize the CIGN network. The CIGN network consists of a generator based on a residual network and a discriminator based on a convolutional neural network. The generator transforms the visible light image into a pseudo-infrared image with a structure similar to the real infrared image through downsampling, upsampling, and residual block operations. The discriminator determines the authenticity of the input image, assisting the generator in producing a more realistic pseudo-infrared image. The cross-modal image generation loss function consists of perceptual style transfer loss and cross-regulation loss. The perceptual style transfer loss includes perceptual loss and style loss, respectively improving the generated pseudo-infrared image by improving high-level features and style information. The content term of the cross-regulation loss ensures image content consistency, while the edge term preserves edge information, enhancing image structural clarity. By calculating the loss function and adjusting the CIGN network parameters, the quality of the pseudo-infrared image is improved.

[0147] Image Registration Module: This module employs the Multi-Scale Registration Network (MSRN) to register the infrared image to be registered with the pseudo-infrared image, and optimizes the registration process using a registration loss function. The MSRN network workflow consists of multi-scale feature extraction, coarse deformation field estimation, refined deformation field estimation, and image resampling. First, a shared multi-level feature extractor extracts features at different levels. The deformation field is roughly estimated at the coarsest level, then gradually refined as the level decreases. Finally, at the finest level, a spatial transformation is performed on the infrared image to be registered based on the deformation field to obtain the registered image. The registration loss function consists of a bidirectional similarity loss and a smoothing loss. The forward and backward components of the bidirectional similarity loss constrain the feature space similarity between the distorted infrared image and the pseudo-infrared image, while the smoothing loss ensures the smoothness of the deformation field. This loss function optimizes the MSRN network parameters, improving registration accuracy.

[0148] Image Fusion Module: This module employs a semantically guided fusion network, combining semantic features extracted by a segmentation network. A semantic embedding module enhances the semantic information of the fused image, generating a fused image. The semantically guided fusion network comprises a backbone fusion network and a segmentation network. First, a dual-channel feature extraction network extracts infrared and visible light image features separately, while the semantic segmentation network SegFormer acquires semantic information. Next, the semantic information is embedded into the backbone fusion network through the semantic embedding module. The two cross-attention mechanisms in this module enhance the semantic expressive power of the modal features. Finally, the integrated features are input into the fusion module for fusion reconstruction, generating the fused image. The fusion loss function consists of MS-SSIM loss, gradient loss, and saliency loss. MS-SSIM loss maintains a clear intensity distribution in the fused image, gradient loss encourages the recovery of texture details, and saliency-aware loss preserves salient objects in the image. This loss function optimizes the fusion network parameters, generating a high-quality fused image.

[0149] Joint Training Module: A joint training framework is designed to feed the fusion loss back to the registration network, optimizing the registration effect, and iteratively training according to a specific process. Before training, a set of infrared and visible light images is selected as training data. During iterative training, each iteration includes three steps: first, updating the registration network parameters using the Adam optimizer based on the joint loss function; second, generating a registered infrared image using the current registration network parameters; and third, inputting the registered image into the fusion network and updating the fusion network parameters using the Adam optimizer based on the fusion loss function. The fusion loss function consists of MS-SSIM loss, gradient loss, and saliency loss, evaluating the quality of the fused image from different aspects. Through multiple iterations, the registration and fusion networks collaborate to improve the registration and fusion effects.

[0150] This invention addresses the registration and fusion of infrared and visible light images. By introducing a cross-modal image generation network (CIGN), a multi-scale registration network (MSRN), and a semantically guided fusion network, it solves the problems of large modal differences, low registration accuracy, and poor fusion quality in traditional image registration and fusion methods, achieving high-precision and high-quality image registration and fusion. Specifically, the CIGN network effectively reduces the modal differences between infrared and visible light images and generates realistic pseudo-infrared images through a specific loss function; the MSRN network significantly improves the accuracy of image registration by performing multi-level feature extraction and deformation field estimation on distorted and pseudo-infrared images; and the semantically guided fusion network incorporates rich semantic information, further enhancing the quality and semantic expressive power of the fused image.

[0151] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited to the above embodiments. Obviously, anyone skilled in the art can make various changes and modifications to the present invention without departing from its scope and spirit. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A multimodal image registration and fusion method based on deep learning, characterized in that, The method includes: S1. Image Preprocessing: The visible light image to be registered is processed using a cross-modal image generation network. vis Converted into a pseudo-infrared image with a structure similar to that of a real infrared image. S2. Image Registration: A multi-scale registration network is used to accurately estimate the infrared image to be registered through multi-scale feature extraction and a coarse-to-fine strategy. With pseudo-infrared images The deformation field between them yields the registered infrared image. S3. Image Fusion: Utilizing a semantically guided fusion network, combined with semantic features extracted by a segmentation network. seg By enhancing the semantic information of the fused image through a semantic embedding module, a fused image I is generated. fus ; S4. Joint Training: Design a joint training framework to feed the fusion loss back to the registration network, optimize the registration effect and improve the quality of the fused image.

2. The deep learning-based multimodal image registration and fusion method according to claim 1, characterized in that, The cross-modal image generation network consists of two generators based on residual networks and two discriminators based on convolutional neural networks. The generators learn residual mapping through a series of downsampling and upsampling operations and residual blocks to convert visible light images into pseudo-infrared images with structures similar to real infrared images. The discriminators process the input images through multi-layer convolution and normalization and perform binary classification of real and fake images.

3. The deep learning-based multimodal image registration and fusion method according to claim 1, characterized in that, The operation steps of the multi-scale registration network are as follows: a) Multi-scale feature extraction: using a shared multi-level feature extractor F k k = 1, 2, ..., K, from the infrared image to be registered and pseudo-infrared images Extracting multi-level features, F k Refers to the feature extractor at the k-th scale; b) Coarse Deformation Field Estimation: The coarse deformation field estimation module M is used. C Predict the coarse deformation field between two images First, the correlation map between the two features is calculated. Then, it is concatenated with the original features and processed through a six-layer convolutional neural network to obtain the coarse deformation field. c) Refine the deformation field estimation: refine the coarse deformation field This is provided as input to the refined deformation field estimation module M. R First, the matching cost map is calculated, then processed through a six-layer convolutional neural network. Finally, the output of the convolutional neural network is residually connected to the coarse deformation field to obtain the refined deformation field. d) Image resampling: When k = K, the final deformed field is obtained. The registered infrared image is reconstructed using a resampling layer to obtain the registered infrared image.

4. The deep learning-based multimodal image registration and fusion method according to claim 1, characterized in that, The semantically guided fusion network is designed using a cascade principle, including a backbone fusion network and a segmentation network. It extracts features from infrared and visible light images through a dual-channel feature extraction network, and embeds the semantic features extracted by the segmentation network into the backbone fusion network using a semantic embedding module.

5. The deep learning-based multimodal image registration and fusion method according to claim 4, characterized in that, The semantic embedding module consists of a semantic-driven cross-attention mechanism and a modality-driven cross-attention mechanism. By calculating the global context representation, it enables complementary interaction of features from different subspaces, thereby enhancing the semantic expressive power of modal features. The semantically driven cross-attention mechanism utilizes semantic features f seg Generate query vector Q s Calculate the global context representation G ir and G vis , Where K ir and K vis Representing modal features {f ir ,f vis The corresponding key vector, V ir and V vis Represents the corresponding value vector; by querying the vector Q s With global context representation G ir and G vis Multiply to obtain semantically dominant attention-weighted features S ir and S vis ; The modality-dominated cross-attention mechanism introduces the modality query vector Q. ir Q vis , used to characterize modal features {f ir ,f vis The intrinsic modal properties of}, the key vector K s Sum vector V s From semantic features f seg Generate; through computing the global context of semantic features. Obtain modality-dominated cross-attention weighted features M ir and M vis M ir =Q ir ·G s M vis =Q vis ·G s .

6. The deep learning-based multimodal image registration and fusion method according to claim 1, characterized in that, The joint training method involves feeding the fusion loss back into the registration network, and training the registration network together with the registration loss. The training steps are as follows: a) Initialization: Select a set of infrared and visible light images to construct a training set; b) Iterative training: Perform a maximum of M iterations. Each iteration includes three steps: training the registration network, generating the registration image, and training the fusion network. The registration network training updates the parameters using the Adam optimizer based on the joint loss function obtained by weighting the registration loss and the fusion loss. The fusion network training updates the parameters using the Adam optimizer based on the fusion loss function. c) End of iteration: Repeat the above steps until all iterations are completed.

7. The deep learning-based multimodal image registration and fusion method according to claim 1, characterized in that, The cross-modal image generation loss function of the cross-modal image generation network consists of perceptual style transfer loss and cross-regulation loss. The perceptual style transfer loss is obtained by weighting the perceptual loss and style loss. The perceptual loss is used to ensure that the generated pseudo-infrared images are consistent with the high-level features of real infrared images at different levels of the VGG-19 network, while also considering the differences between visible light images and generated pseudo-infrared images at the same level. The style loss is used to maintain the style features of the generated images. The cross-regulation loss is obtained by weighting the content term and the edge term to ensure that the generated pseudo-infrared images are consistent with real infrared images in terms of content.

8. The deep learning-based multimodal image registration and fusion method according to claim 1, characterized in that, The registration loss of the registration network consists of a bidirectional similarity loss and a smoothing loss; the bidirectional similarity loss includes forward and backward similarity losses, which are used to constrain the feature space similarity between unregistered infrared images and pseudo-infrared images. The forward similarity loss measures the similarity between unregistered infrared images and pseudo-infrared images in the feature space by calculating the differences between corresponding hierarchical features. The backward similarity loss calculates the difference between the pseudo-infrared image after inverse deformation and the original unregistered infrared image, ensuring that not only is the transformation from unregistered infrared image to pseudo-infrared image reasonable, but the similarity requirement is also met when returning from pseudo-infrared image to unregistered infrared image through inverse transformation. The smoothing loss is used to ensure the smoothness of the deformation field by calculating and summing the L1 norm of the deformation field gradient to penalize the drastically changing parts of the deformation field.

9. The deep learning-based multimodal image registration and fusion method according to claim 1, characterized in that, The fusion loss is obtained by weighting MS-SSIM loss, gradient loss, and saliency loss; the MS-SSIM loss is used to maintain a clear intensity distribution in the fused image; the gradient loss is used to encourage the recovery of texture details; and the saliency loss is used to preserve salient objects in the image.

10. A deep learning-based multimodal image registration and fusion system implemented using the method described in any one of claims 1-9, characterized in that, The system includes: Image preprocessing module: Utilizes a cross-modal image generation network to process the visible light image to be registered (I). vis Converted into a pseudo-infrared image with a structure similar to that of a real infrared image. Image registration module: Employs a multi-scale registration network, accurately estimating the infrared image to be registered through multi-scale feature extraction and a coarse-to-fine strategy. With pseudo-infrared images The deformation field between them yields the registered infrared image. Image fusion module: Employs a semantically guided fusion network, combining semantic features f extracted by the segmentation network. seg By enhancing the semantic information of the fused image through a semantic embedding module, a fused image I is generated. fus ; Joint Training Module: Design a joint training framework to feed the fusion loss back to the registration network, optimize the registration effect and improve the quality of the fused image.

Images