Image registration fusion method and device, equipment and storage medium

Abstract

The invention discloses an image registration fusion method and device, equipment and a storage medium. When the method provided by the embodiment of the invention is executed, firstly, a visible light image and an initial infrared image of a target object can be acquired; and inputting the visible light image and the initial infrared image of the target object into the image registration fusion model for registration fusion to obtain a fused image. Wherein the image registration fusion model is used for carrying out style migration on the visible light image of the target object and carrying out deformation estimation and deformation on the initial infrared image of the target object. And the image registration fusion model is also used for carrying out image fusion on the visible light image after style migration and the initial infrared image after deformation to obtain a fused image. And finally, outputting the fused image. The image registration and fusion precision is improved, the fusion effect is optimized, and the final image quality is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly relates to an image registration and fusion method, device, equipment and storage medium. Background Art

[0002] Due to different working principles, different types of imaging devices have their own advantages and limitations. Visible light sensors can provide clear images under normal lighting conditions, but perform poorly in low-light environments. Infrared sensors, on the other hand, can work in the dark, capturing temperature differences and thermal features, and are suitable for fields such as security. To make up for the deficiencies of both, image registration and fusion technologies have emerged to combine images of the two modalities. However, traditional image registration methods rely on techniques such as optical flow estimation and are difficult to handle the registration problems of images with large modality differences, resulting in unsatisfactory registration effects and further affecting the fusion effect. In addition, due to the modality differences between visible light and infrared images, the fusion effect is also affected, leading to a decrease in the quality of the final fused image.

[0003] Therefore, how to overcome the modality differences between visible light and infrared images, improve the accuracy of image registration and fusion, optimize the fusion effect and enhance the quality of the final image is a technical problem that those skilled in the art urgently need to solve. Summary of the Invention

[0004] Based on the above problems, this application provides an image registration and fusion method, device, equipment and storage medium, which can overcome the modality differences between visible light and infrared images, improve the accuracy of image registration and fusion, optimize the fusion effect and enhance the quality of the final image.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] An image registration and fusion method, the method includes:

[0007] Obtain a visible light image and an initial infrared image of a target object;

[0008] Input the visible light image and the initial infrared image of the target object into an image registration and fusion model for registration and fusion to obtain a fused image; the image registration and fusion model is used to perform style transfer on the visible light image of the target object, perform deformation estimation and deformation on the initial infrared image of the target object, and perform image fusion on the style-transferred visible light image and the deformed initial infrared image to obtain the fused image;

[0009] Output the fused image.

[0010] In a possible implementation manner, the image registration and fusion model includes a cross-modal style transfer model, a multi-scale progressive registration model, and a feature interaction pooling and fusion model connected in sequence;

[0011] Among them, the cross-modal style transfer model is used to perform style transfer on the visible light image to generate a pseudo-infrared image; the multi-scale progressive registration model is used to perform deformation estimation on the initial infrared image based on the pseudo-infrared image to obtain the deformation field of the initial infrared image, and the deformation field is used to perform deformation on the initial infrared image to obtain a deformed infrared image; the feature interaction pooling fusion model is used to perform image fusion on the deformed infrared image and the visible light image to obtain the fused image.

[0012] In a possible implementation, the cross-modal style transfer model includes a visible light image style transfer module; the visible light image style transfer module includes a first convolutional layer, a first domain transformation model, and a first diffusion model connected in sequence; the first domain transformation model includes a first downsampling layer, a first bottleneck layer, and a first upsampling layer connected in sequence; the first diffusion model includes a second downsampling layer, a second bottleneck layer, and a second upsampling layer connected in sequence, and the second downsampling layer, the second bottleneck layer, and the second upsampling layer all have an attention mechanism;

[0013] The first convolutional layer is used to perform convolution on the visible light image to obtain a visible light feature map;

[0014] The first downsampling layer is used to perform non-linear transformation processing, reduce spatial resolution operation, increase channel number operation, and ordinary feature enhancement processing on the visible light feature map to obtain a first optimized feature map; the first bottleneck layer is used to perform ordinary feature enhancement processing and local feature extraction processing on the first optimized feature map to obtain a first local feature map; the first upsampling layer is used to perform non-linear transformation processing, restore spatial resolution operation, reduce channel number operation, and ordinary feature enhancement processing on the local feature map to obtain a final feature map;

[0015] The second downsampling layer is used to perform non-linear transformation processing, reduce spatial resolution operation, increase channel number operation, self-attention calculation, and residual connection processing on the final feature map to obtain a second optimized feature map; the second bottleneck layer is used to perform global information enhancement processing based on the attention mechanism and local feature extraction processing on the second optimized feature map to obtain a second local feature map; the second upsampling layer is used to perform non-linear transformation processing, restore spatial resolution operation, reduce channel number operation, and feature enhancement based on the attention mechanism on the second local feature map to obtain the pseudo-infrared image.

[0016] In a possible implementation, the multi-scale progressive registration model includes a two-stream pyramid feature extractor, a first encoder, a displacement estimator, and a feature optimization module connected in sequence; the feature optimization module is also connected to the first encoder;

[0017] Among them, the dual-stream pyramid feature extractor is used to extract features from the initial infrared image and the pseudo-infrared image respectively to obtain a first feature vector and a second feature vector; the first encoder is used to encode the first feature vector and the second feature vector respectively to obtain a first encoded vector and a second encoded vector; the displacement estimator and the feature optimization module are used to perform iterative disparity calculations on multiple feature pairs one by one in combination with the second encoded vector, so as to obtain the deformation field. A feature pair includes the feature values at corresponding positions in the first feature vector and the second feature vector; during the iteration, the displacement estimator processes each feature pair to obtain a disparity value, and the feature optimization module combines the current disparity value and other features for optimization and update until the final deformation field is calculated.

[0018] In a possible implementation manner, the feature interaction pooling fusion model includes a second encoder, a residual link module, an interaction pooling module, and a decoder connected in sequence;

[0019] Among them, the second encoder is used to encode the deformed infrared image and the visible light image respectively to obtain a first encoded feature and a second encoded feature; the residual link module is used to combine the features of the deformed infrared image and the first encoded feature to obtain a first combined image, and combine the features of the visible light image and the second encoded feature to obtain a second combined image; the interaction pooling module is used to perform interaction fusion processing on the first combined image and the second combined image to obtain an initial fusion image; the decoder is used to perform decoding processing on the initial fusion image to obtain the fusion image.

[0020] In a possible implementation manner, the first downsampling layer includes a first normalization module, a first activation function layer, a second convolutional layer, and a first ordinary residual block connected in sequence; the first bottleneck layer includes a second ordinary residual block and a first local feature extraction residual block connected in sequence; the first upsampling layer includes a second normalization module, a second activation function layer, a third convolutional layer, and a third ordinary residual block connected in sequence;

[0021] Among them, the first normalization module is used to perform a normalization operation on the visible light image to obtain a normalized visible light image; the first activation function layer is used to perform a non-linear transformation on the normalized visible light image to obtain an activated visible light image; the second convolutional layer is used to reduce the spatial resolution of the activated visible light image and increase its number of channels to obtain a feature map; the first ordinary residual block is used to enhance the features of the feature map to obtain a first optimized feature map; the second ordinary residual block is used to enhance the features of the first optimized feature map to obtain an optimized feature map; the first local feature extraction residual block is used to extract local features from the optimized feature map to obtain a first local feature map; the second normalization module is used to perform a normalization operation on the first local feature map to obtain a first normalized local feature map; the second activation function layer is used to perform a non-linear transformation on the first normalized local feature map to obtain a first activated local feature map; the third convolutional layer is used to restore the spatial resolution of the first activated local feature map and reduce its number of channels to obtain a first compressed feature map; the third ordinary residual block is used to enhance the features of the first compressed feature map to obtain the final feature map.

[0022] In a possible implementation, the second downsampling layer includes a third normalization module, a third activation function layer, a fourth convolutional layer, and a first attention mechanism residual block connected in sequence; the second bottleneck layer includes a second attention mechanism residual block and a second local feature extraction residual block connected in sequence; the second upsampling layer includes a fourth normalization module, a fourth activation function layer, a fifth convolutional layer, and a third attention mechanism residual block connected in sequence;

[0023] Among them, the third normalization module is used to perform a normalization operation on the final feature map to obtain a normalized feature map; the third activation function layer is used to perform a non-linear transformation on the normalized feature map to obtain an activated feature map; the fourth convolutional layer is used to reduce the spatial resolution of the activated feature map and increase its number of channels to obtain an enhanced feature map; the first attention mechanism residual block is used to perform self-attention calculation and residual connection processing on the enhanced feature map to obtain a second optimized feature map; the second attention mechanism residual block is used to perform global information enhancement on the optimized feature map to obtain a global feature map; the second local feature extraction residual block is used to perform local feature extraction on the global feature map to obtain a second local feature map; the fourth normalization module is used to perform a normalization operation on the second local feature map to obtain a second normalized local feature map; the fourth activation function layer is used to perform a non-linear transformation on the second normalized local feature map to obtain a second activated local feature map; the fifth convolutional layer is used to restore the reduced spatial resolution of the second activated local feature map and reduce its number of channels to obtain a second compressed feature map; the third attention mechanism residual block is used to perform feature enhancement on the second compressed feature map to obtain the pseudo-infrared image.

[0024] An image registration and fusion device, the device includes:

[0025] An acquisition unit, configured to acquire a visible light image and an initial infrared image of a target object;

[0026] A registration and fusion unit, configured to input the visible light image and the initial infrared image of the target object into an image registration and fusion model for registration and fusion to obtain a fused image; the image registration and fusion model is used to perform style transfer on the visible light image of the target object, perform deformation estimation and deformation on the initial infrared image of the target object, and perform image fusion on the style-transferred visible light image and the deformed initial infrared image to obtain the fused image;

[0027] An output unit, configured to output the fused image.

[0028] An image registration and fusion device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the computer program, the image registration and fusion method as described above is implemented.

[0029] A computer-readable storage medium, in which instructions are stored, and when the instructions are run on a terminal device, the terminal device is made to execute the image registration and fusion method as described above.

[0030] Compared with the prior art, the present application has the following beneficial effects:

[0031] The present application provides an image registration and fusion method, apparatus, device, and storage medium. Specifically, when implementing the image registration and fusion method provided in the embodiments of the present application, first, a visible light image and an initial infrared image of a target object are obtained as input data. These two types of images capture different information of the target object through different imaging principles. The visible light image reflects the visible features of the object, while the infrared image reveals temperature differences and thermal features. Next, these two images are input into an image registration and fusion model for processing. The core function of this model is to first perform style transfer on the visible light image, transforming its features into a form suitable for registration with the infrared image; at the same time, perform deformation estimation and deformation on the initial infrared image for better matching with the style-transferred visible light image. Finally, these two processed images are fused to obtain a fused image, which can integrate the advantages of the two modalities and provide clearer and more comprehensive target information. Ultimately, the fused image is output as the final visual data result. The present application effectively solves the problem of modal differences between visible light and infrared images by performing style transfer on the visible light image and performing deformation estimation and deformation on the infrared image at the same time. At the same time, the image registration and fusion model generates a fused image that can retain the texture details of the visible light image while highlighting the thermal features in the infrared image, thus achieving a higher-quality fused image. Description of the Drawings

[0032] To more clearly illustrate the technical solutions in this embodiment or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0033] Figure 1 It is a flowchart of an image registration and fusion method provided in an embodiment of the present application;

[0034] Figure 2 It is a schematic structural diagram of an image registration and fusion model provided in an embodiment of the present application;

[0035] Figure 3 It is a schematic structural diagram of a cross-modal style transfer model provided in an embodiment of the present application;

[0036] Figure 4 It is a schematic structural diagram of a domain transformation model provided in an embodiment of the present application;

[0037] Figure 5 It is a schematic structural diagram of a diffusion model provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of the structure of a multi-scale progressive registration model provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the structure of a feature interaction pooling fusion model provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of the structure of an image registration and fusion device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application will be described below.

[0042] Different types of imaging devices have their own advantages and limitations due to their different working principles. Each sensor is designed for specific application scenarios and frequency ranges, which enables them to capture information in specific bands. Taking common visible light sensors as an example, they mainly capture information in light bands that the human eye can perceive, so they can present clearer and more realistic images under normal lighting conditions. However, in poor lighting conditions, such as low light or dark environments, these visible light sensors often face challenges. Due to the inability to effectively capture enough light, the imaging effect may become blurred, details are lost, and the image lacks layering and depth.

[0043] Compared with the limitations of visible light sensors, infrared sensors capture thermal radiation from objects to form images, so they are not restricted by lighting conditions. Infrared sensors can still work in completely dark environments and provide clear image information. By sensing the thermal characteristics of objects, these sensors can show details that cannot be seen in visible light environments, such as temperature changes, hidden heat sources, and heat distribution of objects. Infrared images are particularly important in monitoring, security and other fields because they can reveal details that are not noticeable in conventional visual range. In low-light environments, images from infrared sensors can effectively fill in the information gaps in visible light images, providing users with a more complete and accurate scene display.

[0044] In order to achieve the complementary advantages of visible light and infrared images, image registration and fusion technology came into being. Through accurate image registration, the image information from two different sensors can be effectively combined to generate a more comprehensive and clear visual effect. By fusing the information of these two modalities, more accurate and comprehensive image representation can be provided under different environments and lighting conditions.

[0045] However, traditional image registration methods usually rely on techniques such as optical flow estimation. These techniques perform well in single-modal image registration. However, in the process of multi-modal image registration, due to the large modal differences between visible light and infrared images, the accuracy of traditional image registration techniques often fails to meet the requirements. Optical flow estimation assumes that the changes in the image are small, while the changes between different modal images are more significant. Therefore, registering different modal images through the optical flow estimation method will result in unsatisfactory registration effects, which in turn affects the quality of the fused image.

[0046] In addition, due to the inherent (i.e., modal) differences between visible light and infrared images, when fusing, it is not only necessary to retain the texture details in the visible light image but also to highlight the thermal features in the infrared image. Therefore, when performing image fusion, although the fusion can combine the information of the two images, due to the inherent differences between the images, the final fusion effect often fails to meet expectations, resulting in a decline in the quality of the final fused image.

[0047] To solve this problem, in the embodiments of the present application, an image registration and fusion method, device, equipment, and storage medium are provided. First, the visible light image and the initial infrared image of the target object can be obtained. Then, these images are input into the image registration and fusion model for registration and fusion to obtain the final fused image. The image registration and fusion model first performs style transfer on the visible light image, and at the same time performs deformation estimation and deformation on the initial infrared image. After that, the model fuses the visible light image after style transfer and the initial infrared image after deformation processing to generate a high-quality fused image. Finally, the fused image is output. The present application effectively solves the modal difference problem between visible light and infrared images by performing style transfer on the visible light image and performing deformation estimation and deformation on the infrared image. The fused image generated by the image registration and fusion model when fusing the style-transferred visible light image and the deformed initial infrared image not only retains the texture details of the visible light image but also can highlight the thermal features in the infrared image, thereby achieving a higher-quality fusion effect.

[0048] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0049] See Figure 1 , which is a flowchart of an image registration and fusion method provided by an embodiment of the present application. As Figure 1 shown, the image registration and fusion method may include steps S101-S103:

[0050] S101: Obtain the visible light image and the initial infrared image of the target object.

[0051] "Obtain the visible light image and the initial infrared image of the target object" means obtaining two types of image data of the same target object from different sensors or devices. The visible light image is an ordinary image visible to the human eye, showing the appearance and color of the target object; while the initial infrared image is an image obtained through an infrared sensor, which reflects the thermal radiation characteristics of the target object, used to show the thermal distribution or temperature change of the object, and usually provides supplementary information for details that cannot be recognized by visible light.

[0052] Exemplarily, assume that the monitoring object (i.e., the target object) is the heat loss of a building. First, obtain the visible light image of the building through an ordinary camera to see the appearance of the building. Then, use an infrared thermal imager to take the infrared image of the building and record the temperature distribution on the building surface. By obtaining these two images simultaneously, the visible light and infrared information can be combined to analyze the heat loss situation of the building, and then effective energy-saving measures can be taken.

[0053] S102: Input the visible light image and the initial infrared image of the target object into an image registration and fusion model for registration and fusion to obtain a fused image.

[0054] Process the visible light image and the initial infrared image of the target object through an image processing model to generate a final fused image.

[0055] Specifically, first input the visible light image and the infrared image into an image registration and fusion model for processing. The main tasks of this model are:

[0056] Style transfer: Perform style transfer on the visible light image to make it more visually in line with the target requirements, which may be to change the color, contrast, etc., to make its visual effect more consistent with that of the infrared image.

[0057] Deformation estimation and deformation: Perform deformation estimation and adjustment on the infrared image so that it is aligned with the visible light image in terms of spatial position, so as to ensure the precise superposition of the two images on the same target.

[0058] Image fusion: Fuse the style-transferred visible light image and the deformed infrared image to generate a fused image that combines the information of both. The purpose of this is to simultaneously retain the details of the visible light image and the thermal distribution information of the infrared image, thus providing more comprehensive visual data.

[0059] Exemplarily, assume that the exterior wall of a building is being analyzed, and its visible light image (showing the appearance) and infrared image (showing the heat distribution) are obtained. After inputting these two images into the image registration and fusion model, the model first adjusts the style of the visible light image to make it easier to combine with the infrared image, then adjusts the position and shape of the infrared image to make it completely aligned with the visible light image, and finally combines the two to obtain a new fused image that can simultaneously display the building appearance and heat distribution for easy analysis.

[0060] See Figure 2 , Figure 2 FIG. is a schematic structural diagram of an image registration and fusion model provided by an embodiment of the present application. Specifically, the image registration and fusion model includes a cross-modal style transfer model, a multi-scale progressive registration model, and a feature interaction pooling and fusion model that are connected in sequence.

[0061] Cross-modal style transfer model: The role of this module is to convert the visible light image into a pseudo-infrared image. Through style transfer technology, the model can adjust the style of the visible light image to be similar to that of the infrared image. Style transfer means that on the basis of retaining the content of the visible light image, its visual features (such as color, texture, etc.) are adjusted to be closer to the style of the infrared image, making the subsequent registration process smoother and more effective.

[0062] Multi-scale progressive registration model: This model is used to perform deformation estimation on the initial infrared image based on the pseudo-infrared image. Through this deformation estimation, the model can calculate a deformation field (i.e., a field representing how to transform the image). This deformation field is used to adjust the initial infrared image to align it with the pseudo-infrared image, thereby obtaining a deformed infrared image (deformed infrared image). Multi-scale progressive registration means that the model gradually performs image alignment at multiple scales, thereby improving the accuracy of registration.

[0063] Feature interaction pooling and fusion model: This model fuses the deformed infrared image with the visible light image. Through feature interaction pooling, the model can perform information interaction at multiple feature levels and combine the useful information of the two images. The finally generated fused image contains the advantages of the visible light image and the infrared image, may have both detailed presentation and heat information, thus providing more visual and analysis information.

[0064] The design purpose of the entire image registration and fusion model is to use cross-modal style transfer to adjust and transform the image, use the multi-scale progressive registration model to accurately align the image, and finally synthesize a more complete and information-rich fused image through feature interaction pooling and fusion. The image processed in this way can simultaneously reflect the details of the visible light image and the heat distribution of the infrared image, and is widely used in medical imaging, remote sensing monitoring, industrial detection and other fields.

[0065] See Figure 3 , Figure 3 which is a schematic structural diagram of a cross-modal style transfer model provided by an embodiment of the present application. Specifically, the cross-modal style transfer model includes a visible light image style transfer module. The visible light image style transfer module includes a first convolutional layer, a first domain transformation model, and a first diffusion model connected in sequence.

[0066] The first convolutional layer can perform a convolutional operation on the input visible light image to generate a visible light feature map. The convolutional operation helps to extract low-level features (such as edges, textures, etc.) from the original image, and these features will provide a basis for subsequent processing.

[0067] In a possible implementation manner, the first domain transformation model includes a first downsampling layer, a first bottleneck layer, and a first upsampling layer connected in sequence. For the schematic structural diagram of the first domain transformation model, see Figure 4 .

[0068] First downsampling layer: This layer processes the visible light feature map through operations such as non-linear transformation, reducing spatial resolution, increasing the number of channels, and enhancing features, to obtain a "first optimized feature map".

[0069] Specifically, non-linear transformation helps to enhance the representation ability of the image; reducing spatial resolution reduces the computational amount and makes the features more representative; increasing the number of channels can provide a richer feature representation for the subsequent model; and feature enhancement helps to improve the significance of the features.

[0070] First bottleneck layer: This layer performs ordinary feature enhancement processing on the "first optimized feature map" after downsampling processing, and performs local feature extraction, and finally obtains a "first local feature map".

[0071] Local feature extraction aims to extract relatively local detailed information in the image, which helps to improve the performance of detailed features.

[0072] First upsampling layer: This layer performs non-linear transformation on the "first local feature map", restores the spatial resolution, reduces the number of channels, and through feature enhancement, finally obtains a "final feature map".

[0073] Restoring the spatial resolution helps to restore the low-resolution feature map to a high-resolution image, and at the same time reducing the number of channels reduces the computational complexity.

[0074] In a possible implementation, the first downsampling layer includes a first normalization module, a first activation function layer, a second convolutional layer, and a first ordinary residual block connected in sequence. The first bottleneck layer includes a second ordinary residual block and a first local feature extraction residual block connected in sequence. The first upsampling layer includes a second normalization module, a second activation function layer, a third convolutional layer, and a third ordinary residual block connected in sequence.

[0075] Specifically, (1) the first downsampling layer includes:

[0076] The first normalization module: Normalizes the input visible light image to obtain a normalized visible light image, enabling the pixel value range of the image to adapt to subsequent processing. The normalized image can improve processing efficiency.

[0077] The first activation function layer: Performs a non-linear transformation on the normalized visible light image to obtain an activated visible light image. Commonly used activation functions include ReLU or Leaky ReLU.

[0078] The second convolutional layer: Performs a convolution operation on the activated visible light image (i.e., the activated visible light image), reducing its spatial resolution and increasing the number of channels to obtain a feature map. Reducing the spatial resolution can reduce the computational amount, while increasing the number of channels provides more feature information for subsequent processing.

[0079] The first ordinary residual block: Enhances the feature map using residual connection, which helps to avoid gradient vanishing, promotes feature transmission and learning, improves network performance, and finally outputs the first optimized feature map.

[0080] (2) The first bottleneck layer includes:

[0081] The second ordinary residual block: Further enhances the features of the first optimized feature map, can extract or correct relatively abstract features, making the features more prominent and effective to obtain an optimized feature map.

[0082] The first local feature extraction residual block: Extracts local features from the optimized feature map (i.e., the optimized feature map), focusing on the detailed parts of the image to obtain the first local feature map. The extraction of local features helps to more precisely capture the subtle differences in the image, providing important information for the final image generation.

[0083] (3) The first upsampling layer includes:

[0084] The second normalization module: Normalizes the first local feature map to make it suitable for subsequent activation and convolution operations to obtain the first normalized local feature map.

[0085] The second activation function layer: performs a non-linear transformation on the normalized first normalized local feature map to obtain the first activated local feature map, similar to the activation function mentioned above.

[0086] The third convolutional layer: through convolution operations, restores the spatial resolution of the first activated local feature map and reduces the number of channels, restores the details and structure of the image, and at the same time reduces the computational complexity, thereby obtaining the first compressed feature map.

[0087] The third ordinary residual block: enhances the features of the restored first activated local feature map (i.e., the first compressed feature map), further improves the quality and expression ability of the image, and finally outputs the enhanced final feature map.

[0088] The entire structure extracts and optimizes a pseudo-infrared image with infrared features from a visible light image through step-by-step feature extraction and enhancement, combined with operations such as convolution, normalization, activation function, and residual block. In the design of each layer, there are clear goals: extract key information through downsampling, enhance features through residual blocks, restore the details of the image through upsampling, and finally convert these features into a form suitable for the pseudo-infrared image.

[0089] In a possible implementation, the first diffusion model includes a second downsampling layer, a second bottleneck layer, and a second upsampling layer connected in sequence, and the second downsampling layer, the second bottleneck layer, and the second upsampling layer all have an attention mechanism. See the structural schematic diagram of the first diffusion model in Figure 5 .

[0090] The second downsampling layer: This layer performs operations such as non-linear transformation, reducing spatial resolution, and increasing the number of channels on the "final feature map", and combines self-attention calculation and residual connection to obtain the "second optimized feature map".

[0091] The self-attention mechanism can help the model focus on the key information in the image and suppress the unimportant parts; the residual connection helps to avoid the problem of gradient disappearance in deep networks and promotes information flow.

[0092] The second bottleneck layer: This layer performs global information enhancement processing and local feature extraction based on the attention mechanism on the "second optimized feature map", and finally obtains the "second local feature map".

[0093] Global information enhancement can strengthen the global features in the image through the attention mechanism, thereby improving the overall information representation of the image.

[0094] The second upsampling layer: This layer performs a non-linear transformation on the "second local feature map", restores the spatial resolution, reduces the number of channels, and performs feature enhancement based on the attention mechanism, and finally generates a pseudo-infrared image.

[0095] The pseudo-infrared image is the product of style transfer of a visible light image and already has visual features similar to those of an infrared image.

[0096] In a possible implementation, the second downsampling layer includes a third normalization module, a third activation function layer, a fourth convolutional layer, and a first attention mechanism residual block connected in sequence. The second bottleneck layer includes a second attention mechanism residual block and a second local feature extraction residual block connected in sequence. The second upsampling layer includes a fourth normalization module, a fourth activation function layer, a fifth convolutional layer, and a third attention mechanism residual block connected in sequence.

[0097] Specifically, (1) the second downsampling layer includes:

[0098] The third normalization module: performs a normalization operation on the feature map (i.e., the final feature map), aiming to standardize the mean and variance of the feature map, thereby eliminating the dimensionality differences between different channels and avoiding gradient explosion or gradient disappearance.

[0099] The third activation function layer: applies a non-linear activation function (such as ReLU) to the normalized feature map (i.e., the standardized feature map).

[0100] The fourth convolutional layer: reduces the spatial resolution of the activated feature map through convolution operations while increasing the number of channels. This helps to extract deeper features from the input feature map, and at the same time performs spatial compression in the convolution operation, reducing unimportant spatial information and focusing on more abstract high-level features.

[0101] The first attention mechanism residual block: performs self-attention calculation on the enhanced feature map (i.e., the enhanced feature map). This mechanism calculates the dependencies between different regions and assigns different weights to each position, enabling the network to focus on more important feature parts. The residual connection helps to avoid possible gradient disappearance problems in deep networks and helps features to propagate better.

[0102] (2) The second bottleneck layer includes:

[0103] The second attention mechanism residual block: improves the ability to capture global information by further enhancing the global information of the feature map (i.e., the optimized feature map). The purpose of this layer is to integrate the information in the feature map, focus on the global context of the input image, and thus enhance the overall features.

[0104] The second local feature extraction residual block: extracts local information from the global feature map and pays attention to the features in local regions of the image. This enables the network to process local features in a finer granularity and capture more local and detailed information.

[0105] (3) The second upsampling layer includes:

[0106] Fourth Normalization Module: It performs a normalization operation on the local feature map (i.e., the second local feature map) to ensure that the features input to the subsequent activation function layer are numerically normalized.

[0107] Fourth Activation Function Layer: It performs a non-linear activation process on the normalized local feature map (i.e., the second normalized local feature map).

[0108] Fifth Convolutional Layer: This convolutional layer is responsible for restoring the spatial resolution of the local feature map (i.e., the second activated local feature map) and reducing its number of channels. This operation can restore the local feature map from low-dimensional compressed features to higher-dimensional spatial information and prepare for generating the final output (pseudo-infrared image).

[0109] Third Attention Mechanism Residual Block: It further enhances the features of the compressed feature map (i.e., the second compressed feature map). The role of this block is to re-strengthen the learning of important features through the self-attention mechanism, making the final output pseudo-infrared image have stronger expressiveness in spatial features.

[0110] The final output of the network is a pseudo-infrared image. Through multiple feature extractions, enhancements, and compressions of the above layers, an image with target features is finally generated, which may be used for tasks such as enhancing infrared image processing, generation, or classification.

[0111] By combining multi-layer normalization, activation, convolution, and self-attention mechanisms, spatial, local, and global features can be fully extracted and enhanced, and finally a target image (pseudo-infrared image) is generated. Each layer realizes different functions, ensuring the layer-by-layer extraction of information, feature strengthening, and effective synthesis. This design is particularly suitable for image generation tasks that require complex feature learning.

[0112] The purpose of this cross-modal style transfer model is to extract rich feature information from visible light images through multiple convolutional layers, downsampling and upsampling layers, bottleneck layers, and attention mechanisms, and convert it into a pseudo-infrared image with the style of an infrared image. This process enables the generated pseudo-infrared image to visually resemble a real infrared image and possess important information contained in the infrared image (such as heat distribution, etc.) through a series of complex feature enhancements, local and global information extractions, and restoration of spatial resolution.

[0113] In a possible implementation, the cross-modal style transfer model further includes an infrared image style transfer module. The infrared image style transfer module also includes a first convolutional layer, a first domain transformation model, and a first diffusion model connected in sequence.

[0114] It should be noted that the "first convolutional layer, first domain transformation model, and first diffusion model" included in the infrared image style transfer module are similar in structure and function to the "first convolutional layer, first domain transformation model, and first diffusion model" included in the visible light image style transfer module, and will not be elaborated here.

[0115] See Figure 6 , Figure 6 which is a schematic structural diagram of a multi-scale progressive registration model provided by an embodiment of the present application. Specifically, the multi-scale progressive registration model includes a two-stream pyramid feature extractor, a first encoder, a displacement estimator, and a feature optimization module connected in sequence. The feature optimization module is also connected to the first encoder.

[0116] Two-stream pyramid feature extractor: The task of this extractor is to process the initial infrared image and the pseudo-infrared image respectively, extract their features, and thus obtain a first feature vector and a second feature vector. The pyramid structure usually means that the image is processed at multiple scales, which helps to extract features at different levels. The first feature vector and the second feature vector contain information about different scales and levels in the image.

[0117] First encoder: The role of the first encoder is to encode the extracted features and convert them into a compact representation form, so as to obtain a first encoding vector and a second encoding vector. The first encoding vector and the second encoding vector represent a high-level abstraction of the image features and can be used for subsequent registration and deformation calculations.

[0118] Displacement estimator: The main task of the displacement estimator is to calculate the deformation amount (or disparity) of each pair of features (feature pairs) in the image, that is, to describe the spatial relationship or displacement between the two images. In each iteration, the displacement estimator processes the feature pairs (for example, (a1, b1), (a2, b2), etc.) respectively and calculates the displacement (ψ) of each pair of features. The displacement estimator uses optical flow to obtain the disparity between two feature maps. The displacement estimator is a multi-layer disparity estimation network, which uses local correlation metrics and depth convolution to extract features, and finally outputs a standardized deformation field, Gaussian blur, and normalization operations through dilated convolution, Gaussian blur, and normalization operations.

[0119] Feature optimization module: The role of the feature optimization module is to gradually optimize and update the deformation field between the images by combining the disparities (such as ψ1, ψ2, ψ3, etc.) calculated by the displacement estimator and the current feature vectors. This module optimizes the deformation field in an iterative manner, making the registration result more and more accurate. The feature optimization module realizes two key steps of feature fusion and enhancement, generates a weight distribution for each feature map, retains the feature importance of the spatial position, and dynamically adjusts the importance between channels, further improving the feature expression ability. The output feature map integrates information from multiple feature sources.

[0120] It should be noted that during the iteration process, the displacement estimator and the feature optimization module work alternately: the displacement estimator calculates the disparity of each feature pair. The feature optimization module optimizes and updates it based on the calculated disparity and other features.

[0121] Exemplarily, assume that the first feature vector is [A1 A2 A3], the second feature vector is [B1 B2 B3], so the first coding vector is [a1 a2 a3], the second coding vector is [b1 b2 b3], and thus there are feature pairs (a1, b1), (a2, b2), and (a3, b3). When the displacement estimator and the feature optimization module perform iterative disparity calculations, the displacement estimator first processes (a1, b1) to obtain ψ1, and then the feature optimization module processes [b1 b2 b3], (a1, b1), and ψ1 to obtain the first intermediate result. Next, the displacement estimator processes (a2, b2) to obtain ψ2, and then the feature optimization module processes the first intermediate result, [b1 b2 b3], and ψ2 to obtain the second intermediate result. Finally, the displacement estimator processes (a3, b3) to obtain ψ3, and then the feature optimization module processes the second intermediate result, [b1 b2 b3], and ψ3 to obtain the final result, and the displacement estimator processes the final result to obtain ψ4.

[0122] The multi-scale progressive registration model calculates the deformation field between two images in a step-by-step optimization manner. It mainly extracts multi-scale features through a two-stream pyramid feature extractor, encodes the features through an encoder, and calculates the final deformation field through the combination of a displacement estimator and a feature optimization module in an iterative process. The ultimate goal is to achieve the precise registration of the two images.

[0123] See Figure 7 , Figure 7 which is a schematic structural diagram of a feature interaction pooling fusion model provided by an embodiment of the present application. Specifically, the feature interaction pooling fusion model includes a second encoder, a residual connection module, an interaction pooling module, and a decoder connected in sequence. The interaction pooling module includes a max pooling layer, an average pooling layer, and a global average pooling layer; the decoder includes a convolutional layer, a sigmoid activation function, and a residual layer.

[0124] Specifically, the second encoder: The second encoder encodes the input deformed infrared image and visible light image to generate corresponding encoded features. Specifically, it encodes the deformed infrared image to obtain the first encoded feature; it encodes the visible light image to obtain the second encoded feature.

[0125] Through encoding, the feature information of the two images is extracted for subsequent processing and fusion.

[0126] Residual Link Module: The residual link module combines the encoded features with the original image to generate an intermediate image with fused information. Specifically, it combines the deformed infrared image with the first encoded feature to obtain the first combined image; and combines the visible light image with the second encoded feature to obtain the second combined image.

[0127] Through residual connection, the original information of the image is retained, and at the same time, encoded features are introduced to enhance the expressive ability of the image.

[0128] Interaction Pooling Module: The interaction pooling module performs interactive fusion on the combined image. Through different pooling operations, it enhances the diversity and expressive ability of the features.

[0129] Specifically, the interaction pooling module performs pooling operations to interact and fuse the information from different features, and finally obtains the initial fused image.

[0130] "Interactive fusion" refers to using operations such as max pooling, average pooling, and global average pooling to extract and fuse features of the image, so as to effectively fuse information of different modalities to generate a preliminary fused image.

[0131] Decoder: The decoder performs decoding processing on the initial fused image to restore the spatial information of the image and obtain the final fused image. Specifically, the convolutional layer is used to extract the fused feature information. The sigmoid activation function is used for pixel-level restoration of the image to ensure that the pixel values of the output image are within an appropriate range (such as [0,1]). The residual layer is used to further enhance the restoration effect of the image through residual connection, so that more details are retained in the decoded image.

[0132] The decoder can restore a high-quality fused image from the initially interactively fused image to complete the final image fusion task.

[0133] Among them, "decoding processing" refers to the decoder performing convolutional operations on the fused image, restoring the spatial resolution of the image, and generating the final fused image through the last activation function (such as sigmoid). The decoder usually restores the detailed information and structure of the image through successive convolutional layers and residual connections, so as to obtain a high-quality fusion result.

[0134] This fusion method improves the quality of the fused image through different levels of feature fusion and pooling operations, and is especially suitable for processing scenarios of multi-modal images, such as the joint analysis of infrared and visible light images.

[0135] S103: Output the fused image.

[0136] The output fused image is the final output result of the model, which is a fused image generated after all the above processing steps. This image synthesizes the information of the deformed infrared image and the visible light image, with higher information content and better visual effects.

[0137] The finally generated fused image contains information from different images and can provide more comprehensive and richer visual features. The output indicates that this fused image will be presented as the result or used for subsequent processing.

[0138] Based on the content of S101 - S103, the working process of the image registration and fusion model includes three main steps: First, obtain the visible light image and the initial infrared image of the target object. These images will be used as input data and processed to achieve a better image fusion effect. Then, input the visible light image and the initial infrared image of the target object into the image registration and fusion model for processing. The functions of this model include three main steps: First, perform style transfer on the visible light image to make its visual style more in line with the target requirements; second, perform deformation estimation and deformation on the initial infrared image to align it more accurately with the visible light image; finally, perform image fusion on the style - transferred visible light image and the deformed initial infrared image to obtain the final fused image. Finally, output the fused image to provide a higher - quality and more information - rich image result. In this application, by performing style transfer on the visible light image and simultaneously performing deformation estimation and deformation processing on the infrared image, the problem of modal differences between visible light and infrared images is effectively solved. The style transfer technology makes the visible light image closer to the infrared image in style, thereby reducing the influence caused by differences in illumination, texture, etc. The deformation estimation and processing of the infrared image ensure that the spatial structures of the two images are well - aligned. In addition, during the image registration and fusion process, the style - transferred visible light image and the deformed initial infrared image are effectively fused. This fusion method can not only retain the detailed texture and structural information in the visible light image but also highlight the thermal features in the infrared image, helping to reveal temperature differences and other thermal components that are difficult to capture in the visible light image. In this way, the finally generated fused image has higher quality, can display rich texture details, and can highlight key information, thereby improving the overall scene presentation effect and analysis accuracy.

[0139] See Figure 8 , Figure 8 is a schematic structural diagram of an image registration and fusion device provided by an embodiment of this application. As Figure 8 shown, the image registration and fusion device includes:

[0140] An acquisition unit 801, configured to acquire a visible light image and an initial infrared image of a target object;

[0141] The registration and fusion unit 802 is configured to input the visible light image and the initial infrared image of the target object into an image registration and fusion model for registration and fusion to obtain a fused image. The image registration and fusion model is used to perform style transfer on the visible light image of the target object, perform deformation estimation and deformation on the initial infrared image of the target object, and perform image fusion on the style-transferred visible light image and the deformed initial infrared image to obtain the fused image.

[0142] The output unit 803 is configured to output the fused image.

[0143] In a possible implementation, the image registration and fusion model includes a cross-modal style transfer model, a multi-scale progressive registration model, and a feature interaction pooling and fusion model that are connected in sequence.

[0144] Among them, the cross-modal style transfer model is used to perform style transfer on the visible light image to generate a pseudo-infrared image. The multi-scale progressive registration model is used to perform deformation estimation on the initial infrared image based on the pseudo-infrared image to obtain a deformation field of the initial infrared image, and the deformation field is used to perform deformation on the initial infrared image to obtain a deformed infrared image. The feature interaction pooling and fusion model is used to perform image fusion on the deformed infrared image and the visible light image to obtain the fused image.

[0145] In a possible implementation, the cross-modal style transfer model includes a visible light image style transfer module. The visible light image style transfer module includes a first convolutional layer, a first domain transformation model, and a first diffusion model that are connected in sequence. The first domain transformation model includes a first downsampling layer, a first bottleneck layer, and a first upsampling layer that are connected in sequence. The first diffusion model includes a second downsampling layer, a second bottleneck layer, and a second upsampling layer that are connected in sequence, and the second downsampling layer, the second bottleneck layer, and the second upsampling layer all have an attention mechanism.

[0146] The first convolutional layer is configured to perform convolution on the visible light image to obtain a visible light feature map.

[0147] The first downsampling layer is configured to perform non-linear transformation processing, spatial resolution reduction operation, channel number increase operation, and ordinary feature enhancement processing on the visible light feature map to obtain a first optimized feature map. The first bottleneck layer is configured to perform ordinary feature enhancement processing and local feature extraction processing on the first optimized feature map to obtain a first local feature map. The first upsampling layer is configured to perform non-linear transformation processing, spatial resolution restoration operation, channel number reduction operation, and ordinary feature enhancement processing on the local feature map to obtain a final feature map.

[0148] The second downsampling layer is used to perform non-linear transformation processing, spatial resolution reduction operation, channel number increase operation, self-attention calculation, and residual connection processing on the final feature map to obtain a second optimized feature map; the second bottleneck layer is used to perform global information enhancement processing and local feature extraction processing based on the attention mechanism on the second optimized feature map to obtain a second local feature map; the second upsampling layer is used to perform non-linear transformation processing, spatial resolution restoration operation, channel number reduction operation, and feature enhancement based on the attention mechanism on the second local feature map to obtain the pseudo-infrared image.

[0149] In a possible implementation manner, the multi-scale progressive registration model includes a two-stream pyramid feature extractor, a first encoder, a displacement estimator, and a feature optimization module connected in sequence; the feature optimization module is further connected to the first encoder;

[0150] Among them, the two-stream pyramid feature extractor is used to extract features from the initial infrared image and the pseudo-infrared image respectively to obtain a first feature vector and a second feature vector; the first encoder is used to encode the first feature vector and the second feature vector respectively to obtain a first encoded vector and a second encoded vector; the displacement estimator and the feature optimization module are used to perform iterative disparity calculation on multiple feature pairs one by one in combination with the second encoded vector to obtain the deformation field, and a feature pair includes the feature values at corresponding positions in the first feature vector and the second feature vector; during the iteration process, the displacement estimator processes each feature pair to obtain a disparity value, and the feature optimization module combines the current disparity value and other features for optimization and update until the final deformation field is calculated.

[0151] In a possible implementation manner, the feature interaction pooling fusion model includes a second encoder, a residual link module, an interaction pooling module, and a decoder connected in sequence;

[0152] Among them, the second encoder is used to encode the deformed infrared image and the visible light image respectively to obtain a first encoded feature and a second encoded feature; the residual link module is used to perform feature combination on the deformed infrared image and the first encoded feature to obtain a first combined image, and perform feature combination on the visible light image and the second encoded feature to obtain a second combined image; the interaction pooling module is used to perform interaction and fusion processing on the first combined image and the second combined image to obtain an initial fusion image; the decoder is used to perform decoding processing on the initial fusion image to obtain the fusion image.

[0153] In a possible implementation, the first downsampling layer includes a first normalization module, a first activation function layer, a second convolutional layer, and a first ordinary residual block connected in sequence; the first bottleneck layer includes a second ordinary residual block and a first local feature extraction residual block connected in sequence; the first upsampling layer includes a second normalization module, a second activation function layer, a third convolutional layer, and a third ordinary residual block connected in sequence;

[0154] Among them, the first normalization module is used to perform a normalization operation on the visible light image to obtain a normalized visible light image; the first activation function layer is used to perform a non-linear transformation on the normalized visible light image to obtain an activated visible light image; the second convolutional layer is used to reduce the spatial resolution of the activated visible light image and increase its number of channels to obtain a feature map; the first ordinary residual block is used to enhance the features of the feature map to obtain a first optimized feature map; the second ordinary residual block is used to enhance the features of the first optimized feature map to obtain an optimized feature map; the first local feature extraction residual block is used to extract local features from the optimized feature map to obtain a first local feature map; the second normalization module is used to perform a normalization operation on the first local feature map to obtain a first normalized local feature map; the second activation function layer is used to perform a non-linear transformation on the first normalized local feature map to obtain a first activated local feature map; the third convolutional layer is used to restore the spatial resolution of the first activated local feature map and reduce its number of channels to obtain a first compressed feature map; the third ordinary residual block is used to enhance the features of the first compressed feature map to obtain the final feature map.

[0155] In a possible implementation, the second downsampling layer includes a third normalization module, a third activation function layer, a fourth convolutional layer, and a first attention mechanism residual block connected in sequence; the second bottleneck layer includes a second attention mechanism residual block and a second local feature extraction residual block connected in sequence; the second upsampling layer includes a fourth normalization module, a fourth activation function layer, a fifth convolutional layer, and a third attention mechanism residual block connected in sequence;

[0156] Among them, the third normalization module is used to perform a normalization operation on the final feature map to obtain a normalized feature map; the third activation function layer is used to perform a non-linear transformation on the normalized feature map to obtain an activated feature map; the fourth convolutional layer is used to reduce the spatial resolution of the activated feature map and increase its number of channels to obtain an enhanced feature map; the first attention mechanism residual block is used to perform self-attention calculation and residual connection processing on the enhanced feature map to obtain a second optimized feature map; the second attention mechanism residual block is used to perform global information enhancement on the optimized feature map to obtain a global feature map; the second local feature extraction residual block is used to perform local feature extraction on the global feature map to obtain a second local feature map; the fourth normalization module is used to perform a normalization operation on the second local feature map to obtain a second normalized local feature map; the fourth activation function layer is used to perform a non-linear transformation on the second normalized local feature map to obtain a second activated local feature map; the fifth convolutional layer is used to restore the reduced spatial resolution of the second activated local feature map and reduce its number of channels to obtain a second compressed feature map; the third attention mechanism residual block is used to perform feature enhancement on the second compressed feature map to obtain the pseudo-infrared image.

[0157] In addition, an embodiment of the present application further provides an image registration and fusion device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the image registration and fusion method described above is implemented.

[0158] In addition, an embodiment of the present application further provides a computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are run on a terminal device, the terminal device is caused to execute the image registration and fusion method described above.

[0159] An embodiment of the present application provides an image registration and fusion device. First, the first acquisition unit 301 acquires mapping data from the circular buffer as the data to be detected. The mapping data is the combined information of system calls that have undergone rule filtering, rule supplementation, and form mapping. Then, the rule filtering unit 302 performs rule filtering on the data to be detected to obtain filtered data. The input unit 303 inputs the filtered data into a pre-constructed abstract syntax tree to obtain a detection result. Subsequently, the extraction unit 304 extracts data associated with the detection result from the mapping data as supplementary data, so that the output data supplementation unit 305 can use the supplementary data to supplement the detection result to obtain an attack behavior output result. Through style transfer of visible light images and deformation estimation and deformation of infrared images, the present application effectively addresses the modality differences between visible light and infrared images. The registration and fusion model combines the processed visible light image and infrared image, ensuring that the generated fusion image not only retains the texture details of the visible light image but also highlights the thermal features in the infrared image, achieving a higher quality fusion effect.

[0160] The above has introduced in detail an image registration and fusion method, device, equipment, and storage medium provided by the present application. The embodiments in the specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple. For the relevant parts, refer to the description in the method part. It should be noted that for those of ordinary skill in the art in the technical field of the present application, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

[0161] It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously. Here, A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (one) of the following" or its similar expression refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0162] It should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

Claims

1. An image registration and fusion method, characterized in that The method includes: Obtaining a visible light image and an initial infrared image of a target object; Inputting the visible light image and the initial infrared image of the target object into an image registration and fusion model for registration and fusion to obtain a fused image; the image registration and fusion model is used to perform style transfer on the visible light image of the target object, perform deformation estimation and deformation on the initial infrared image of the target object, and perform image fusion on the visible light image after style transfer and the deformed initial infrared image to obtain the fused image; Outputting the fused image.

2. The method according to claim 1, characterized in that, The image registration and fusion model includes a cross-modal style transfer model, a multi-scale progressive registration model, and a feature interaction pooling and fusion model connected in sequence; Among them, the cross-modal style transfer model is used to perform style transfer on the visible light image to generate a pseudo-infrared image; the multi-scale progressive registration model is used to perform deformation estimation on the initial infrared image based on the pseudo-infrared image to obtain a deformation field of the initial infrared image, and the deformation field is used to perform deformation on the initial infrared image to obtain a deformed infrared image; the feature interaction pooling and fusion model is used to perform image fusion on the deformed infrared image and the visible light image to obtain the fused image.

3. The method according to claim 2, wherein The cross-modal style transfer model includes a visible light image style transfer module; the visible light image style transfer module includes a first convolutional layer, a first domain transformation model, and a first diffusion model connected in sequence; the first domain transformation model includes a first downsampling layer, a first bottleneck layer, and a first upsampling layer connected in sequence; the first diffusion model includes a second downsampling layer, a second bottleneck layer, and a second upsampling layer connected in sequence, and the second downsampling layer, the second bottleneck layer, and the second upsampling layer all have an attention mechanism; The first convolutional layer is used to perform convolution on the visible light image to obtain a visible light feature map; The first downsampling layer is used to perform non-linear transformation processing, reduce spatial resolution operation, increase channel number operation, and ordinary feature enhancement processing on the visible light feature map to obtain a first optimized feature map; the first bottleneck layer is used to perform ordinary feature enhancement processing and local feature extraction processing on the first optimized feature map to obtain a first local feature map; the first upsampling layer is used to perform non-linear transformation processing, restore spatial resolution operation, reduce channel number operation, and ordinary feature enhancement processing on the local feature map to obtain a final feature map; The second downsampling layer is used to perform non-linear transformation processing, reduce spatial resolution operation, increase channel number operation, self-attention calculation, and residual connection processing on the final feature map to obtain a second optimized feature map; the second bottleneck layer is used to perform global information enhancement processing based on the attention mechanism, local feature extraction processing on the second optimized feature map, and obtain a second local feature map; the second upsampling layer is used to perform non-linear transformation processing, restore spatial resolution operation, reduce channel number operation, and feature enhancement based on the attention mechanism on the second local feature map to obtain the pseudo-infrared image.

4. The method according to claim 2, wherein The multi-scale progressive registration model includes a dual-stream pyramid feature extractor, a first encoder, a displacement estimator, and a feature optimization module connected in sequence; the feature optimization module is also connected to the first encoder; Among them, the dual-stream pyramid feature extractor is used to extract features from the initial infrared image and the pseudo-infrared image respectively to obtain a first feature vector and a second feature vector; the first encoder is used to encode the first feature vector and the second feature vector respectively to obtain a first encoded vector and a second encoded vector; the displacement estimator and the feature optimization module are used to perform iterative disparity calculations on multiple feature pairs one by one in combination with the second encoded vector, so as to obtain the deformation field. A feature pair includes the feature values at corresponding positions in the first feature vector and the second feature vector; during the iteration process, the displacement estimator processes each feature pair to obtain a disparity value, and the feature optimization module combines the current disparity value and other features for optimization and update until the final deformation field is calculated.

5. The method according to claim 2, wherein The feature interaction pooling fusion model includes a second encoder, a residual connection module, an interaction pooling module, and a decoder connected in sequence; Among them, the second encoder is used to encode the deformed infrared image and the visible light image respectively to obtain a first encoded feature and a second encoded feature; the residual connection module is used to combine the features of the deformed infrared image and the first encoded feature to obtain a first combined image, and combine the features of the visible light image and the second encoded feature to obtain a second combined image; the interaction pooling module is used to perform interaction fusion processing on the first combined image and the second combined image to obtain an initial fusion image; the decoder is used to decode the initial fusion image to obtain the fusion image.

6. The method according to claim 3, wherein The first downsampling layer includes a first normalization module, a first activation function layer, a second convolutional layer, and a first ordinary residual block connected in sequence; the first bottleneck layer includes a second ordinary residual block and a first local feature extraction residual block connected in sequence; the first upsampling layer includes a second normalization module, a second activation function layer, a third convolutional layer, and a third ordinary residual block connected in sequence; Among them, the first normalization module is used to perform a normalization operation on the visible light image to obtain a normalized visible light image; the first activation function layer is used to perform a non-linear transformation on the normalized visible light image to obtain an activated visible light image; the second convolutional layer is used to reduce the spatial resolution of the activated visible light image and increase its number of channels to obtain a feature map; the first ordinary residual block is used to enhance the features of the feature map to obtain a first optimized feature map; the second ordinary residual block is used to enhance the features of the first optimized feature map to obtain an optimized feature map; the first local feature extraction residual block is used to extract local features from the optimized feature map to obtain a first local feature map; the second normalization module is used to perform a normalization operation on the first local feature map to obtain a first normalized local feature map; the second activation function layer is used to perform a non-linear transformation on the first normalized local feature map to obtain a first activated local feature map; the third convolutional layer is used to restore the spatial resolution of the first activated local feature map and reduce its number of channels to obtain a first compressed feature map; the third ordinary residual block is used to enhance the features of the first compressed feature map to obtain the final feature map.

7. The method according to claim 3, wherein The second downsampling layer includes a third normalization module, a third activation function layer, a fourth convolutional layer, and a first attention mechanism residual block connected in sequence; the second bottleneck layer includes a second attention mechanism residual block and a second local feature extraction residual block connected in sequence; The second upsampling layer includes a fourth normalization module, a fourth activation function layer, a fifth convolutional layer, and a third attention mechanism residual block connected in sequence; Among them, the third normalization module is used to perform a normalization operation on the final feature map to obtain a standardized feature map; the third activation function layer is used to perform a non-linear transformation on the standardized feature map to obtain an activated feature map; the fourth convolutional layer is used to reduce the spatial resolution of the activated feature map and increase its number of channels to obtain an enhanced feature map; The first attention mechanism residual block is used to perform self-attention calculation and residual connection processing on the enhanced feature map to obtain a second optimized feature map; the second attention mechanism residual block is used to enhance the global information of the optimized feature map to obtain a global feature map; the second local feature extraction residual block is used to extract local features from the global feature map to obtain a second local feature map; the fourth normalization module is used to perform a normalization operation on the second local feature map to obtain a second normalized local feature map; the fourth activation function layer is used to perform a non-linear transformation on the second normalized local feature map to obtain a second activated local feature map; the fifth convolutional layer is used to restore the reduced spatial resolution of the second activated local feature map and reduce its number of channels to obtain a second compressed feature map; the third attention mechanism residual block is used to enhance the features of the second compressed feature map to obtain the pseudo-infrared image.

8. An image registration and fusion device, characterized in that, The device includes: An acquisition unit, configured to acquire a visible light image and an initial infrared image of a target object; A registration and fusion unit, configured to input the visible light image and the initial infrared image of the target object into an image registration and fusion model for registration and fusion to obtain a fused image; the image registration and fusion model is configured to perform style transfer on the visible light image of the target object, perform deformation estimation and deformation on the initial infrared image of the target object, and perform image fusion on the visible light image after style transfer and the deformed initial infrared image to obtain the fused image; An output unit, configured to output the fused image.

9. An image registration and fusion device, characterized in that, It includes: A memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the image registration and fusion method according to any one of claims 1-7 is implemented.

10. A computer-readable storage medium, characterized in that, Instructions are stored in the computer-readable storage medium. When the instructions run on the terminal device, the terminal device is caused to execute the image registration and fusion method according to any one of claims 1-7.

Images