Infrared and visible light image fusion feature enhancement method

Abstract

The invention relates to the field of machine vision equipment, in particular to an infrared and visible light image fusion feature enhancement method, which comprises the following steps: image preprocessing: an infrared image and a visible light image are two images with different modalities obtained by using an infrared camera and a visible light camera in the same scene, and the resolution of the light image is taken as a reference; adjusting the resolution of the infrared image to be consistent with the pixel resolution of the visible light image, and then cutting a region (H, W) which is consistent in position and is the same in size from the visible light image by taking the region (H, W) of the infrared image as a reference; the invention discloses a preprocessing network. Respectively carrying out global structure texture detail extraction on an infrared image region and a visible light image region, extracting an infrared feature image and a visible light feature image of the 8-channel region, and obtaining a 16-channel enhanced preprocessing feature map; the preprocessed feature map comprises eight regional infrared feature images and eight regional visible light feature images;

Description

Technical Field

[0001] This invention relates to the field of machine vision equipment, and mainly focuses on enhancing the feature extraction capability of infrared / visible multimodal images for use in infrared / visible fusion image target detection equipment. Background Technology

[0002] Infrared and visible light multimodal image fusion technology makes up for the lack of information representation in single-spectral images, and can improve visual detection capabilities in fields such as security, night driving and military reconnaissance.

[0003] Visible light sensors are highly dependent on illumination, and their performance degrades in low-light or nighttime environments. Infrared sensors, on the other hand, have low resolution and struggle to effectively capture details and color information. Single-modal sensors cannot meet the demands for comprehensive and accurate information acquisition in complex scenarios, posing a challenge to applications such as security monitoring, nighttime driving, and military reconnaissance. To address this issue, infrared and visible light fusion technology combines multimodal image information to overcome the limitations of single-modal information representation, effectively improving target detection capabilities in night vision environments.

[0004] Due to limitations in imaging sensor technology, infrared images have significantly lower resolution and size than visible light images. Furthermore, image scale variations and rotations can cause image misalignment during registration using single feature matching methods, leading to detail blurring during fusion due to feature inaccuracies. Existing multimodal image fusion methods also suffer from shortcomings in handling resolution differences and feature extraction capabilities.

[0005] (1) Infrared and visible light images often differ significantly in resolution and size (infrared: approximately 10). 5 Pixels; Visible light: Approximately 10 6 Because infrared images typically have low resolution and small size, and because they lack registration, existing fusion algorithms struggle to handle misaligned infrared images. Tiny misalignments can produce artifacts in edge and detail areas, significantly reducing the quality of the fused image.

[0006] (2) Traditional fusion methods retain few detailed features. Multimodal images contain shared basic structural features (such as road and building outlines) while also adding complementary detailed features (such as infrared thermal radiation and visible light texture). Multimodal images contain shared basic structural features (such as road and building outlines) and complementary detailed features (such as infrared thermal radiation and visible light texture). Figure 6 As shown, the wavelet transform-based fusion method failed to effectively preserve these detailed features, resulting in problems such as loss of water surface ripples and tree edge artifacts in the fused image.

[0007] To address the aforementioned issues, this patent proposes a multimodal image registration and fusion network based on feature decomposition, which focuses on improving the problems of resolution differences, registration accuracy, and preservation of detail information between infrared and visible light images. Summary of the Invention

[0008] The technical problem to be solved by this invention is: how to provide a method for enhancing the fusion information of infrared and visible light images that are inconsistent in image resolution and photosensitive features.

[0009] The technical solution adopted in this invention is: a method for enhancing the fusion features of infrared and visible light images, comprising the following steps:

[0010] Step 1: Image preprocessing. The infrared image and the visible light image are two images of different modalities obtained using an infrared camera and a visible light camera from the same scene. Based on the resolution of the light image, the resolution of the infrared image is adjusted to be consistent with the pixel resolution of the visible light image. Then, based on a region (H, W) in the infrared image, a region (H, W) of the same size and position is cropped out from the visible light image.

[0011] Step 2: In the preprocessing network, global structural texture detail extraction is performed on regions (H, W) of both the infrared image and the visible light image, extracting 8-channel infrared and visible light feature images of the regions (H, W) to obtain a 16-channel enhanced preprocessed feature map F. 1 , Preprocessed feature map F 1 Includes 8 infrared feature images of region (H, W) and 8 visible light feature images of region (H, W);

[0012] Step 3: In the multi-scale shared feature extraction network, the preprocessed feature map F... 1 The process yields a 32-channel (H / 2, W / 2) feature map F with identical weights across the three convolutional layers, allowing for sharing and reuse. 2 64-channel (H / 4, W / 4) feature map F 3 Feature map F of 128 channels (H / 8, W / 8) 4 ,in,

[0013] Step 4: F 1 F 2 F 3 F 4 The feature map is input into the shift estimation module, and the infrared and visible light fused image is obtained by calculating local correlation and shift calibration in a hierarchical manner.

[0014] In step one, when adjusting the resolution of the infrared image to match the pixel resolution of the visible light image, based on the resolution of the light image, a bilinear interpolation or nonlinear interpolation algorithm is used for the infrared image to improve the pixel resolution of the infrared image to match the pixel resolution of the visible light image.

[0015] In step two, the preprocessing network uses two independent feature extraction networks: a 3×3 convolutional layer 1 and a 3×3 convolutional layer 2, to extract global structural texture details from visible light and infrared images, respectively. Each feature extraction network consists of a convolutional layer with a depth of 4 and a kernel size of 3×3, and the number of channels is set to 8.

[0016] In step three, the feature images are merged in the multi-scale shared feature extraction network, first the infrared feature images and then the visible light feature images. The merged feature image set is then passed through a 3×3 convolutional layer in the multi-scale shared feature extraction network to obtain a 32-channel (H / 2, W / 2) feature map F. 2 32-channel (H / 2, W / 2) feature map F 2 A 64-channel (H / 4, W / 4) feature map F is obtained through a 3×3 convolutional layer in a multi-scale shared feature extraction network. 3 64-channel (H / 4, W / 4) feature map F 3 A 128-channel (H / 8, W / 8) feature map F is obtained through a 3×3 convolutional layer in a multi-scale shared feature extraction network. 4 The 3×3 convolutional layers in the multi-scale shared feature extraction network have the same weights and can be shared and reused. Each convolutional layer has a depth of 4, a kernel size of 3×3, and 32, 64, and 128 channels, respectively. The three-layer network structure parameters are shared.

[0017] In step three, the preprocessed feature map F 1 Including infrared preprocessing feature maps and visible light preprocessing feature map 32-channel (H / 2, W / 2) feature map H 2 Including infrared feature maps and visible light feature map 64-channel (H / 4, W / 4) feature map F 3 Including infrared feature maps and visible light feature map 128-channel (H / 8, W / 8) feature map F 4 Including infrared feature maps and visible light feature map

[0018] In step four, the displacement estimation module processes the infrared feature map. and visible light feature map Processing to obtain the displacement matrix H 3 According to the displacement matrix H 3 infrared feature map Perform shift correction to generate a calibrated infrared feature map. In the displacement estimation module, the visible light feature map... and Processing to obtain the displacement matrix H 2 According to the displacement matrix H 2 infrared feature map Perform shift correction to generate a calibrated infrared feature map. In the displacement estimation module, the visible light feature map... and infrared feature map Processing to obtain the displacement matrix H 1 According to the displacement matrix H 1 Infrared preprocessing feature map Perform shift correction to generate calibrated infrared preprocessed feature maps. In the displacement estimation module, the visible light preprocessed feature map is processed. and infrared preprocessing feature map Processing to obtain the displacement matrix H 0 According to the displacement matrix H 0 The original infrared image is shift-corrected to generate a calibrated infrared feature map. Will The infrared and visible light images are overlaid on the original visible light image to generate a fused infrared / visible light image. In step four, the displacement estimation module consists of three convolutional layers with a depth of 4, 64 channels, and a kernel size of 3×3. The equivalent receptive field of the convolution is:

[0019]

[0020] Among them, R n s represents the size of the receptive field of the nth layer. j Let w represent the stride of the j-th layer, w be the kernel size, and l be a variable (integer) from 1 to n. When using three convolutional layers, the R3 is 7. The output features are used to calculate local correlations.

[0021]

[0022] Where C k F is the correlation feature map of the k-th local block. vis It is a visible light characteristic. It is the kth local block of the infrared feature, c is the number of channels, x and y represent the pixel positions, and d is a variable (integer) of 1-c.

[0023] The local correlation feature map is then input into the upsampling layer to obtain the preliminary displacement matrix H:

[0024]

[0025] H = UP((L4…L1)(Concat(C k ,F ir (4)

[0026] Where L i (X) is a dilated convolutional layer, where X is the input feature map, and W... i ,b i F represents the convolutional kernel and bias of the i-th layer, respectively, and σ represents the non-linear activation function. ir Infrared feature map and It is the kth local block of the infrared feature, and UP represents the upsampling operation.

[0027] In step four, during shift calibration, the original infrared features are calibrated using a displacement matrix. First, a normalized regular grid G ​​with the same spatial dimension (h,w) as the input feature map F is generated. Each point in grid G ​​corresponds to the position (I',j') of the output feature map, and its coordinates are normalized to the range [-1,1].

[0028]

[0029] The input displacement matrix H is then applied to the regular mesh G to generate a deformed sampling mesh G. def For mesh G def The sampling coordinates of each point p in the source feature map are given by the following formula:

[0030] G def (p ′ i,j ) = p ′ i,j +H(p ′ i,j (6)

[0031] Finally, using the deformed sampling mesh G def For the source feature map F src Resampling is performed to obtain the final registered feature map F. warped Since the sampling coordinates are usually non-integer, we use bilinear interpolation to calculate...

[0032] Calculate the output value:

[0033]

[0034] in, Let F represent the set of four nearest neighbor integer pixel coordinates around the sampling point, where q is the coordinate within the neighborhood, and F is the set of coordinates within the neighborhood. src(q) is the value of the source feature map at q, and k is the bilinear interpolation kernel function.

[0035] The superior effects of this invention are as follows:

[0036] Infrared and visible light images are input into an infrared and visible light multi-scale structural feature extraction network. Due to the significant differences in information features between infrared and visible light images, a preprocessing network is used to extract global structural texture details, which enhances the basic features of each image and reduces cross-modal feature interference.

[0037] An infrared image registration network based on multi-scale structural feature extraction simultaneously captures global contour information and local detail information. It employs a cascaded iterative strategy, refining the displacement estimation layer by layer—using mid-level feature maps, shallow feature maps, and preprocessed feature maps—to optimize displacement estimation upwards, ultimately obtaining a high-precision displacement deviation matrix. Accurate image registration is then achieved through a spatial transformation network. This network can achieve precise pixel-level registration of infrared and visible light images of different sizes, resolutions, and those exhibiting rotation and distortion, realizing the registration and fusion of original feature information from infrared and visible light images.

[0038] In the feature extraction network, the visible light image and the infrared image are respectively obtained through independent feature extraction networks to obtain the global structural texture details of the visible light image and the infrared image. Then, a shared convolutional network with three scales decreasing is used to extract the key structural information in the registration task. This method can reduce the computational cost of the network. Attached Figure Description

[0039] Figure 1 This is a structural diagram of a feature enhancement method for infrared and visible light image fusion.

[0040] Figure 2 Basic principles of registration for infrared and visible light images of different resolutions;

[0041] Figure 3 For infrared and visible light image preprocessing network structure;

[0042] Figure 4 For multi-scale shared feature extraction networks;

[0043] Figure 5 This is an infrared and visible light image registration structure based on multi-scale structural features;

[0044] Figure 6 This is a schematic diagram showing the detailed information of a multimodal fused image. Detailed Implementation

[0045] An infrared and visible light image fusion feature enhancement method, such as Figure 1As shown, two images of different modalities, namely infrared and visible light images, are obtained using an infrared camera and a visible light camera in the same scene. Then, the resolution of the infrared and visible light images is matched by a resolution matching module. Next, the central regions of the infrared and visible light images are cropped and extracted by a cropping and extraction module. Then, the features of the visible light image are preprocessed in a preprocessing network. Then, the shared multi-scale structural features of the infrared and visible light images are extracted and registered in a multi-scale shared feature extraction network. Finally, the local correlation and shift calibration of the infrared and visible light images are calculated hierarchically and stepwise in a shift estimation module to obtain the infrared and visible light fused image.

[0046] Specifically, the following steps are included:

[0047] Step 1: Image preprocessing (registration of visible light and infrared images). The infrared and visible light images are two images of different modalities obtained using an infrared camera and a visible light camera from the same scene. Based on the resolution of the visible light image, the resolution of the infrared image is adjusted to match the pixel resolution of the visible light image. Then, based on a region (H, W) in the infrared image, a region (H, W) of the same size and position is cropped out from the visible light image.

[0048] like Figure 2 As shown, the basic principle of visible light image and infrared image registration is as follows: a region in the visible light image is cropped out that is in a position consistent with that in the infrared image; four layers of feature information of the infrared and visible light (cropped region) images are extracted through a neural network, and alignment network parameters are generated; image registration is performed through layer-by-layer feature matching to generate a fused image of visible light and infrared, while compensating for the loss of fused image features caused by resolution mismatch.

[0049] Infrared images have lower resolution than visible light images. To address this difference in resolution, the visible light image resolution is used as a baseline. For the infrared image, algorithms such as bilinear interpolation or nonlinear interpolation are employed to improve the pixel resolution, making it consistent with the visible light image resolution. Consistent pixel resolution means that the distance between adjacent pixels in the infrared and visible light images is consistent.

[0050] Using the infrared image resolution (H, W) as a benchmark, a fixed and relatively consistent region is cropped from the visible light image, with a visible light image resolution of (H, W). The selection of a fixed and relatively consistent region is ensured by the installation positions of the infrared and visible light cameras. During this stage, there is a positional frequency shift between the infrared and visible light images. Furthermore, since the infrared image is obtained through interpolation from the original infrared image, some features of the infrared image are lost.

[0051] Step 2: In the preprocessing network, global structural texture detail extraction is performed on regions (H, W) of both the infrared image and the visible light image, extracting 8-channel infrared and visible light feature images of the regions (H, W) to obtain a 16-channel enhanced preprocessed feature map F. 1 , Preprocessed feature map F 1 It includes 8 infrared feature images of region (H, W) and 8 visible light feature images of region (H, W).

[0052] like Figure 3 As shown, the infrared and visible light images of the cropped region are fed into a preprocessing network for global structural texture detail extraction, resulting in 8-channel infrared and visible light feature images with resolutions of (H, W). The preprocessing network employs two independent feature extraction networks: a 3×3 convolutional layer and a 3×3 convolutional layer, to extract global structural texture details from the visible light and infrared images, respectively. Each feature extraction network consists of a convolutional layer with a depth of 4 and a kernel size of 3×3, with 8 channels to avoid feature confusion caused by weight sharing, which could affect the registration effect. The output is an enhanced preprocessed feature map. It includes 8 infrared feature images with a resolution of (H, W) and 8 visible light feature images with a resolution of (H, W). The output preprocessed feature map contains the global structural texture details of the image while preserving the local structural features of the original image.

[0053] The 3×3 convolutional layer 1 and the 3×3 convolutional layer are two weight-independent convolutional networks, with weights obtained through training. Eight infrared feature images with resolution (H, W) and eight visible light feature images with resolution (H, W) are combined by performing a feature image set, first the infrared feature images and then the visible light feature images, outputting a 16-channel preprocessed feature map with resolution (H, W).

[0054] Step 3: In the multi-scale shared feature extraction network, the preprocessed feature map F... 1 The process yields a 32-channel (H / 2, W / 2) feature map F with identical weights across the three convolutional layers, allowing for sharing and reuse. 2 64-channel (H / 4, W / 4) feature map F 3 Feature map F of 128 channels (H / 8, W / 8) 4 ,in,

[0055] like Figure 4 As shown, the merged F 1The feature image set is fed into a 3×3 convolutional layer 3 in a multi-scale shared feature extraction network, which outputs a 32-channel (H / 2, W / 2) feature map with a resolution of (H / 2, W / 2).

[0056] 32-channel F 2 The feature maps are fed into a 3×3 convolutional layer 3 in a multi-scale shared feature extraction network, outputting a 64-channel (H / 4, W / 4) feature map.

[0057] 64-channel F 3 The feature maps are fed into a 3×3 convolutional layer 3 in a multi-scale shared feature extraction network, which outputs a 128-channel (H / 8, W / 8) feature map.

[0058] The three convolutional network layers in the multi-scale shared feature extraction module have identical weights, allowing for shared reuse. Each convolutional layer has a depth of 4, a kernel size of 3×3, and channel counts of 32, 64, and 128, respectively, sharing the same network structure parameters. Feature maps F of (H / 2, W / 2) with 32 channels are extracted respectively. 2 64-channel (H / 4, W / 4) feature map F 3 Feature map F of 128 channels (H / 8, W / 8) 4 The feature map F of the 32 channels (H / 2, W / 2) 2 It mainly contains fine-grained information such as edges and textures; a 64-channel (H / 4, W / 4) feature map F 3 Feature images capture local structural information within a large region; 128-channel (H / 8, W / 8) feature map F 4 Feature images capture the global geometric structure and target semantic boundaries of the entire image, which helps to align the overall structure and target semantics between modalities. The merged features first enter the first shared convolutional layer, generating a 32-channel (H / 2, W / 2) feature map F. 2 This provides support for accurate pixel-level registration; then it enters the second convolutional layer, generating a 64-channel (H / 4, W / 4) feature map F. 3 The first layer enhances the consistency of the local structure; the second layer extracts deep features, resulting in a 128-channel (H / 8, W / 8) feature map F. 4 It extracts the spatial layout and high-level structural semantics of images, providing structural constraints for cross-modal global registration.

[0059] Step 4: F 1 F 2 F 3 F 4The feature map is input into the shift estimation module, and the infrared and visible light fused image is obtained by calculating local correlation and shift calibration in a hierarchical manner.

[0060] like Figure 5 As shown, the displacement estimation module obtains the displacement field by calculating local correlations. Deep features have a global structural representation and are easier to align. This model starts from deep features and obtains the initial displacement field through the displacement estimation module. First, F... vis ,F ir The convolutional layers consist of three layers with a depth of 4, 64 channels, and a kernel size of 3×3. The equivalent receptive field of the convolution is:

[0061]

[0062] Among them, R n s represents the size of the receptive field of the nth layer. j This represents the stride of the j-th layer. K is the kernel size, and l is a variable from 1 to n. Using three convolutional layers results in an R3 of 7, which reduces the number of block regions and avoids introducing irrelevant long-range information. Subsequently, the output features are subjected to local correlation calculation:

[0063]

[0064] Where C k F is the correlation feature map of the k-th local block. vis It is a visible light characteristic. This represents the k-th local block of the infrared features, where c is the number of channels. By calculating the correlation of each local feature block, the spatial correspondence between features is effectively captured, providing rich matching information for subsequent displacement field estimation. The local correlation feature map is then input into the upsampling layer to obtain the preliminary displacement matrix H:

[0065]

[0066] Where L i (X) is a dilated convolutional layer, where X is the input feature map, and W... i ,b i These represent the convolutional kernel and bias of the i-th layer, respectively, with σ representing the nonlinear activation function. By progressively increasing the expansion rate layer by layer, the receptive field is expanded while maintaining the feature map resolution, thereby effectively aggregating global information to optimize displacement estimation. Finally, the initial displacement field output by the cascade is upsampled and amplified to the target size to obtain the final displacement matrix H.

[0067] Shift calibration module

[0068] The displacement calibration module calibrates the original infrared features using a displacement matrix H, starting from a deep level and performing multi-scale calibration to ultimately obtain a high-precision registered infrared image. First, it generates an image that matches the input feature map F.

[0069] A normalized regular grid G ​​with the same spatial dimensions (H, W). Each point in the grid corresponds to the position (i, j) of the output feature map, and its coordinates are normalized to the range [-1, 1].

[0070]

[0071] The input displacement matrix H is then applied to the regular mesh G to generate a deformed sampling mesh.

[0072] For each point p in the grid, its sampled coordinates on the source feature map are given by the following formula:

[0073] G def (p ′ ) = p ′ +H(p ′ (7)

[0074] Finally, using the deformed sampling mesh G def For the source feature map F src Resampling is performed to obtain the final registered feature map F. warped Since the sampling coordinates are usually non-integer, we use bilinear interpolation to calculate...

[0075] Calculate the output value:

[0076]

[0077] in, Let F represent the set of four nearest neighbor integer pixel coordinates around the sampling point, where q is the coordinate within the neighborhood, and F is the set of coordinates within the neighborhood. src (q) is the value of the source feature map at q, and k is the bilinear interpolation kernel function.

[0078] Through the synergistic effect of displacement estimation and calibration modules, this network is able to learn end-to-end the ability to accurately estimate complex deformations from multi-scale features, laying the foundation for subsequent high-quality image fusion.

[0079] This invention provides a method for training the overall model, where the network's total loss function L... Reg Including photometric loss L PH and endpoint error loss L EP .

[0080] Photometric loss L PH

[0081] Photometric loss is intended to ensure the infrared characteristics corrected by the registration network. In terms of pixel intensity, it should be similar to the original infrared image without distortion. irHighly consistent. This loss function is particularly sensitive to textured regions, driving the network to learn accurate local detail alignment. Its definition is as follows:

[0082]

[0083] in: It is the registration feature obtained after displacement calibration. F ir These are the original infrared features. H and W represent the height and width of the image, respectively. ||·||1 represents the L1 norm, which calculates the sum of the absolute intensity differences pixel-by-pixel between the prediction and the ground truth.

[0084] Endpoint error loss L EP

[0085] Endpoint error loss directly supervises the displacement field output by the network in the geometric deformation domain. It ensures alignment accuracy in global and weakly textured regions by minimizing the geometric distance between the network-predicted deformation field and the true deformation field. Its definition is as follows:

[0086]

[0087] Wherein: H ir′→vis It is the direct prediction of the registration network at a specific scale, used for aligning F. ir The deformation field. H gt This is the true deformation field. ||·||2 represents the L2 norm, used to calculate the Euclidean distance between the predicted displacement vector and the true displacement vector.

[0088] Multi-scale supervision and total loss function L Reg

[0089] The total registration loss L is defined as the weighted sum of losses across all scales:

[0090]

[0091] Where: k represents different feature scale levels. α is a hyperparameter used to balance the importance of feature photometric loss and endpoint error loss. γ is the weighting coefficient for different scale losses.

Claims

1. An infrared, visible image fusion feature enhancement method, characterized in that Comprising the following steps: Step one, image preprocessing, infrared image and visible light image are two different modal images obtained by using infrared camera and visible light camera in the same scene, taking the resolution of the light image as the benchmark, adjusting the resolution of the infrared image to be consistent with the pixel resolution of the visible light image, then taking a region (H, W) of the infrared image as the benchmark, cutting out a region (H, W) of the same size and consistent position in the visible light image; Step two, in the pre-processing network, the region (H, W) of the infrared image and the region (H, W) of the visible light image are respectively subjected to global structure texture detail extraction, and 8-channel infrared feature images and visible light feature images of the region (H, W) are extracted, so that 16-channel enhanced pre-processing feature map F is obtained 1 , Pre-processing feature map F 1 including 8 infrared feature images of the region (H, W) and 8 visible light feature images of the region (H, W) Step three, in the multi-scale shared feature extraction network, the pre-processed feature map F 1 is processed to obtain three layer convolution network weight same, can carry out shared reuse 32 channel's (H / 2, W / 2) feature map H 2 , 64 channel's (H / 4, W / 4) feature map F 3 , 128 channel's (H / 8, W / 8) feature map F 4 , wherein, Step four, F 1 , F 2 , F 3 , F 4 The feature map is input into a shift estimation module, and through hierarchical step-by-step calculation of local correlation and shift calibration, an infrared and visible light fusion image is obtained.

2. The method of claim 1, wherein the method further comprises: In step one, when taking the resolution of the light image as the benchmark to adjust the resolution of the infrared image to be consistent with the pixel resolution of the visible light image, a bilinear interpolation or nonlinear interpolation algorithm is used for the infrared image to improve the pixel resolution of the infrared image to be consistent with the pixel resolution of the visible light image.

3. The method of claim 1, wherein the method further comprises: In step two, the preprocessing network uses two independent feature extraction networks of 3x3 convolution layer 1 and 3x3 convolution layer to extract the global structure texture details of the visible light image and the infrared image respectively, each feature extraction network is composed of a convolution layer with a depth of 4 and a convolution kernel size of 3x3, and the channel number is set to 8.

4. The method of claim 1, wherein the method further comprises: In step three, the feature image sets are combined in the multiscale shared feature extraction network in the order of infrared feature image first and visible light feature image second, and the combined feature image sets are obtained through a 3*3 convolution layer in the multiscale shared feature extraction network to obtain a 32-channel (H / 2, W / 2) feature map F 2 The 32-channel (H / 2, W / 2) feature map F 2 The 64-channel (H / 4, W / 4) feature map F is obtained through a 3*3 convolution layer in the multiscale shared feature extraction network 3 The 64-channel (H / 4, W / 4) feature map F 3 The 128-channel (H / 8, W / 8) feature map F is obtained through a 3*3 convolution layer in the multiscale shared feature extraction network 4 The 3*3 convolution layer network in the multiscale shared feature extraction network has the same network weight and can be shared and reused, each layer of convolution has a depth of 4, the convolution kernel size is 3*3, the channel number is 32, 64 and 128 respectively, and the three-layer network structure parameters are shared.

5. The method of claim 1, wherein the method further comprises: In step three, the pre-processed feature map F 1 comprising infrared pre-processed feature maps and visible light pre-processed feature maps 32-channel (H / 2, W / 2) feature map F 2 comprising infrared feature maps and visible light feature maps 64-channel (H / 4, W / 4) feature map F 3 comprising infrared feature maps and visible light feature maps 128-channel (H / 8, W / 8) feature map F 4 comprising infrared feature maps and visible light feature maps In step four, the displacement estimation module processes the infrared feature map. and visible light feature map Processing to obtain the displacement matrix H 3 According to the displacement matrix H 3 infrared feature map Perform shift correction to generate a calibrated infrared feature map. In the displacement estimation module, the visible light feature map... and Processing to obtain the displacement matrix H 2 According to the displacement matrix H 2 infrared feature map Perform shift correction to generate a calibrated infrared feature map. In the displacement estimation module, the visible light feature map... and infrared feature map Processing to obtain the displacement matrix H 1 According to the displacement matrix H 1 Infrared preprocessing feature map Perform shift correction to generate calibrated infrared preprocessed feature maps. In the displacement estimation module, the visible light preprocessed feature map is processed. and infrared preprocessing feature map Processing to obtain the displacement matrix H 0 According to the displacement matrix H 0 The original infrared image is shift-corrected to generate a calibrated infrared feature map. Will The infrared and visible light fused image is generated by overlaying the original visible light image with it.

6. The method of claim 1, wherein the method further comprises: In step four, the displacement estimation module is a convolution layer with three layers, a depth of 4, a channel number of 64, and a convolution kernel size of 3x3, and the equivalent receptive field of convolution is: wherein R n represents the receptive field size of the n-th layer, s j represents the stride of the j-th layer, w is the size of the convolution kernel, l is a variable from 1 to n, R3 is 7 when three layers of convolution are used, and the output features are subjected to local correlation calculation: where C k is the correlation feature map of the k-th local block, F vis is the visible light feature, is the k-th local block of the infrared feature, c is the channel number, x, y represent the pixel position, and d is a variable of 1-c. Subsequently, the local correlation feature map is input into an upsampling layer to obtain a preliminary displacement matrix H: H = UP((L4...L1)(Concat(C k ,F ir ))) (4) where L i (X) is an expanded convolutional layer, X is the input feature map, W i ,b i are the convolution kernel and bias of the i-th layer, respectively, and σ represents a nonlinear activation function. F ir is the infrared feature map, and is the k-th local block of the infrared feature, and UP represents an up-sampling operation.

7. The method of claim 1, wherein the method further comprises: In step four, when shifting calibration, the original infrared feature is calibrated by the displacement matrix, first, a normalized regular grid G with the same spatial dimension (h, w) as the input feature map F is generated, each point in the grid G corresponds to the position (I', j') of the output feature map, and its coordinates are normalized to the range of [-1, 1]: The input displacement matrix H is then applied to the regular grid G to generate a deformed sampling grid G def For each point p in the grid G def its sampling coordinates on the source feature map are given by the following equation: G def (p ′ i,j )=p ′ i,j +H(p ′ i,j ) (6) Finally, the deformed sampling grid G def is used to resample the source feature map F src to obtain the final registered feature map F warped Since the sampling coordinates are usually non-integers, we use bilinear interpolation to compute the output values: wherein, denotes a set of four nearest-neighbor integer pixel coordinates around the sample point, q is a coordinate within the neighborhood, F src (q) is the value of the source feature map at q, k is a bilinear interpolation kernel function.

Images