Multi-modal image registration method based on disparity estimation

Abstract

The application discloses a kind of multi-modal image registration methods based on disparity estimation.The specific steps are as follows:(1) build array imaging system;(2) construct data set;(3) input image into neural network, adopt double branch strategy, extract common features and unique features;(4) use channel attention enhancement module for feature enhancement, construct matching cost based on disparity, and get disparity map through disparity regression;(5) use the obtained disparity map, align images through homography distortion, realize the registration and fusion of multi-modal images;(6) construct loss function, including mean absolute loss function and least square generative adversarial loss function;(7) input the image of test set into neural network, get style generation graph, predicted disparity map and aligned and fused multi-modal image.The method of the application can obtain disparity map from multi-modal images of different angles, and then obtain registered and fused multi-modal images.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of far-infrared waveband and deep learning, and extracts common features from images of different modalities and performs stereo matching to estimate disparity, thereby realizing alignment and fusion of cross-modality images. BACKGROUND

[0002] People are in a multi-modal environment, and artificial intelligence wants to better understand the environment and needs to have the ability to analyze multi-modal information. With the development of deep learning, many researchers try to extract features from multi-modal information through neural networks to obtain better data performance. The relationship and influence between various directions of multi-modal learning research will affect the final result. Currently, there are five research directions of multi-modal learning: representation, transformation, alignment, fusion, and collaborative learning. Multi-modal image alignment and fusion are widely used in medical, biological, military, and computer vision fields, so how to handle multi-modal heterogeneous data has important research significance and landing value.

[0003] In the 1980s, Marr of the Massachusetts Institute of Technology in the United States proposed a visual computing theory based on binocular matching, which can generate a stereoscopic image with depth by processing two planar images with parallax, thereby realizing three-dimensional reconstruction. Subsequently, many international studies have been carried out in the field of binocular vision and have achieved a series of results. Although domestic research started late, with the continuous development of computer technology, binocular stereo vision has played a huge application value in industry, agriculture, military and other fields. However, stereo vision is limited to the range of visible light images of binoculars, and cannot meet the task requirements in extreme environments such as weak light and overexposure.

[0004] Visible light imaging has rich contrast, color and shape information, which can accurately and quickly obtain matching features between binocular images, and then obtain the depth information of the scene. However, visible light images also have defects, such as in strong light, fog, rain, snow or at night, the imaging quality is greatly reduced, thereby affecting the accuracy and accuracy of matching. While the infrared band imaging has the advantage of not being affected by fog, rain, snow and light, which can make up for the shortcomings of visible light band imaging, so as to obtain more complete and accurate fusion information. Modern scholars have carried out research on the registration problem of infrared and visible light images of two different modalities, and have done a lot of effective work. However, due to the inconsistency of the field of view, spatial resolution and sensor position of infrared and visible light images, the difference between the two modalities is too large, and the translation, rotation and other problems between images are inevitable. The efficient and accurate registration of multi-modal images has great challenges. At present, the registration of multi-modal images mostly ignores the disparity difference of different pixels in the image, only realizes the local alignment of multi-modal images, and the accuracy and effect are poor, which still has a large gap from people's requirements and actual engineering needs. Therefore, researching more accurate, efficient and robust multi-modal registration method is the current research focus. SUMMARY

[0005] The application provides a multi-modal image registration method based on disparity estimation. Considering the large inter-model difference between infrared images and visible light images, the application designs a neural network with double branches to extract common features and unique features of different modal images, ensuring that the common features extracted from the two modal images correspond to each other. Then, the extracted common features are stereo matched to obtain the required disparity information. While ensuring the lightness of the model as much as possible, the disparity of the collected images of different modalities is estimated to realize the alignment and fusion of multi-modal images.

[0006] The technical scheme adopted by the application is as follows:

[0007] The multi-modal image registration method based on disparity estimation comprises the following steps:

[0008] Step 1: simultaneously collecting infrared images and visible light images of different wave bands, and performing distortion correction and stereo correction on the images to obtain multi-modal stereo image pairs of infrared-visible light;

[0009] Step 2: constructing a data set including a training set and a test set; the data set includes the multi-modal stereo image pairs obtained in step 1 and the multi-modal stereo image pairs obtained by using a generative adversarial network;

[0010] Step 3, inputting the multi-modal stereo image pair data in the training set into a neural network, the neural network comprising a feature extraction encoder, a style transfer decoder, a stereo matching subnetwork, and an alignment fusion module; for input images of two different modalities of infrared-visible light, the neural network adopts a double-branch strategy, respectively extracting common features and unique features of different modalities of images through the feature extraction encoder, and then cross-feeding different features of different modalities into the style transfer decoder to complete the style transfer of the images;

[0011] Step 4, feeding the common features of different modalities of images obtained in step 3 into the stereo matching subnetwork, the stereo matching subnetwork comprising a channel attention enhancement module, a matching cost construction module, a cost aggregation module, and a disparity regression module; the channel attention enhancement module first performs feature enhancement on the common features of different modalities of images, then the matching cost construction module constructs a disparity-based matching cost according to the enhanced features, the matching cost is regularized through the cost aggregation module, and finally the output disparity map is obtained through the disparity regression module;

[0012] Step 5, using the disparity map obtained in step 4, aligning different modalities and different views of images to the same view through the alignment fusion module by using the homography distortion method to realize the registration and fusion of multi-modal images;

[0013] Step 6, constructing a loss function of the neural network, the loss function comprising a mean absolute loss function and a least squares generative adversarial loss function; training the neural network;

[0014] Step 7, in the test phase, inputting the multi-modal stereo image pair data in the test set into the trained neural network to obtain the registered and fused multi-modal images.

[0015] Further, in step 2, the specific steps of obtaining the multi-modal stereo image pair by using the generative adversarial network are as follows: first, using the generative adversarial network to pre-train the network model on the existing multi-modal paired data set to realize the style conversion of visible light images to infrared images; then, using the trained network model to perform style conversion on the existing stereo data set of visible light images to generate the required multi-modal stereo image pair.

[0016] Further, in step 3, the feature extraction encoder is divided into a common feature extraction encoder and a unique feature extraction encoder, wherein the common feature extraction encoder uses a basic residual block to extract reliable features, and after down-sampling, the common features of the input images are obtained; the unique feature extraction encoder uses a large convolution kernel and a max-pooling operation for down-sampling in a shallow network, and uses a convolution with a step of 2 for down-sampling in a deep network to obtain features of different scales, and extracts style features of the input images.

[0017] Further, in step 3, before the image is input into the unique feature extraction encoder for style feature extraction, a blur operation is used to destroy the structural information of the image to obtain abstract style features, avoiding the interference of false structural information.

[0018] Further, the blur operation is: set a fixed window size of random block to slide on the image, and after each sliding, randomly shuffle the order of all pixels in the random block, and traverse the entire image; then, mean filtering is performed on the shuffled image to remove noise points, and a structural blurred image is obtained.

[0019] Further, in step 3, the style transfer decoder receives the common features and different scale deep unique features from the feature extraction encoder, and uses deconvolution for upsampling operation; the style transfer decoder uses different scale unique features in the deep network, uses a U-Net type network structure to continuously perform feature connection and fusion, and connects the common features in the shallow network, and through the operation of upsampling-feature connection-feature fusion, obtains the generated image after style transfer.

[0020] Further, in step 3, the style transfer decoder performs two decodings in the decoding process, the first is intra-domain decoding, which uses common features and unique features of the same modality to realize intra-modal image reconstruction; the second is cross-domain decoding, which uses common features and unique features of different modalities to complete cross-modal cross-style conversion, ensuring that different features have their corresponding modal properties, while realizing image style transfer.

[0021] Further, in step 3, in step 4, the channel attention enhancement module enhances the common features of the input images of different modalities, gives high weight to the channels corresponding to the close common features, and gives low weight to the channels with large differences, to improve the matching degree of the common features of different modalities.

[0022] Further, the specific step of training the neural network in step 6 is: step 61, without using the stereo matching sub-network, only using the feature encoder and the style transfer decoder to complete the image style conversion task, training the neural network and making it converge, so that the different modal common features with higher matching degree can be obtained; step 62, training the stereo matching sub-network, at this time, the other parts of the neural network are frozen, and the network parameters are fixed and do not change; using the common features of different modalities extracted by the feature extraction encoder, stereo matching is performed, and the scene disparity map is obtained by disparity regression; step 63, the whole neural network is trained together, the mean absolute loss function and the least square generative adversarial loss function are used for supervision, the style transfer part and the stereo matching part in the network model are trained together, the network is further converged, more accurate disparity estimation results are achieved, and better multi-modal image alignment and fusion effect is realized.

[0023] The application uses a cross-modal network structure to process images of different modalities, extracts the required common features, and realizes the alignment and fusion of multi-modal images under the premise of ensuring accuracy. Compared with the prior art, the application has the following advantages:

[0024] (1) The application relates to the processing of modal differences between far-infrared band images and visible light images, and can obtain corresponding accurate disparity maps from different modal images of different angles, and then realize the alignment and fusion of cross-modal images.

[0025] (2) When extracting features of different modal images, a double-branch strategy is proposed to extract common features and unique features, and cross-modal different features are crossed to generate style images, complete image style transfer, and realize modal conversion of input images.

[0026] (3) The channel attention enhancement module proposed in the application enhances the channels of common features, so that the common features of different modalities are more corresponding and close, which improves the accuracy of stereo matching and disparity estimation, and finally enhances the multi-modal image registration and fusion effect.

[0027] (4) The multi-modal fusion image generated by the method of the application conforms to human visual perception and has important applications in video monitoring, target tracking and other directions. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a flowchart of the method of the application;

[0029] Figure 2 is a whole network structure diagram of the method of the application;

[0030] Figure 3 is a structure diagram of the common feature extraction encoder in the embodiment of the application;

[0031] Figure 4 is the structural diagram of the unique feature extraction encoder in the embodiment of the present application;

[0032] Figure 5 is the structural diagram of the style transfer decoder in the embodiment of the present application;

[0033] Figure 6 is the structural diagram of the channel attention enhancement module of the stereo matching subnetwork in the embodiment of the present application. DETAILED DESCRIPTION

[0034] The embodiments of the present application will be described in detail below, which are implemented on the premise of the technical solution of the present application, and detailed implementation modes and specific operation processes are given, but the protection scope of the present application is not limited to the following implementations.

[0035] The embodiment provides a multi-modal image registration method based on disparity estimation, as shown in Figure 1 , comprising the following steps:

[0036] (1) Build a multi-camera array imaging system, use different modal cameras to synchronously capture a scene, obtain multi-modal stereo image pair data of different waveband infrared images and visible light images, and correct distortion and stereo.

[0037] (2) Construct a data set, including a training set and a test set, the data set includes multi-modal stereo image pair data obtained by using a generative adversarial network (GAN), and image pair data collected by the multi-camera array imaging system, and the multi-modal stereo image pair data contains the required disparity information.

[0038] (3) Input the multi-modal stereo image pair data in the training set into a neural network model, adopt a double-branch strategy, extract the corresponding common features and unique features of different modal images through a feature extraction encoder, and cross feed different features of different modalities into a style transfer decoder to realize style transfer of the images.

[0039] (4) The common features of different modal images obtained are fed into a stereo matching subnetwork, the features are enhanced through a channel attention enhancement module, the matching cost based on disparity is constructed according to the enhanced features, the matching cost is regularized through a cost aggregation module, and finally the output disparity map is obtained through a disparity regression module.

[0040] (5) In the alignment and fusion module, the output disparity map obtained is used to align different modal images of different perspectives to the same perspective through homographic distortion, so as to realize registration and fusion of multi-modal images.

[0041] (6) Construct a loss function of the neural network model, the loss function includes an average absolute loss function (L1 Loss) and a least square generative adversarial loss function (LSGAN Loss); wherein, the average absolute loss function is constructed to reduce the absolute error of the generated style image and the final output disparity map, and the least square generative adversarial loss function is constructed to make the generated style image have higher quality and better effect; then the neural network is trained.

[0042] (7) In the test phase, the multi-modal images in the test set are input into the trained neural network to obtain the generated image after style transfer, the predicted disparity map, and the multi-modal image after registration and fusion.

[0043] 1. The part of the multi-camera array imaging system involved in step (1) is described in detail:

[0044] An imaging system is built in this embodiment, a multi-camera array is composed of one visible light camera and two infrared cameras, and the simultaneous acquisition of different waveband infrared images and visible light images is completed. Among them, the visible light camera is located at the midpoint of the two infrared cameras, the three cameras are placed horizontally, and the same shooting scene is synchronously acquired. Then the intrinsic and extrinsic parameters of the camera are obtained through the camera calibration method, the distortion correction is performed, and the stereo epipolar line correction is performed on the different modal images with different angles on the left and right, to obtain standard multi-modal stereo image data.

[0045] 2. The data set construction part involved in step (2) is described in detail:

[0046] This embodiment uses generative adversarial neural network GANs for style transfer to generate multi-modal stereo image pair data, solving the problem of lack of binocular multi-modal data set. This method first uses the generative adversarial network to pre-train the network model on the existing infrared-visible light multi-modal paired data set, realizes the style conversion of visible light image to infrared image, and then runs the trained network model on the existing visible light image stereo data set, performs style conversion, generates infrared images with the same angle, and obtains the required infrared-visible light multi-modal stereo image pair.

[0047] 3. The overall network structure of this embodiment is shown in Figure 2 The feature extraction encoder, style transfer decoder, stereo matching subnetwork, and alignment fusion module parts involved in steps (3), (4), and (5) are described in detail below, and these network modules constitute the entire neural network model:

[0048] 3.1 Feature extraction encoder, divided into common feature extraction encoder and unique feature extraction encoder, respectively used for extracting common features and unique features of different modal images, and the network model structure diagrams are respectively shown in Figure 3 andFigure 4 .

[0049] Table 1 Specific structure of common feature extraction (left) and unique feature extraction (right) encoder

[0050]

[0051] The common feature extraction encoder uses basic residual blocks to extract reliable features, and sets the step size to 2 to perform two times of down-sampling, and finally outputs the effective feature representations out2 and out3 (see Table 1) of the same resolution as the common features of the input image; the unique feature extraction encoder uses a large convolution kernel with a step size of 2 to perform convolution down-sampling at the shallow network, and uses a maximum pooling operation to weaken the structural information, and uses a basic residual block with a step size of 2 to perform down-sampling at the deep network, a total of six times of down-sampling to obtain features of different scales, and finally outputs four different scales of deep feature representations out4, out5, out6 and out7 (see Table 1) as the style features extracted from the input image.

[0052] In addition, for the unique feature extraction encoder, a blur operation is first used on the input image to destroy the structural information of the image, and then the image is input into the encoder for style feature extraction. This embodiment uses a random block operation, sets a fixed window size block patch to slide on the image, and randomly rearranges the order of all pixels in the block after each sliding, and traverses the entire image. Then, the image after the random block rearrangement is subjected to mean filtering to remove noise and obtain a structural blur image. Using such a blur operation to destroy the structural information of the image is more conducive to obtaining abstract style features, avoids the interference of false structural information, and improves the matching accuracy.

[0053] 3.2 Style transfer decoder, as shown in Figure 5 , uses the common features and unique features from the feature extraction encoder to generate the modality conversion image after style transfer.

[0054] Table 2 Specific structure of style transfer decoder

[0055]

[0056] The decoder receives common features (cont_out2 and cont_out3 in Table 2) and four unique features at different scales (sty_out4, sty_out5, sty_out6, and sty_out7 in Table 2) from the feature extraction encoder. It performs convolution operations using basic residual blocks to obtain reliable feature representations and then uses deconvolution operations to upsample the feature maps. The network layers progress from deep to shallow, receiving multimodal features at different scales. Combined with convolution operations, it performs feature connections and fusion. Through continuous upsampling-feature connection-feature fusion operations, style transfer of different modal images is achieved.

[0057] 3.3 The stereo matching subnetwork includes a channel attention enhancement module, a matching cost construction module, a cost aggregation module, and a disparity regression module.

[0058] 3.3.1 The channel attention enhancement module can enhance the common features of different modalities, improving the matching degree of common features across different modalities. For example... Figure 6 As shown, common features from different modalities derived from the common feature extraction encoder are input into this module. Feature concatenation is performed at the entry point, followed by a series of convolutional operations to obtain an intermediate feature map of the same size as a single input feature. Finally, a 1×1 convolution and sigmoid function activation are applied to obtain the output attention weight feature. This weight feature is multiplied by the two input common features, and then added to the corresponding original features, ultimately outputting the enhanced common features from different modalities. The channel attention enhancement mechanism used in this module assigns high weights to channels that are close in the common features, while assigning low weights to channels that are significantly different. This effectively increases the matching degree of common features from different modalities, improving the accuracy of the final disparity estimation and alignment fusion.

[0059] 3.3.2 The matching cost construction module utilizes feature maps extracted from different viewpoints, which, after channel attention enhancement, construct a matching cost for subsequent disparity solving. This method uses two input images from different viewpoints and modalities, obtains effective shared feature representations through a shared feature extraction encoder, enhances these features through a channel attention enhancement module, and then performs feature translation and concatenation across different disparities to construct a 4D matching cost volume.

[0060] C(c,d,h,w)=Cat <F r (c,h,w),f shift [F t (c,h,w),d]>

[0061] Where f shift [·,·] indicates that the target feature F is given a disparity d. t Move the warp to the reference feature Fr , Cat<.,.> represents concatenating the warped target feature and the corresponding reference feature in the disparity dimension, (h, w) and c represent the spatial and channel dimensions of the extracted feature, respectively. This method can obtain a 4D matching cost C(c, d, h, w).

[0062] 3.3.3 Cost aggregation module, used for information aggregation and regularization of the constructed matching cost volume. Because the constructed matching cost is 4D dimension, 3D convolution is needed for filtering processing to achieve excellent regularization effect. In order to utilize more context information, the embodiment adopts a stacked hourglass structure based on 3D-CNN architecture, uses multiple intermediate layer supervision, from fine to coarse and then from coarse to fine, to aggregate and regularize the cost volume.

[0063] 3.3.4 Disparity regression module, used for disparity regression of the aggregated and regularized matching cost, and outputting the estimated scene disparity map. This method adopts a soft argmin operation to perform disparity regression to estimate a continuous and accurate disparity map. This operation is to perform a softmax operation σ(.) on the matching cost C d under the disparity d to calculate a probability voxel, and to perform disparity weighted summation according to the disparity range (-D max to D max ) to obtain the final estimated disparity

[0064]

[0065] 3.4 Alignment and fusion module, which utilizes the already obtained disparity map to realize the alignment and fusion of different view and different modality images. The embodiment adopts a homography warping operation. For each pixel of the target image, the homography warping operation is used to move and warp the pixel to the reference image according to the obtained disparity value. After performing this operation on all pixels, the image of the target view is moved to the reference view, and the conversion between different view images is completed. This method converts different view and different modality images to the same view to realize the alignment of the images, and then uses a fusion operation to set different weights of the two modality images to realize the fusion of the multi-modality images.

[0066] 4. The double-branch strategy part of the network model involved in step (3) is described in detail.

[0067] In order to ensure the complementarity of the common features and the unique features extracted by the encoder and improve the accuracy of feature matching, the network model of the method adopts a double-branch strategy. As shown in Figure 2As shown, for two different modal input images, the network uses a 1-way shared common feature encoder to extract common features of different modalities, and uses 2-way unique feature encoders and 2-way style transfer decoders, corresponding to the encoding and decoding of the two different modalities, respectively.

[0068] In the decoding process, two decodings are performed, the first is intra-domain decoding, using common features and unique features of the same modality to realize intra-modal original image reconstruction; the second is cross-domain decoding, using common features and unique features of different modalities to complete cross-modal cross-style conversion, ensuring that different features have their corresponding modal properties, while realizing image style transfer.

[0069] 5. The training neural network part involved in step (6) is described in detail:

[0070] In order to better extract common features of different modal images, complete feature matching and obtain accurate disparity maps, a three-step training strategy is proposed in this method.

[0071] First, without using a stereo matching subnetwork, only using feature encoders and decoders to complete the image style conversion task, the neural network model is trained and converged, so that different modal common features with high matching degree can be obtained. In this step, the average absolute loss function and the least squares generative adversarial loss function are used to supervise the neural network, reduce the error of generated style images, and improve the quality of style images.

[0072] Second, the stereo matching subnetwork is trained, at this time the other parts of the neural network are frozen and the network parameters are fixed and do not change. Using the different modal common features obtained by the common feature extraction encoder, stereo matching is performed, and the scene disparity map is obtained by disparity regression. Here, the average absolute loss function is used to supervise the output disparity map, so that the neural network can accurately estimate the scene disparity map.

[0073] The third step is to train the entire neural network model jointly. The loss functions used in the first two steps are used for supervision, the style transfer part and the stereo matching part of the network model are jointly trained, the network is further converged, more accurate disparity estimation results are achieved, and better multi-modal image alignment and fusion effects are achieved.

[0074] 6. The neural network model test part involved in step (7) is described in detail:

[0075] In the testing phase, first, the neural network model is loaded with the trained network parameters, then the test multi-modal image is input into the neural network, the common features of different modalities are obtained through the feature encoding and decoding described in step (3), and then input into the stereo matching sub-network in step (4) to obtain the accurate scene disparity map, and finally through the alignment and fusion module described in step (5), the registered and fused multi-modal image is obtained.

Claims

1. A method for multi-modal image registration based on disparity estimation, characterized in that, The method comprises the following steps: Step 1: simultaneously collecting infrared images and visible light images of different wave bands, and performing distortion correction and stereoscopic correction on the images to obtain infrared-visible light multi-modal stereoscopic image pairs; Step 2: constructing a data set, including a training set and a test set; the data set includes the multi-modal stereoscopic image pairs obtained in step 1 and the multi-modal stereoscopic image pairs obtained by using a generative adversarial network; Step 3: inputting the multi-modal stereoscopic image pair data in the training set into a neural network, wherein the neural network comprises a feature extraction encoder, a style transfer decoder, a stereoscopic matching subnetwork and an alignment fusion module; for the input images of two different modalities of infrared-visible light, the neural network adopts a double-branch strategy, extracts corresponding common features and unique features of different modalities of images through the feature extraction encoder, then cross feeds the cross-modal different features into the style transfer decoder to complete the style transfer of the images; the style transfer decoder receives the common features and deep unique features of different scales from the feature extraction encoder, and performs upsampling operation using deconvolution; the style transfer decoder uses unique features of different scales in the deep network to continuously perform feature connection and fusion in a U-Net type network structure, and connects the common features in the shallow network, and through continuous upsampling-feature connection-feature fusion operation, obtains the generated images after style transfer; the style transfer decoder decodes twice in the decoding process, the first is intra-domain decoding, which uses common features and unique features of the same modality to realize the reconstruction of the original image of the same modality; the second is cross-domain decoding, which uses common features and unique features of different modalities to complete cross-modal cross style conversion, so that different features have their corresponding modal properties, and the style transfer of the images is realized; Step 4: feeding the common features of different modalities of images obtained in step 3 into the stereoscopic matching subnetwork, which comprises a channel attention enhancement module, a matching cost construction module, a cost aggregation module and a disparity regression module; the channel attention enhancement module first enhances the common features of different modalities of images, then the matching cost construction module constructs the matching cost based on the disparity according to the enhanced features, then the cost aggregation module performs regularization on the matching cost, and finally the disparity regression module obtains the output disparity map; Step 5: using the disparity map obtained in step 4, aligning different modalities and different view images to the same view through the alignment fusion module by using the homography distortion method to realize the registration and fusion of multi-modal images; Step 6: constructing a loss function of the neural network, which comprises a mean absolute loss function and a least squares generative adversarial loss function; training the neural network; Step 7: in the test stage, inputting the multi-modal stereoscopic image pair data in the test set into the trained neural network to obtain the registered and fused multi-modal images.

2. The method of claim 1, wherein, In step 2, the specific steps of obtaining the multi-modal stereo image pair by using the generative adversarial network are as follows: first, the generative adversarial network is used to pre-train the network model on the existing multi-modal paired data set, so as to realize the style conversion from the visible light image to the infrared image; then, the trained network model is used for style conversion on the existing stereo data set of the visible light image, so as to generate the required multi-modal stereo image pair.

3. The method of claim 1, wherein, In step 3, the feature extraction encoder is divided into a common feature extraction encoder and a unique feature extraction encoder, wherein the common feature extraction encoder uses a basic residual block to extract reliable features, and after down-sampling, the common features of the input image are obtained; the unique feature extraction encoder uses a large convolution kernel and a maximum pooling operation for down-sampling at a shallow network, and uses a convolution with a step of 2 for down-sampling at a deep network, so as to obtain features of different scales and extract style features of the input image.

4. The method of multi-modal image registration based on disparity estimation according to claim 3, characterized in that, In step 3, before the image is input to the unique feature extraction encoder for style feature extraction, a blur operation is used to destroy the structural information of the image, so as to obtain abstract style features and avoid the interference of false structural information.

5. The method of multi-modal image registration based on disparity estimation according to claim 4, characterized in that, The blur operation is as follows: a random block with a fixed window size is set to slide on the image, and after each sliding, the order of all pixels in the random block is randomly disturbed, and the whole image is traversed; then, the mean filter is used on the image after the random block is disturbed, so as to remove the noise and obtain a structural blur image.

6. The method of claim 1, wherein, In step 3 and step 4, the channel attention enhancement module enhances the common features of the input images of different modalities, gives high weight to the channels corresponding to the close features in the common features, and gives low weight to the channels with large differences, so as to improve the matching degree of the common features of different modalities.

7. The method of claim 1, wherein, In step 6, the specific steps of training the neural network are as follows: Step 61, without using the stereo matching sub-network, only using the feature encoder and the style transfer decoder to complete the image style conversion task, training the neural network and making it converge, so that the common features of different modalities with high matching degree can be obtained; Step 62, the stereo matching sub-network is trained, at this time, the other parts of the neural network are frozen, and the network parameters are fixed and unchanged; the common features of different modalities extracted by the feature extraction encoder are used for stereo matching, and the scene disparity map is obtained by disparity regression; Step 63, the whole neural network is trained together, the mean absolute loss function and the least square generative adversarial loss function are used for supervision, the style transfer part and the stereo matching part in the network model are trained together, the network is further converged, more accurate disparity estimation results are achieved, and better multi-modal image alignment and fusion effect is realized.

Images