A cross-modal image optical flow prediction method based on RAFT network

Abstract

The present invention discloses a cross-modal image optical flow prediction method based on a RAFT network, which includes the following steps: S1: designing and constructing a credibility RAFT network; S2: training to obtain a cross-modal credibility RAFT network; S3: preprocessing the input image; S4: obtaining the predicted optical flow and credibility logarithmic map: sending two pre-processed input images into the cross-modal credibility RAFT network to obtain the predicted optical flow and credibility logarithmic map; S5: post-processing the optical flow and credibility logarithmic map to obtain a prediction of the credibility of each area of ​​the optical flow. This method can more accurately predict the optical flow between cross-modal images and has good generalization. The method of the present invention is conducive to promoting the development of the field of cross-modal image registration and optical flow prediction, reducing the burden of training set annotation, and has high practical application value.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a cross-modal image optical flow prediction method based on a RAFT network. Background Art

[0002] Multispectral image fusion technology plays a vital role in the field of vision. Images of different modalities have distinct properties. For example, visible light and near-infrared images best align with human perception and general human cognition. Near-infrared light is invisible to the human eye, so near-infrared light can be used to supplement the illumination in low-light environments without affecting the visual experience of the actual environment. To fully combine the advantages of both modalities, visible light and near-infrared images captured from the same scene are often fused. However, since it's impossible to ensure that the near-infrared camera and visible light camera are in exactly the same position and angle when capturing the images, the resulting visible and near-infrared images are often misaligned, hindering subsequent fusion or other operations. Therefore, these two cross-modal images must first be registered.

[0003] The paper "RAFT: Recurrent All-Pairs Field Transforms for Optical Flow," published at the European Conference on Computer Vision 2020 (402-419), proposes a recurrent iterative optical flow prediction network. This network extracts features from the input image, constructs a correlation matrix, and iteratively queries the correlation matrix to update the optical flow prediction, ultimately obtaining a highly accurate and detailed optical flow estimation result. While the paper performs well on a training set of visible light images labeled with true optical flow values, it is difficult to generalize to cross-modal image registration tasks where true optical flow values ​​are lacking. Furthermore, due to the lack of a reliable prediction of the optical flow, areas that the network finds difficult to estimate accurately are retained, introducing artifacts and human-like traces.

[0004] Invention patent CN115393404A discloses a method, apparatus, device, and storage medium for dual-light image registration. The method first converts an acquired infrared image into a visible light image through a modal conversion network. Feature detection and matching are then performed on the converted visible light image and the input visible light image, and a homography matrix is ​​calculated to register the two images. Finally, the input infrared image is transformed based on the homography matrix to obtain the registered infrared image and visible light image. However, because the infrared image and visible light image themselves come from signals in different wavelength bands, the visible light image converted from the infrared image cannot be guaranteed to be sufficiently accurate. The transformation capability of the homography matrix is ​​limited, and its effect is generally poor in scenes that are not on the same plane. Summary of the Invention

[0005] The purpose of the present invention is to address the problems existing in the prior art and design a cross-modal image optical flow prediction method based on the RAFT network. This method can more accurately predict the optical flow between cross-modal images, has good generalization, and has high practical value.

[0006] A cross-modal image optical flow prediction method based on RAFT network, the method comprising the following steps:

[0007] S1: Design and construct a credibility RAFT network: Add a credibility estimation module to the cyclic iterative structure of the RAFT network to estimate the credibility of each area of ​​the output optical flow;

[0008] S2: Training a cross-modal image optical flow prediction network: The credibility RAFT network is trained using cross-modal data augmentation to obtain a cross-modal credibility RAFT network.

[0009] S3: Input image preprocessing: Perform image preprocessing on the two input images to make them conform to the input format of the credibility RAFT network;

[0010] S4: Obtain predicted optical flow and credibility logarithmic map: The two pre-processed input images are fed into the cross-modal credibility RAFT network to obtain the predicted optical flow and credibility logarithmic map;

[0011] S5: Post-process the optical flow and credibility logarithmic map to obtain a prediction of the credibility of each area of ​​the optical flow.

[0012] The specific decomposition is described as follows:

[0013] In S1, the credibility RAFT network is designed and constructed: In the RAFT network, there is a GRU (Gated Recurrent Unit) that reads the features transformed by the current optical flow. Figure 2 and features Figure 1 The correlation between the two and the content encoding information of the input image 1 are used to update the internal latent variables and output a new optical flow estimate.

[0014] The credibility prediction module is connected after the GRU module of the RAFT network and participates in multiple loop iterations of the RAFT network. In each iteration, it outputs the credibility of the current estimated optical flow. The credibility module also reads in the latent variables, features, Figure 1 ,feature Figure 2 and optical flow, the features Figure 2 Obtain the transformed features after optical flow transformation Figure 2 , with the characteristics Figure 1The correlation is calculated, and after the features are initially integrated using 3×3 convolution, they are connected with the latent variables in the channel dimension, and the credibility logarithmic map is output through two convolutional layers.

[0015] In S2, the credibility RAFT network is trained using cross-modal data enhancement, which means: on the existing visible light image training set annotated with the true value of optical flow, data enhancement is performed through custom mapping functions such as brightness remapping, color channel exchange, Gaussian blur, and image sharpening to weaken the brightness relationship between the image and the visible light image, simulate images of multiple different modalities, and train the credibility RAFT network's cross-modal image registration capabilities.

[0016] The training process uses a custom loss function. For each loop iteration, the loss between the estimated credibility, the predicted optical flow, and the true optical flow value is calculated, and the final loss is weighted to calculate the gradient and backpropagation.

[0017] The custom loss function is:

[0018]

[0019]

[0020] Among them, β, λ, γ, θ, and α are all hyperparameters. i represents the credibility graph obtained after the network's i-th cycle iteration, f gt represents the true value of the optical flow of the training set, f i represents the optical flow predicted by the network at the i-th loop iteration, L i Represents the loss Loss calculated from the output value of the network's i-th loop iteration, L total represents the total loss, which is obtained by weighting the losses of all loop iterations, and η is the weighting coefficient.

[0021] Preferably, the custom mapping function brightness remapping uses the following two mapping functions:

[0022]

[0023] Where 0.5≤a≤1,0 <b≤1;

[0024]

[0025] where 0 <a<1,0.5≤b≤1,d≥1。

[0026] In the above two formulas, I is the original brightness of the image, I′ is the brightness of the image after brightness remapping, and a, b, and d are constants generated by the random number generator each time the image is enhanced.

[0027] The color channel swapping is to scramble all channels of the multi-channel input image and reorganize them into a new multi-channel image in a random order.

[0028] Adding Gaussian blur is to use the Gaussian blur kernel G to convolve the input image:

[0029]

[0030] Where I represents the input image, I' represents the image after Gaussian blur data enhancement, Represents a convolution operation.

[0031] The image sharpening is achieved by the Laplacian operator in the following manner:

[0032] I′=cΔI+kI

[0033] Where c and k are constants. c is used to adjust the sharpening intensity, and k is used to prevent the brightness from exceeding the brightness range after addition. ΔI represents the result of convolving the image I with the following Laplacian mask:

[0034]

[0035] In S3, image preprocessing: If the input image is a single-channel image, it is replicated three times to form a three-channel image. If the length or width of the input image is not a multiple of 8, it is padded with 0 pixels until the length and width are divisible by 8.

[0036] In S4, the two preprocessed images are simultaneously fed into the cross-modal credibility RAFT network to obtain a set of predicted optical flows and a set of predicted credibility logarithmic maps. The output results of the last iterative cycle are selected as the predicted optical flows and credibility logarithmic maps respectively.

[0037] In S5, the post-processing includes:

[0038] 5.1 Take the exponent of the credibility logarithmic graph to obtain the final credibility graph;

[0039] 5.2 Based on the predicted optical flow, the input image I2 is transformed to obtain the registered input image, and the pixels filled in S1 are removed to obtain the final registration result.

[0040] The credibility RAFT (Recurrent All-Pairs Field Transforms) network described in the method of the present invention includes a credibility estimation module. During the cyclic iteration of the credibility RAFT network, each iteration can output the predicted optical flow and the credibility map of the optical flow. The credibility map is a single-channel image with a length and width consistent with the optical flow. Its pixel value characterizes the credibility of the optical flow at that position, which greatly improves the availability of the optical flow and helps to eliminate artifacts and artificial traces caused by registration errors in difficult-to-register areas to subsequent fusion operations. The network is trained on the existing visible light optical flow dataset through a new cross-modal data enhancement method, and can more accurately predict the optical flow between cross-modal images, has good generalization, and has high practical value.

[0041] This paper uses cross-modal data augmentation methods such as brightness remapping to simulate multiple modal images using visible light images. This expands the network's training data to multiple modalities, reduces the network's reliance on image brightness relationships when predicting optical flow, improves image generalization, and enables the RAFT network to handle cross-modal image registration scenarios such as visible light and near-infrared, where true optical flow values ​​are difficult to obtain, thus enhancing its practicality. A credibility prediction module is added that, along with outputting optical flow, outputs the network's reliability predictions for each region of the optical flow. This allows the selective use of reliable optical flow regions in subsequent tasks, reducing the negative impact of registration errors on subsequent tasks.

[0042] The method of the present invention is conducive to promoting the development of the fields of cross-modal image registration and optical flow prediction, reducing the burden of training set annotation, and has high practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a cross-modal image optical flow prediction method based on the RAFT network implemented by the present invention.

[0044] Figure 2 This is a flowchart of the operation of the credibility RAFT network according to an embodiment of the present invention.

[0045] Figure 3 This is a diagram of the credibility RAFT network structure of an embodiment of the present invention.

[0046] Figure 4 This is a flow chart of the credibility RAFT network iteration module according to an embodiment of the present invention.

[0047] Figure 5 2 is a structural diagram of a credibility estimation module according to an embodiment of the present invention.

[0048] Figure 6 This is the result of the embodiment of the present invention on the simulation data set, where: Figure 6 (a) is the input image 1, Figure 6 (b) is the input image 2, Figure 6 (c) is the credibility prediction graph, Figure 6 (d) is the optical flow estimation result, Figure 6 (e) is the true value of optical flow.

[0049] Figure 7 This is a display of the results of the embodiment of the present invention in a real visible light-near infrared image scene, where: Figure 7 (a) is the input near-infrared image, Figure 7 (b) is the input visible light image, Figure 7 (c) is the near-infrared image after registration. Figure 7 (d) is the credibility prediction result, Figure 7 (e) is the optical flow estimation result, Figure 7 (f) Comparison of the near-infrared image registration results with the visible light image. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, the present invention discloses a cross-modal image optical flow prediction method based on RAFT network, comprising the following steps:

[0052] S1: Design and construct a credibility RAFT network: Add a credibility estimation module to the cyclic iterative structure of the RAFT network to estimate the credibility of each area of ​​the output optical flow;

[0053] S2: Training a cross-modal image optical flow prediction network: The credibility RAFT network is trained using cross-modal data augmentation to obtain a cross-modal credibility RAFT network.

[0054] S3: Input image preprocessing: Perform image preprocessing on the two input images to make them conform to the input format of the credibility RAFT network;

[0055] S4: Obtain predicted optical flow and credibility logarithmic map: The two pre-processed input images are fed into the cross-modal credibility RAFT network to obtain the predicted optical flow and credibility logarithmic map;

[0056] S5: Post-process the optical flow and credibility logarithmic map to obtain a prediction of the credibility of each area of ​​the optical flow.

[0057] Figure 2 This is a flowchart of the cross-modal credibility RAFT network. The model takes in two cross-modal images, Input Image 1 and Input Image 2, and outputs a predicted optical flow through the RAFT network structure trained using cross-modal data augmentation. The credibility prediction module then outputs an estimate of the credibility.

[0058] like Figure 3 As shown, the credibility RAFT reads two cross-modal images and passes them through the weight-sharing encoder to obtain features Figure 1 and features Figure 2 , construct the correlation matrix, and repeatedly pass through the loop iteration module n times, and output the optical flow predicted by this iteration and the logarithmic map of the optical flow credibility in each loop iteration.

[0059] Figure 3 This is the internal structure of the iterative module. It reads the optical flow output from the previous loop and the semantic feature map of the content encoding features of input image 1, updates the latent variables, and outputs the predicted optical flow for this loop iteration. The credibility prediction module is connected after the GRU module and reads the updated latent variables and predicted optical flow from this loop iteration.

[0060] like Figure 4 The credibility prediction module uses the latent variable feature map of the GRU module and the feature map of the input image 1 after passing through the encoder. Figure 1 And the features of the input image after the encoder Figure 2 , the optical flow predicted by this iteration is input, and the specific steps are:

[0061] 1.1 Feature Map Transformation. In order to obtain the credibility of the optical flow obtained in this iteration, it is necessary to calculate the correlation between the transformed input image 2 and the original input image 1 under the current estimated optical flow. To this end, the correlation is calculated on the high-dimensional features. In the RAFT network, the two input images are passed through a weight-sharing encoder to obtain the extracted high-dimensional feature map, and the high-dimensional feature features of the input image 2 are extracted using the optical flow. Figure 2 Transformation, obtain the transformed features Figure 2 , the features Figure 1 and the transformed features Figure 2 Dot product to obtain the correlation graph. The formula can be expressed as:

[0062] corr(x,y,c)=f1(x,y,c)*f2(x+flow(x,y,1),y+flow(x,y,2),c)

[0063] Where corr represents the correlation graph and f1 represents the feature Figure 1 , f2 represents the feature Figure 2 , flow represents the optical flow predicted by this loop iteration, (x, y) represents the pixel position, and c represents the feature map channel.

[0064] The correlation graph is passed through a 3×3 convolutional layer for feature integration.

[0065] 1.2 Feature joint decoding. In the RAFT network, the state information of the current iteration is recorded through latent variables. In each iteration, the state of the latent variable is changed through certain inputs, and the optical flow estimated in this iteration is decoded based on the current state. Therefore, the latent variable records information related to the current optical flow estimation. The latent variable has the same length and width as the correlation graph after feature integration described in 1.1. These two feature maps are connected in the channel dimension to form a new feature map, which passes through a decoding layer consisting of a 3×3 convolutional layer and a nonlinear activation function ReLU;

[0066] 1.3 Credibility Regression. The output of 1.2 is finally subjected to a 1×1 pixel-wise convolution to obtain a credibility logarithmic map with a channel number of 1. At this point, the size of the credibility logarithmic map is 1 / 8 the size of the input image. Using bilinear interpolation, the credibility logarithmic map is interpolated to the original image size.

[0067] Data enhancement is used to simulate cross-modal images, improve network generalization, and apply a custom loss function to train the network. The idea of ​​the present invention comes from the following observations: images of different modalities often have similar structures and the same object edges, but the gradients of the object edges may be in the same direction or in opposite directions, and the gradient strength of the object edges may also be different; the brightness of the same object in different modal images is also different. At the same time, due to the lack of cross-modal image optical flow datasets in the field of cross-modal image registration, it is difficult to directly train deep learning models through cross-modal image optical flow training data. To this end, targeted data enhancement can be applied to the existing visible light optical flow dataset to simulate cross-modal images.

[0068] 2.1 Aiming at the problem of brightness difference between cross-modal images, we introduce image brightness remapping to enhance visible light images. Image brightness remapping is to pre-design multiple value ranges in the [0,1] interval to include [0,1] After reading in the image, a random function is selected to recalculate the image brightness. This changes the brightness consistency between the two visible light images and simulates the brightness inconsistency between cross-modal images. To address the gradient flipping problem between cross-modal images, non-monotonic functions in the interval [0, 1] are designed to simulate it, such as quadratic functions, absolute value functions, and piecewise linear functions. Since images of different modalities generally come from different image sensors in real scenes and may have different imaging qualities, this is simulated by randomly adding Gaussian blur and sharpening to the input image. In addition, the three channels of the visible light image are randomly swapped to further expand the training set.

[0069] All the above cross-modal data enhancement methods are based on probability p The input images are processed independently of each other, that is, for each data augmentation method, there is a probability p is added to the processing of the input image. In this method, p=0.5 is used.

[0070] 2.2 Dataset Selection The present invention uses the Flying Chairs dataset for training visible light optical flow prediction networks. The dataset is composed of a realistic background scene and a simulated foreground of moving chairs.

[0071] Since both the foreground and background come from real scenes, this dataset can represent real scenes to a certain extent. And since this dataset is obtained through simulation, the displacement and rotation are controllable, and the annotated optical flow true value has a high credibility.

[0072] In the Flying Chairs public dataset, in addition to conventional data augmentation methods such as image cropping, image rotation, and image scaling, the cross-modal data augmentation method described in step 2.1 is also used to simulate a cross-modal dataset.

[0073] 2.3 Selection of Custom Loss Function. This embodiment uses the following custom loss function:

[0074]

[0075] Among them, β, λ, γ, θ, and α are all hyperparameters. In the present invention, β=1, λ=1, γ=0.1, θ=0.8, and α=0.01.

[0076] c i represents the credibility of the network after the output of the ith loop iteration is exponentially plotted, f gt represents the true value of the optical flow of the training set, f i represents the optical flow predicted by the network at the i-th loop iteration, L i Indicates the loss calculated for this iteration.

[0077] The weighted sum of the losses of all iterations during training gives the final total loss L total :

[0078]

[0079] 2.4 Training Parameter Settings. In this example, the initial learning rate is 0.0001, a 1-cycle learning rate decay algorithm is used, AdamW is used as the optimizer, the regularization coefficient is 0.00005, and the batch size is set to 4. The original RAFT network structure in the network is initialized using the RAFT network pre-trained model on Flying Chairs, and the newly added credibility estimation module is initialized using Kaiming.

[0080] After preprocessing, the two input images are fed into the cross-modal RAFT network. The preprocessing method is as follows: Since the convolutional layers in the RAFT network have downsampling, with a total of 8 times downsampling, the input images need to satisfy the requirement that both the length and width of the image are divisible by 8. Therefore, the input images need to be padded to add pixels with a value of 0. The image size after padding is:

[0081]

[0082] Where s0 represents the length or width of the original input image, and s represents the length or width of the padded image. Indicates rounding up. Since this algorithm supports any channel of a three-channel visible light image, near-infrared image, or visible light image as input, but the network needs to read a three-channel input image, a single-channel input image needs to be replicated three times to form a three-channel image.

[0083] like Figure 3 The iterative structure of the cross-modal credibility RAFT network is similar to that of the RAFT network. Multiple iterations of the same network structure can yield more refined confidence predictions for optical flow predictions. This method uses 12 iterations, and only the output of the last iteration is used for prediction. For the optical flow output by the network, the input image I2 can be directly transformed according to the displacement represented by the optical flow to obtain the result after registration with the input image I1. The credibility map output by the network is in logarithmic form.

[0084] The logarithmic credibility map output by the network needs to be exponentially calculated to obtain the final credibility estimate. Each pixel value in the credibility map is a positive number, indicating the credibility of the optical flow output by the network at that location. Larger pixel values ​​in the credibility map indicate greater uncertainty in the optical flow at that location, meaning that the optical flow prediction at that location is less accurate.

[0085] In this embodiment, the credibility logarithmic map is indexed to obtain the final credibility map; the input image I2 is transformed based on the predicted optical flow to obtain the registered input image, and the pixels filled in S3 are removed to obtain the final registration result.

[0086] Figure 5This is the experimental result on the simulation dataset. In a set of images in the visible light simulation dataset, the R channel of one image and the B channel of another image are taken as the cross-modal input images, such as Figure 5 (a) and 5(b).

[0087] The optical flow results predicted by this method are as follows Figure 5 (d) shows that the corresponding true value is Figure 5 As shown in (e), the reliability of the prediction is Figure 5 As shown in (c), we can see that the cross-modal registration results are good. In areas where accurate prediction cannot be made due to occlusion, the pixel values ​​in the credibility map are high, indicating that the credibility of this area is low.

[0088] Figure 6 The experimental results are based on visible-near infrared images taken in real scenes. The registered image I2 is superimposed on the input image I1 in the channel dimension to obtain the registration results for comparison. Figure 6 (f),The overall registration result is relatively accurate.,For roads with a single scene and high noise, the,prediction reliability of this method is low, which is consistent with the actual,condition.

[0089] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A cross-modal image optical flow prediction method based on RAFT network, the method comprising the following steps: S1: Design and construct a credibility RAFT network: Add a credibility estimation module to the cyclic iterative structure of the RAFT network to estimate the credibility of each area of ​​the output optical flow; In S1, the GRU in the RAFT network reads the correlation between the feature map 2 and the feature map 1 after the current optical flow transformation and the content encoding information of the input image 1, updates the internal latent variables, and outputs a new optical flow estimate; the credibility prediction module is connected after the GRU module of the RAFT network, participates in multiple loop iterations of the RAFT network, and outputs the credibility of the current estimated optical flow at each iteration; the credibility module simultaneously reads the latent variables, feature map 1, feature map 2 and optical flow, transforms feature map 2 according to the optical flow to obtain the transformed feature map 2, calculates the correlation with feature map 1, and after preliminarily integrating the features using 3×3 convolution, connects it with the latent variables in the channel dimension, and decodes through two convolution layers to output a credibility logarithmic map; S2: Training a cross-modal image optical flow prediction network: The credibility RAFT network is trained using cross-modal data augmentation to obtain a cross-modal credibility RAFT network. In S2, the cross-modal data augmentation method for training the credibility RAFT network means: on the existing visible light image training set annotated with the true value of optical flow, data augmentation is performed by using custom mapping functions such as brightness remapping, color channel swapping, Gaussian blurring, and image sharpening to weaken the brightness relationship between the image and the visible light image, simulate images of multiple different modalities, and train the credibility RAFT network's cross-modal image registration capabilities; S3: Input image preprocessing: Perform image preprocessing on the two input images to make them conform to the input format of the credibility RAFT network; S4: Obtain predicted optical flow and credibility logarithmic map: The two pre-processed input images are fed into the cross-modal credibility RAFT network to obtain the predicted optical flow and credibility logarithmic map; S5: Post-process the optical flow and credibility logarithmic map to obtain a prediction of the credibility of each area of ​​the optical flow.

2. The cross-modal image optical flow prediction method based on RAFT network according to claim 1 is characterized in that: In S2, the training process uses a custom loss function. For each loop iteration, the loss between the estimated credibility, the predicted optical flow, and the true optical flow value is calculated, and the final loss is weighted to calculate the gradient and backpropagation. The custom loss function is: Among them, β, λ, γ, θ, and α are all hyperparameters; c i represents the credibility graph obtained after the network's i-th cycle iteration, f gt represents the true value of the optical flow of the training set, f i represents the optical flow predicted by the network at the i-th loop iteration, l i Represents the loss Loss calculated from the output value of the network's i-th loop iteration, L total represents the total loss, which is obtained by weighting the losses of all loop iterations, and η is the weighting coefficient.

3. The cross-modal image optical flow prediction method based on RAFT network according to claim 1, characterized in that: The custom mapping function brightness remapping uses the following two mapping functions: Where 0.5≤a≤1,0 <b≤1; where 0 <a<1,0.5≤b≤1,d≥1; In the above two formulas, I is the original brightness of the image, I ′ is the brightness of the image after brightness remapping, and a, b, and d are constants generated by a random number generator each time image enhancement is performed.

4. The cross-modal image optical flow prediction method based on RAFT network according to claim 1, characterized in that: The color channel swapping is to scramble all channels of the multi-channel input image and reorganize them into a new multi-channel image in a random order.

5. The cross-modal image optical flow prediction method based on RAFT network according to claim 1, characterized in that: Adding Gaussian blur is to use the Gaussian blur kernel G to convolve the input image: Where I represents the input image, I' represents the image after Gaussian blur data enhancement, Represents a convolution operation.

6. The cross-modal image optical flow prediction method based on RAFT network according to claim 1, characterized in that: The image sharpening is achieved by the Laplacian operator in the following manner: I′=cΔI+kI Where c and k are constants. c is used to adjust the sharpening intensity, and k is used to prevent the brightness from exceeding the brightness range after addition. ΔI represents the result of convolving the image I with the following Laplacian template:

7. The cross-modal image optical flow prediction method based on RAFT network according to claim 1, characterized in that: In S3, the image preprocessing is as follows: if the input image is a single-channel image, it is copied three times to form a three-channel image; if the length or width of the input image is not a multiple of 8, it is padded with 0 pixels until the length and width are divisible by 8.

8. The cross-modal image optical flow prediction method based on RAFT network according to claim 1, characterized in that: In S5, the post-processing includes: 5.1 Take the exponent of the credibility logarithmic graph to obtain the final credibility graph; 5.2 Based on the predicted optical flow, the input image I2 is transformed to obtain the registered input image, and the pixels filled in S1 are removed to obtain the final registration result.

Images