A cross-modal image registration method optimized during the testing phase

Abstract

This invention discloses a cross-modal image registration method based on test-phase optimization. The invention uses a test-phase optimization method to train a registration neural network on intramodal images, and then performs registration for cross-modal images. The network model first uses a two-branch skeleton network to input the original image pair and the modality-invariant image pair respectively. Then, it extracts the image features of the two branches and calculates the cost volume separately. Next, it uses a cost aggregation network model to fuse and refine the cost information of the two branches. Finally, the result is input into an iterative homography estimator. This invention first trains the skeleton network and the iterative homography estimator using an intramodal image dataset; after training, the cost aggregation network model is trained on specific test image pairs during the test phase. Experimental results demonstrate that, without the need for additional multispectral or multimodal training images, the average corner error performance of this invention is among the best in the field.

Description

Technical Field

[0001] This invention relates to a modal image registration method in the field of computer vision technology, and in particular to a multispectral or multimodal image registration method that is optimized in the testing phase on a cost aggregation network model. Background Technology

[0002] With the development of various imaging sensors, multispectral and multimodal images have wide applications in computer vision, multispectral imaging, remote sensing, and medical image analysis. However, captured multispectral or multimodal images are often misaligned, hindering the accurate use of complementary information between modalities for further analysis and understanding. Therefore, a fundamental step is to register multispectral and multimodal image pairs.

[0003] Specifically, this task focuses on rigid body registration of multispectral or multimodal images. This task presents the following challenges:

[0004] (1) There is a large nonlinear radiation distortion (NRD) between different modes, which makes intensity- or gradient-based methods sensitive.

[0005] (2) The scarcity of publicly available, comprehensive datasets limits the ability of deep learning techniques to learn satisfactory models.

[0006] (3) Each pair of image modalities has its own unique characteristics, making it difficult to generate a universal registration model.

[0007] Previous work has involved designing spectral or modality-invariant image representations and optimizing them using traditional methods, such as DASC by Kim et al., RIFT by Li et al., and SCB by Cao et al. Relatively few approaches have attempted to address this problem using deep learning techniques. However, given the abundance of homography image data, deep learning techniques have been widely applied to homography image registration (UIR) and have made significant progress in recent years. For example, considering that homography transformations are suitable for most image alignments, various deep homography estimation methods have been proposed, such as DHN by DeTone et al., CLKN by Chang et al., MHN by Le et al., and UDHN by Zhang et al. Some of these use convolutional neural networks (CNNs) to directly regress the homography matrix, while others combine CNNs with untrainable Lucas Kanade iterators to achieve higher performance. Recently, Cao et al. proposed a fully trainable network, IHN, which employs an iterative refinement framework based on cost volume to achieve state-of-the-art performance. Summary of the Invention

[0008] To address the problems existing in the background art, the present invention provides a method for migrating from same-modal image registration to multispectral or multimodal image registration based on testing phase optimization, which is superior in key error indicators compared to other disclosed methods.

[0009] The technical solution adopted in this invention is:

[0010] 1) Train the same-modality image registration network model using the same-modality image registration dataset to obtain the trained same-modality image registration network model;

[0011] 2) Construct a cross-modal image registration network model based on the trained same-modal image registration network model. Input the test cross-modal image pair into the cross-modal image registration network model for testing phase optimization. After optimization, output the registration result of the current test cross-modal image pair.

[0012] The same-modality image registration network model includes a first convolutional neural network, a first feature similarity layer, a first lookup operator layer, and a first iterative homography estimator. The same-modality image registration dataset is input into the first convolutional neural network. After the first feature similarity layer calculates the similarity between the two feature maps output by the first convolutional neural network, it calculates the cost volume of the same-modality image pair, as shown in the following formula:

[0013]

[0014] Where, x S x T Represents the coordinates of two feature maps. This represents the vector of values ​​for two feature maps at corresponding coordinates; ReLU represents the ReLU activation function; T represents the transpose operation; C(x S x T () represents the cost volume between the corresponding coordinates of the two current feature maps;

[0015] The cost volume of image pairs of the same modality is used to calculate the similarity map using a lookup operator, as shown in the following formula:

[0016]

[0017] Where x is the corresponding coordinate of the feature map, and x′ is the coordinate of x after transformation by the estimated homography transformation matrix. This represents an operator that takes a 9×9 neighborhood centered at x′. This indicates an operation that extracts a portion of the coordinate pairs from the original cost volume for searching;

[0018] Finally, the similarity map is input into the first iterative homography estimator, which outputs the registration results of the same modal image pairs.

[0019] The cross-modal image registration network model includes an original image pair branch, a modality-invariant image pair branch, a second search operator layer, a third search operator layer, a cost aggregation module, and a second iterative homography estimator.

[0020] The original image pair branch is connected to the second search operator layer, and the modality-invariant image pair branch is connected to the third search operator layer. Both the second and third search operator layers are connected to the cost aggregation module, which is connected to the second iterative homography estimator. The second iterative homography estimator is connected to both the second and third search operator layers. The parameters of the second iterative homography estimator are the same as those of the first iterative homography estimator in the trained same-modality image registration network model. The output of the second iterative homography estimator is used as the output of the cross-modality image registration network model.

[0021] The original image pair branch includes a second convolutional neural network and a second feature similarity layer, with the second convolutional neural network connected to the second feature similarity layer. The modality-invariant image pair branch includes a Gabor filter layer, a third convolutional neural network, and a third feature similarity layer. The Gabor filter layer is connected to the third feature similarity layer after passing through the third convolutional neural network. The cross-modal image pairs are tested and input into the second convolutional neural network and the Gabor filter layer. The second feature similarity layer is connected to the second search operator layer, and the third feature similarity layer is connected to the third search operator layer. The parameters of the second and third convolutional neural networks are the same as the parameters of the first convolutional neural network in the trained same-modality image registration network model.

[0022] The cost aggregation module includes a confidence fusion layer and a cost aggregation network model. The outputs of the second search operator layer and the third search operator layer are both input into the confidence fusion layer. The outputs of the second search operator layer and the third search operator layer are also input into the cost aggregation network model. The output of the cost aggregation network model is added to the output of the confidence fusion layer and then used as the input of the second iterative homography estimator.

[0023] The cost aggregation network model includes weighted convolutional blocks and refined convolutional blocks. The output of the second search operator layer is added to the output of the third search operator layer and recorded as the total cost volume. The total cost volume is input into the weighted convolutional block. The output of the weighted convolutional block is multiplied by the total cost volume and then input into the refined convolutional block. The output of the refined convolutional block is used as the output of the cost aggregation network model.

[0024] The weighted convolutional block and the refined convolutional block have the same structure, both including a convolutional layer, an instance normalization layer, and an activation layer. The input of the convolutional block is used as the input of the convolutional layer. The convolutional layer is connected to the activation layer after passing through the instance normalization layer. The output of the activation layer is used as the output of the convolutional layer.

[0025] In the Gabor filter layer, for the input original image I(x), the original image I(x) is first convolved with the Gabor filter to obtain the convolution result. Then, the squared magnitude of the convolution result at scale s and direction o is calculated to obtain the magnitude squared map. The calculation formula is as follows:

[0026] A so (x)=||I(x)*G so (x)|| 2

[0027] Among them, G so (x) is the value of the Gabor filter's response function at pixel x in scale s and direction o. so (x) is the value of the squared magnitude map of the image at scale s and direction o at pixel x, |||| 2 The operator representing the square of the magnitude of a complex number;

[0028] Next, along direction o, the amplitude squared maps of all scales are superimposed to generate an amplitude feature map, calculated using the following formula:

[0029]

[0030] Among them, A o (x) represents the value of the magnitude feature map at pixel x, N s Indicates the total number of scales;

[0031] Then, by retrieving the maximum squared amplitude in different directions o from the amplitude feature map, the modal invariance map is obtained, and the calculation formula is as follows:

[0032]

[0033] Where M(x) represents the value of the modality-invariant map at pixel x. This represents the sequence of magnitude feature maps at pixel x in different directions, and max() represents taking the maximum value in the sequence.

[0034] The formula for the total loss function of the cost aggregation network model is as follows:

[0035]

[0036]

[0037]

[0038] in, This represents the total loss function value of the cost aggregation network model, where L represents the number of cascade levels and α is the weighting factor. Let Q' represent the loss calculated at a single cascade level, Q′ represent the set of sampling points, and || represent the number of elements in the set. F represents the contribution of feature point x to the loss function. T←S (x) represents the distortion feature at pixel x, F T (x) represents the feature at the same location as the distortion feature in the target image, T represents the matrix transpose, and τ2 is the second temperature hyperparameter.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. This invention proposes a novel cost aggregation method that operates in a test-phase optimized manner. By refining the cost volume on a pair of test images, this invention successfully achieves the transfer from monomodal image registration to multispectral or multimodal image registration.

[0041] 2. This invention utilizes the original image and modality invariant map to overcome large radiometric differences between modes, and designs a fusion strategy based on overhead volume to achieve more accurate homography estimation.

[0042] 3. This invention utilizes two cost volumes—the original image pair and the modality-invariant graph—to overcome differences between modalities. The two cost volumes are adaptively fused and refined to achieve more accurate homography estimation. Attached Figure Description

[0043] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of a network model for training same-modal image registration in an embodiment of the present invention.

[0045] Figure 3 This is a schematic diagram of the cross-modal image registration network model designed in an embodiment of the present invention.

[0046] Figure 4 This is a schematic diagram of the convolutional block used in the cost aggregation network model designed in this embodiment of the invention.

[0047] Figure 5 This is a schematic diagram of the cost aggregation network model used in the network model designed in this embodiment of the invention. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0049] The embodiments and specific implementation processes of the present invention are as follows:

[0050] like Figure 1 As shown, the present invention includes the following steps:

[0051] 1) Train the same-modality image registration network model using the same-modality image registration dataset to obtain the trained same-modality image registration network model;

[0052] 2) Construct a cross-modal image registration network model based on the trained same-modal image registration network model. Input the test cross-modal image pair into the cross-modal image registration network model and optimize it in the test phase using self-supervised loss. After optimization, output the registration result of the current test cross-modal image pair, i.e. the estimated homography transformation.

[0053] like Figure 2 As shown, the same-modality image registration network model includes a first convolutional neural network, a first feature similarity layer, and a first iterative homography estimator. The same-modality image registration dataset is input into the first convolutional neural network. After the first feature similarity layer calculates the similarity between the two feature maps output by the first convolutional neural network, it calculates the cost volume of the same-modality image pair, as shown in the following formula:

[0054]

[0055] Where, x S x T Represents the coordinates of two feature maps. This represents the vector of values ​​for two feature maps at corresponding coordinates; ReLU represents the ReLU activation function; T represents the transpose operation; C(x S x T () represents the cost volume between the corresponding coordinates of the two current feature maps;

[0056] The cost volume of image pairs of the same modality is used to calculate the similarity map using a lookup operator, as shown in the following formula:

[0057]

[0058] Where x is the corresponding coordinate of the feature map, and x′ is the coordinate of x after transformation by the estimated homography transformation matrix. This represents an operator that takes a 9×9 neighborhood centered at x′. This indicates an operation that extracts a portion of the coordinate pairs from the original cost volume for searching;

[0059] Finally, the similarity map is input into the first iterative homography estimator, which outputs the registration results of the same-modality image pairs. The same-modality image registration network model is a 1-scale IHtN network model. The initial parameters of the first convolutional neural network are those pre-trained in ImageNet.

[0060] like Figure 3As shown, the cross-modal image registration network model includes an original image pair branch, a modality-invariant image pair branch, a first search operator layer, a second search operator layer, a cost aggregation module, and a second iterative homography estimator.

[0061] The original image pair branch is connected to the first search operator layer, and the modality-invariant image pair branch is connected to the second search operator layer. Both the first and second search operator layers are connected to the cost aggregation module, which in turn is connected to the second iterative homography estimator. The second iterative homography estimator is connected to both the first and second search operator layers. The parameters of the second iterative homography estimator are the same as those of the first iterative homography estimator in the trained same-modality image registration network model. The output of the second iterative homography estimator serves as the output of the cross-modality image registration network model. In the search operator layer, the corresponding similarity map is output after retrieving the cost volume.

[0062] In the cost aggregation module, a cost aggregation network model and an improved cost residual method are used to fuse and aggregate similarity maps retrieved from two branches. Specifically, the two similarity maps are initially fused and then combined with the output of the cost aggregation network model to obtain a fused and refined relevance score.

[0063] The original image pair branch includes a second convolutional neural network and a second feature similarity layer, with the second convolutional neural network connected to the second feature similarity layer. The modality-invariant image pair branch includes a Gabor filter layer, a third convolutional neural network, and a third feature similarity layer. The Gabor filter layer is connected to the third feature similarity layer after passing through the third convolutional neural network. Test cross-modal image pairs are used as original image pairs and input into the second convolutional neural network and the Gabor filter layer. The original image pairs are processed by the Gabor filter layer to obtain modality-invariant image pairs. The second feature similarity layer is connected to the first search operator layer, and the third feature similarity layer is connected to the second search operator layer. The parameters of the second and third convolutional neural networks are the same as the parameters of the first convolutional neural network in the trained same-modality image registration network model.

[0064] In the Gabor filter layer, for the input original image I(x), the original image I(x) is first convolved with the Gabor filter to obtain the convolution result. Then, the squared magnitude of the convolution result at scale s and direction o is calculated to obtain the magnitude squared map. The calculation formula is as follows:

[0065] A so (x)=||I(x)*G so (x)|| 2

[0066] Among them, G so(x) is the value of the Gabor filter's response function at pixel x in scale s and direction o. so (x) is the value of the squared magnitude map of the image at scale s and direction o at pixel x, |||| 2 The operator representing the square of the magnitude of a complex number;

[0067] The Gabor filter layer is defined at scale s and direction o as follows:

[0068]

[0069]

[0070] Where x is a 2D coordinate, k so N represents the response coefficient at scale s and direction o. o Let σ represent the total number of directions, and let σ represent the bandwidth, where σ = 2π.

[0071] Next, along direction o, after superimposing the amplitude squared plots of all scales, a total of N s At each scale, an amplitude feature map is generated, and the calculation formula is as follows:

[0072]

[0073] Among them, A o (x) represents the value of the magnitude feature map at pixel x, N s Indicates the total number of scales;

[0074] Then, by retrieving the maximum squared amplitude in different directions o from the amplitude feature map, the modal invariance map is obtained, and the calculation formula is as follows:

[0075]

[0076] Where M(x) represents the value of the modality-invariant map at pixel x. This represents the sequence of magnitude feature maps at pixel x in different directions, and max() represents taking the maximum value in the sequence.

[0077] The second and third feature similarity layers calculate the similarity between the two feature maps generated by the corresponding convolutional neural network, and obtain the corresponding cost volume in the same way as the first feature similarity layer.

[0078] The cost aggregation module comprises a confidence fusion layer and a cost aggregation network model. The outputs of both the first and second search operator layers are input into the confidence fusion layer. These outputs are also input into the cost aggregation network model. The output of the cost aggregation network model is added to the output of the confidence fusion layer and then used as the input to the second iterative homography estimator. During the testing phase optimization, only the parameters of the cost aggregation network model are adjusted, while the parameters of other networks remain unchanged. After optimization, the second iterative homography estimator outputs the registration results of the test cross-modal image pairs.

[0079] In the confidence fusion layer, the initial fusion of the two similarity maps is adaptively performed based on the confidence scores corresponding to the similarity maps of the original image pairs and the similarity maps of the modality-invariant image pairs. The calculation formula is as follows:

[0080]

[0081]

[0082] Among them, S I S represents the similarity graph of the original image pairs. M The similarity map represents modality-invariant image pairs. w(x, S) is the confidence score of pixel x calculated from the similarity map S. The confidence score of pixel x is defined in the form of Softmax and is calculated by comparing the similarity scores of x with all its neighboring points. S is S I or S M τ1 represents the first temperature hyperparameter. Let represent the channel index corresponding to x′ in the similarity graph S, where i represents the 1D variable channel index.

[0083] like Figure 5 As shown, the cost aggregation network model includes weighted convolutional blocks and refined convolutional blocks. The output of the first search operator layer is added to the output of the second search operator layer and recorded as the total cost volume. The total cost volume is input into the weighted convolutional block. The output of the weighted convolutional block is multiplied by the total cost volume and then input into the refined convolutional block. The output of the refined convolutional block is used as the output of the cost aggregation network model.

[0084] The weighted convolutional block and the refined convolutional block have the same structure, such as... Figure 4 As shown, each layer includes a convolutional layer, an instance normalization layer, and an activation layer. The input of the convolutional block serves as the input to the convolutional layer, and the kernel size is 5×5. After passing through the instance normalization layer, the convolutional layer is connected to the activation layer, and the output of the activation layer serves as the output of the convolutional layer.

[0085] The specific calculation process of the cost aggregation network model is as follows: Aggregate similarity graph S I and S MFirst, the data is concatenated, then input into a weighted convolutional block to calculate pixel-by-pixel weights, obtaining the corresponding weight map. The similarity map is then multiplied by the weight map to obtain a weighted similarity map, which is further refined through a refining convolutional block. Both convolutional blocks consist of convolutional layers, instance normalization layers, and ReLU activation layers. The function CAN() is used to represent all operations in the cost aggregation network model, generating the residual similarity map.

[0086] S res (x)=CAN(S I (x), S M (x))

[0087] The formula for calculating the final similarity map in the entire cost aggregation module is as follows:

[0088] S fin (x)=S ini (x)+S res (x)

[0089] Among them, S fin (x) represents the refined similarity graph, S ini (x) represents the initial fusion of two similarity maps, S res (x) represents the residual similarity graph.

[0090] The formula for the total loss function of the cost aggregation network model is as follows:

[0091]

[0092]

[0093]

[0094] in, Let L represent the total loss function value of the cost aggregation network model, L represent the number of cascade levels, and α be a weighting factor, α∈(0,1). The weighting factor α indicates the degree of contribution of subsequent cascades to the loss function. Let Q' represent the loss calculated at a single cascade level, Q′ represent the set of sampling points, and || represent the number of elements in the set. Let F represent the contribution of feature point x to the loss function. T←S (x) represents the distortion feature at pixel x, F T (x) represents the feature at the same location as the distortion feature in the target image, T represents the matrix transpose, and τ2 is the second temperature hyperparameter.

[0095] Specifically, the method for optimizing the cost aggregation part of the network model during the testing phase is as follows:

[0096] Based on the cascading mechanism, the confidence contrast loss proposed in DMP is used for optimization, and different sampling strategies are designed to make learning more effective.

[0097] The goal of the loss function is to force the source image I to be transformed by the predicted homography matrix H during optimization. T←S With target image I T Increasingly aligned. It indicates the distortion feature F at pixel x. T←S (x) Features F at the same location as the target image T The correlation of (x) should be maximized, while the correlations of other features should be minimized. Therefore, the loss is defined as follows:

[0098]

[0099] The sampling strategy for cost aggregation of the network model during the testing phase optimization is as follows:

[0100] Instead of considering all pixels, we select a subset of pixels. Specifically, we first select the P pixels with the highest confidence. Then, we randomly select Q pixels (Q≤P) from this subset, ignoring pixels with confidence less than a threshold T. In this way, we obtain the sample set Q′ used to calculate the loss.

[0101]

[0102] The cascading mechanism in the cost aggregation of the network model during the testing phase optimization is as follows:

[0103] The model follows the L-level cascaded global cross-modal image registration network model in DMP. Therefore, the final loss is defined as:

[0104]

[0105] To verify the effectiveness of this invention, it was validated on publicly available multispectral image datasets CAVE and Harvard, as well as multimodal datasets RGB / NIR and Flash / No-flash, and compared with the most advanced cross-modal image registration methods:

[0106] CAVE is a multispectral dataset captured by a general-purpose classification pixel camera; 682 image pairs were used for testing in the experiment. Harvard University contains images taken by commercial hyperspectral cameras in outdoor and indoor scenes; 2310 image pairs were used for testing in the experiment. RGB / NIR contains image pairs captured by color (RGB) and near-infrared (NIR) cameras in outdoor scenes; 256 image pairs were used for testing in the experiment. Flash / No-flash is a dataset containing images with exposure variations acquired by the same camera; 120 image pairs were used for testing in the experiment. By default, all image pairs in these datasets are aligned and tested at a size of 256×256. To perform extensive evaluation, for each aligned image pair, we keep the target image fixed while using three simulated transformations to warp the source images, including:

[0107] a gt，s = (1.1, 0.1, -10, -0.1, 1.1, 10) T

[0108] a gt，m = (1.15, 0.15, -15, -0.15, 1.15, 15) T

[0109] a gt，l = (1.2, 0.2, -20, -0.2, 1.2, 20) T

[0110] This invention uses the average corner error (ACE) as the evaluation metric. To achieve optimal network performance, the parameter details are as follows:

[0111] In a Gabor filter, the total number of scales N s The number is 4, and the number of directions is N. o The value is 6. τ2 is set to 10 for confidence calculation τ1 = 0.1 and for self-supervised contrastive loss. In the sampling policy, P = 512, Q = 256, and T = 0.001. The number of levels in the cascade L is set to 4, and α = 0.75. Unless otherwise specified, these hyperparameters are fixed in all experiments. We pre-train a 1-scale IHN on the same-modality image dataset MSCOCO and do not use multispectral or multimodal datasets for training. During testing, we resize all images to 128×128 and resize single-channel images to three-channel images to match the IHN input. Furthermore, we optimize using the ADAM optimizer with a learning rate of 0.001. All weights in the convolutional blocks of the cost aggregation network model are initialized to 0. The cost aggregation network model is optimized for 20 epochs for each pair of test images.

[0112] The experiment mainly consists of two parts. The first part is a comparative experiment between the method of this invention and the most advanced image registration methods currently available. The second part is a controlled variable experiment of each module in the network of this invention, which is to illustrate the effectiveness of each module in this invention.

[0113] Part 1: Comparative experiments with the most advanced image registration methods to demonstrate the superior performance of the network in this invention.

[0114] Tables 1 and 2 show the comparison results of the complete model of the present invention (denoted by TCA) with other state-of-the-art methods on the multispectral image datasets CAVE and Harvard, and the multimodal datasets RGB / NIR and Flash / No-flash, respectively.

[0115] As shown in Table 1, the complete model outperforms other published or backup paper methods in terms of ACE percentage (less than 1%), ACE percentage (less than 5%), and average ACE (mACE) performance on CAVE and Harvard.

[0116] Table 1 compares the performance metrics of this invention with those of published state-of-the-art (SOTA) methods on CAVE and Harvard.

[0117]

[0118] As shown in Table 2, the complete model outperforms other published or backup methods in RGB / NIR and Flash / No-flash environments with an ACE of less than 1% and less than 5%. Its average ACE (mACE) is only inferior to one other method.

[0119] Table 2 compares the performance indicators of this invention with those of published state-of-the-art (SOTA) methods in RGB / NIR and Flash / No-flash.

[0120]

[0121] Part Two: The controlled variable experiments for each module in this invention are to illustrate the effectiveness of each module in this invention, and the statistics are the percentage of image pairs with an ACE error of less than 1.

[0122] Effectiveness of the designed dual-branch structure:

[0123] Three variants can be obtained by using both ORI and MIM image pairs as input, and by using only ORI or MIM pairs as input. Table 3 lists the comparison results. It shows that the dual-branch architecture using both ORI and MIM representations is always superior to the architecture using only ORI or MIM representations, proving the complementarity of ORI and MIM images.

[0124] Table 3 is a comparison table of different input methods.

[0125]

[0126] Effectiveness of test-phase optimization: Two variants can be obtained depending on whether test-phase optimization is used. The comparison results are listed in Table 4. It shows that the cost aggregation network model using test-phase optimization outperforms the adaptive fusion method alone by a considerable margin, demonstrating the importance of this strategy.

[0127] Table 4 compares whether the cost aggregation network module underwent optimization during the testing phase.

[0128]

[0129] Effectiveness of the adaptive initialization residual structure: Two variations can be obtained by superimposing the adaptive initialization results with or without using an adaptive residual structure. The comparison results are listed in Table 5. It shows that the residual method consistently outperforms its corresponding method on all four datasets.

[0130] Table 5 compares the use of adaptive residual structures with direct optimization.

[0131]

[0132] The effect of the sample size P: Three variations can be obtained based on the sample size P of 256, 512, and 1024. Table 6 lists the comparison results. It shows that the best performance is achieved when P = 512.

[0133] Table 6 shows the impact of different sampling quantities P on performance.

[0134]

[0135] The above description is merely a specific embodiment of the present invention and is not intended to limit the present invention in any way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above examples based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A cross-modal image registration method based on testing phase optimization, characterized in that, Includes the following steps: 1) Train the same-modality image registration network model using the same-modality image registration dataset to obtain the trained same-modality image registration network model; 2) Construct a cross-modal image registration network model based on the trained same-modal image registration network model. Input the test cross-modal image pair into the cross-modal image registration network model for optimization during the testing phase. After optimization, output the registration result of the current test cross-modal image pair. The same-modality image registration network model includes a first convolutional neural network, a first feature similarity layer, a first lookup operator layer, and a first iterative homography estimator. The same-modality image registration dataset is input into the first convolutional neural network. After the first feature similarity layer calculates the similarity between the two feature maps output by the first convolutional neural network, it calculates the cost volume of the same-modality image pair, as shown in the following formula: in, , Represents the coordinates of two feature maps. , This represents the value vectors of two feature maps at corresponding coordinates, and ReLU represents the ReLU activation function. Indicates the transpose operation; This represents the cost volume between the two current feature maps at their respective coordinates; The cost volume of image pairs of the same modality is used to calculate the similarity map using a lookup operator, as shown in the following formula: in, These are the corresponding coordinates of the feature map. yes The coordinates after transformation by the estimated homography transformation matrix. Indicated by Take as the center Neighborhood operators, This indicates an operation that extracts a portion of the coordinate pairs from the original cost volume for searching; Finally, the similarity map is input into the first iterative homography estimator, which outputs the registration results of the same modal image pairs. The cross-modal image registration network model includes an original image pair branch, a modality-invariant image pair branch, a second search operator layer, a third search operator layer, a cost aggregation module, and a second iterative homography estimator. The original image pair branch is connected to the second search operator layer, and the modality-invariant image pair branch is connected to the third search operator layer. Both the second and third search operator layers are connected to the cost aggregation module, which is connected to the second iterative homography estimator. The second iterative homography estimator is connected to both the second and third search operator layers. The parameters of the second iterative homography estimator are the same as those of the first iterative homography estimator in the trained same-modality image registration network model. The output of the second iterative homography estimator is used as the output of the cross-modality image registration network model.

2. The cross-modal image registration method based on testing phase optimization according to claim 1, characterized in that, The original image pair branch includes a second convolutional neural network and a second feature similarity layer, with the second convolutional neural network connected to the second feature similarity layer. The modality-invariant image pair branch includes a Gabor filter layer, a third convolutional neural network, and a third feature similarity layer. The Gabor filter layer is connected to the third feature similarity layer after passing through the third convolutional neural network. The cross-modal image pairs are tested and input into the second convolutional neural network and the Gabor filter layer. The second feature similarity layer is connected to the second search operator layer, and the third feature similarity layer is connected to the third search operator layer. The parameters of the second and third convolutional neural networks are the same as the parameters of the first convolutional neural network in the trained same-modality image registration network model.

3. The cross-modal image registration method based on testing phase optimization according to claim 1, characterized in that, The cost aggregation module includes a confidence fusion layer and a cost aggregation network model. The outputs of the second search operator layer and the third search operator layer are both input into the confidence fusion layer. The outputs of the second search operator layer and the third search operator layer are also input into the cost aggregation network model. The output of the cost aggregation network model is added to the output of the confidence fusion layer and then used as the input of the second iterative homography estimator.

4. The cross-modal image registration method based on testing phase optimization according to claim 3, characterized in that, The cost aggregation network model includes weighted convolutional blocks and refined convolutional blocks. The output of the second search operator layer is added to the output of the third search operator layer and recorded as the total cost volume. The total cost volume is input into the weighted convolutional block. The output of the weighted convolutional block is multiplied by the total cost volume and then input into the refined convolutional block. The output of the refined convolutional block is used as the output of the cost aggregation network model.

5. The cross-modal image registration method based on testing phase optimization according to claim 4, characterized in that, The weighted convolutional block and the refined convolutional block have the same structure, both including a convolutional layer, an instance normalization layer, and an activation layer. The input of the convolutional block is used as the input of the convolutional layer. The convolutional layer is connected to the activation layer after passing through the instance normalization layer. The output of the activation layer is used as the output of the convolutional layer.

6. The cross-modal image registration method based on testing phase optimization according to claim 2, characterized in that, In the Gabor filter layer, for the input original image First, the original image Convolve the result with a Gabor filter to obtain the convolution result, and then calculate the convolution result at scale s and orientation. The square of the amplitude is used to obtain the amplitude square graph. The calculation formula is as follows: in, This is the value of the Gabor filter's response function at pixel x, along with scale s and direction o. It is the value of the squared magnitude plot of the image at scale s and direction o at pixel x. The operator representing the square of the magnitude of a complex number; Next, in the direction Above, after overlaying the amplitude squared plots at all scales, an amplitude feature map is generated. The calculation formula is as follows: in, This represents the value of the magnitude feature map at pixel x. Indicates the total number of scales; Then, by retrieving the amplitude feature map in different directions The modal invariance plot is obtained by squared the maximum amplitude on the curve, and the calculation formula is as follows: in, This represents the value of the modality-invariant map at pixel x. This represents the sequence of magnitude feature maps at pixel x in different directions. This indicates taking the maximum value in the sequence.

7. The cross-modal image registration method based on testing phase optimization according to claim 3, characterized in that, The formula for the total loss function of the cost aggregation network model is as follows: in, This represents the total loss function value of the cost aggregation network model. Indicates the number of cascade levels. It is a weighting factor. This represents the loss calculated at a single cascade level. Represents the set of sampling points. This indicates the number of elements in the set. This represents the contribution of feature point x to the loss function. Represents pixels Distortion features at the location, This represents the feature at the same location in the target image as the distortion feature, where T denotes the matrix transpose. It is the second temperature hyperparameter.

Images