Heterogeneous image registration method and device based on cost volume

Abstract

The present invention discloses a method and device for heterogeneous image registration based on a cost volume, which comprises the following steps: obtaining a reference image and a template image; extracting a first feature map of the reference image and a second feature map of the template image respectively through a U-Net network; performing an affine transformation on the second feature map based on L groups of predetermined rotation and scale parameters to obtain L third feature maps corresponding to the second feature map; performing Fourier convolution using the third feature map as a three-dimensional convolution kernel and the first feature map as a three-dimensional input signal to obtain L phase correlation maps; constructing a cost volume based on the L phase correlation maps to calculate rotation parameters, scale parameters, and translation parameters between the reference image and the template image; extracting feature maps of the reference image and the template image through a U-Net network, then performing an affine transformation on the extracted feature maps, and generating a phase correlation map in combination with Fourier convolution, the method can reduce the computational complexity of heterogeneous image registration and shorten the registration time.

Description

Technical Field

[0001] The present invention belongs to the technical field of heterogeneous image registration, and in particular relates to a heterogeneous image registration method and device based on cost volume. Background Art

[0002] Image registration is the process of finding correspondences between two or more images of the same scene, captured by the same or different sensors. Heterogeneous image registration specifically refers to the registration of images from different sensors. However, due to differences in the imaging mechanisms and conditions of different sensors, traditional homologous registration methods often face challenges with heterogeneous data. For example, traditional image registration methods calculate image similarity based on grayscale features. However, heterogeneous images often exhibit differences in brightness, contrast, and other intensity factors. Using grayscale features for similarity calculations can result in significant errors.

[0003] Existing heterogeneous image registration methods are mainly divided into traditional registration methods based on hand-designed features and registration methods based on neural networks. Traditional hand-designed registration methods usually use the grayscale information of the image or hand-designed feature descriptors as the feature representation of the image, use these feature representations to extract the key points of the image, and calculate the similarity between the two images at the key points. However, the grayscale information and hand-designed feature descriptors commonly used in general optical image registration often perform poorly on heterogeneous image data due to differences in imaging principles. At the same time, hand-designed feature descriptors are often targeted at specific datasets, have a high dependence on expert knowledge, and do not have good generalization properties.

[0004] Neural network-based methods fall into two categories. One is feature point-based matching, which follows a similar process to traditional methods: first, key points are extracted, and then feature descriptors are extracted using convolutional neural networks at these key points. However, these methods place high demands on the key point extraction process. If large-scale rotations or large scale transformations occur, key point detection is likely to fail, leading to subsequent matching failures. Another type of method uses phase correlation for registration, but this method typically uses two neural networks to estimate rotation scale parameters and translation parameters separately. This significantly increases the computational complexity, creates computational redundancy, and places high demands on the operating equipment. Furthermore, the two-stage model training fails to take into account the equivariance of the network, making it difficult to train. Furthermore, the phase correlation method requires the training and test images to be of the same size; otherwise, it will not work, which severely impacts the generalization of the network. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and device for heterogeneous image registration based on cost volume, so as to reduce the computational complexity of heterogeneous image registration and shorten the registration time.

[0006] The present invention adopts the following technical solution: a heterogeneous image registration method based on a cost volume, comprising the following steps:

[0007] Obtain a reference image and a template image;

[0008] The first feature map of the reference image and the second feature map of the template image are extracted respectively through the U-Net network;

[0009] Performing an affine transformation on the second feature map based on L groups of predetermined rotation scale parameters to obtain L third feature maps corresponding to the second feature map; where L is a positive integer;

[0010] Perform Fourier convolution using the third feature map as the three-dimensional convolution kernel and the first feature map as the three-dimensional input signal to obtain L phase correlation maps;

[0011] The cost volume is constructed based on L phase correlation maps, and the rotation parameters, scale parameters and translation parameters between the reference image and the template image are calculated.

[0012] Furthermore, the cost volume is constructed based on the L phase correlation maps to calculate the rotation parameters, scale parameters and translation parameters between the reference image and the template image, including:

[0013] Cascade L phase correlation graphs on a predetermined channel dimension to generate a cost volume;

[0014] Calculate the maximum matching probability in the cost body;

[0015] Determine the pixel on the cost volume according to the maximum matching probability;

[0016] Extract the rotation parameters, scale parameters, and translation parameters corresponding to the pixel.

[0017] Furthermore, extracting the rotation parameters and scale parameters corresponding to the pixels includes:

[0018] Determine a corresponding phase correlation map according to the pixel;

[0019] A corresponding predetermined rotation scale parameter is selected according to the phase correlation diagram to obtain a rotation parameter and a scale parameter.

[0020] Furthermore, extracting the translation parameters corresponding to the pixels includes:

[0021] Obtain the coordinate value of the pixel in the corresponding phase correlation map and use the coordinate value as the translation parameter.

[0022] Furthermore, the U-Net network consists of three double convolutional layers, two max pooling layers, and one 1×1 convolutional layer;

[0023] Among them, the dual convolutional layer includes two convolutional layers, two batch normalization layers and two ReLU layers.

[0024] Furthermore, the method for generating the predetermined rotation scale parameter is:

[0025] generating a plurality of predetermined rotation parameters according to a rotation parameter threshold, and generating a plurality of predetermined scale parameters according to a scale parameter threshold;

[0026] Any predetermined rotation parameter and a predetermined scale parameter are combined to obtain a predetermined rotation scale parameter.

[0027] Furthermore, the loss function of the U-Net network during training is:

[0028]

[0029] Among them, L oss is the loss function, V (R,S,tx,ty) is a set of the true values of the rotation parameters, scale parameters, and translation parameters of the template image during training. is the set of rotation parameters, scale parameters and translation parameters corresponding to the j-th phase correlation graph during training, j = 1, 2, ..., L.

[0030] Furthermore, performing an affine transformation on the second feature map based on the L groups of predetermined rotation scale parameters includes:

[0031]

[0032] Among them, (x, y) is the pixel in the second feature map, is the pixel corresponding to (x, y) in the third feature map, S i is the scale parameter in the i-th set of predetermined rotation scale parameters, R i is the rotation parameter in the i-th set of predetermined rotation scale parameters, i=1, 2, ..., L.

[0033] Another technical solution of the present invention: a heterogeneous image registration device based on a cost volume, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned heterogeneous image registration method based on a cost volume is implemented.

[0034] The beneficial effects of the present invention are as follows: the present invention extracts feature maps of the reference image and the template image through the U-Net network, then performs affine transformation on the extracted feature maps, and combines Fourier convolution to generate a phase correlation map. The main operations are matrix multiplication and Fourier transform, which can reduce the computational complexity of heterogeneous image registration and reduce the registration time. It has the characteristics of platform independence and can be applied to hardware platforms such as GPU, CPU, and FPGA. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1Schematic diagram of the technical principle of a cost volume-based heterogeneous image registration method according to an embodiment of the present invention;

[0036] Figure 2 Schematic diagram of a template image and a reference image in an embodiment of the present invention;

[0037] Figure 3 Schematic diagram of characteristic graph S1 and characteristic graph S2 in an embodiment of the present invention;

[0038] Figure 4 The four feature maps after affine transformation in the embodiment of the present invention are

[0039] Figure 5 The template image and the reference image after registration in the embodiment of the present invention;

[0040] Figure 6 Schematic diagram of template images and reference images at various stages in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Traditional matching methods can be categorized as optimization-based and hard regression-based. Optimization-based methods take a long time to infer and often struggle to find the optimal solution. Hard regression-based methods, on the other hand, directly use affine parameters as supervisory signals, resulting in poor generalization and accuracy, and requiring a large amount of training data to achieve good results.

[0043] The present invention discloses a heterogeneous image registration method based on cost volume, such as Figure 1 As shown, the method includes the following steps: obtaining a reference image and a template image; extracting a first feature map of the reference image and a second feature map of the template image respectively through a U-Net network; performing an affine transformation on the second feature map based on L groups of predetermined rotation scale parameters to obtain L third feature maps corresponding to the second feature map; wherein L is a positive integer; performing Fourier convolution with the third feature map as a three-dimensional convolution kernel and the first feature map as a three-dimensional input signal to obtain L phase correlation maps; and calculating the rotation parameter, scale parameter and translation parameter between the reference image and the template image based on a cost volume constructed according to the L phase correlation maps.

[0044] The present invention extracts feature maps of the reference image and template image through the U-Net network, then performs affine transformation on the extracted feature maps, and combines them with Fourier convolution to generate a phase correlation map. The main operations are matrix multiplication and Fourier transform, which can reduce the computational complexity of heterogeneous image registration and shorten the registration time. It has the characteristics of platform independence and can be applied to hardware platforms such as GPU, CPU, and FPGA.

[0045] In one embodiment, calculating the rotation parameters, scale parameters, and translation parameters between a reference image and a template image based on a cost volume constructed from L phase correlation maps includes: concatenating the L phase correlation maps along a predetermined channel dimension to generate a cost volume; calculating the maximum matching probability within the cost volume; determining pixels within the cost volume based on the maximum matching probability; and extracting the rotation parameters, scale parameters, and translation parameters corresponding to the pixels. The cost volume is called a cost volume, and the cost refers to the output obtained by performing a correlation operation, such as a similar operation, on the feature maps of two input images.

[0046] In registration scenarios, affine changes such as rotation, scale, and translation exist between the two input images. To determine these parameters, multiple combinations of these parameters can be constructed within a certain range. A cost is calculated for each combination, and the resulting three-dimensional volume, obtained by concatenating the costs calculated for all combinations, is called the cost volume. This cost volume-based registration method transforms the matching problem into one of identifying the optimal matching position within the cost volume. It can calculate the three affine parameters of rotation, scale, and translation in one go, significantly reducing computational effort and time.

[0047] This method is applied to the field of real-time heterogeneous remote sensing image registration. It uses a deep Siamese network to extract image feature maps, specifies rotation and scale ranges, and then combines them into segments. One feature map is transformed using the rotation and scale parameters of each combination, and then its correlation is calculated with the other feature map. The calculated correlations for each combination are concatenated to form a three-dimensional volume. After normalizing the volume using 3D convolution, the location with the maximum response value is determined, and the corresponding dimension is the affine parameter to be solved.

[0048] Specifically, the U-Net network used in the embodiment of the present invention needs to be trained in advance. The U-Net network consists of three double convolutional layers, two maximum pooling layers and one 1×1 convolutional layer; wherein the double convolutional layer includes two convolutional layers, two batch normalization layers and two ReLU layers.

[0049] During the training phase, the parameters of the U-Net network are trained using the template image (i.e., the image to be registered), the reference image, and the true values of the rotation parameters, scale parameters, horizontal translation, and vertical translation corresponding to the template image. The specific process is as follows:

[0050] S01: Input an optical image of size 256*256 as a reference image and a remote sensing image of resolution 192*192 as a template image (optical images and remote sensing images are interchangeable). Perform data preprocessing on the dataset, perform affine transformations such as rotation, scaling, and translation on the remote sensing image, and obtain the true values of the rotation parameters, scaling parameters, and translation parameters as the known dataset.

[0051] S02: Input the preprocessed template image into a deeper U-Net network to obtain a feature map S1 of size [1, c, 48, 48]. Input the preprocessed reference image into a deeper U-Net network to obtain a feature map S2 of size [1, c, 64, 64]. The specific mathematical representation is as follows:

[0052]

[0053] Among them, I 模板 represents the template image, I 参考 represents the reference image, encoder A U-Net network used to represent the template image, encoder B The U-Net network used to represent the reference image is actually the encoder A and encoder B The network structures are the same and the network parameters are updated simultaneously.

[0054] S03: Based on the true rotation parameter values and scale parameter values in the dataset, define the rotation parameter range threshold and scale parameter range threshold during training. Combine the true rotation parameter values and scale parameter values within the rotation parameter range threshold and scale parameter range threshold to obtain a combined parameter set SR, where S is the true scale parameter value and R is the true rotation parameter value. For example, if there are three true rotation parameter values and three true scale parameter values, the resulting parameter set contains nine data sets. The mathematical representation is as follows:

[0055] SR=[(S min ,R min ),...,(S i ,R i ),...,(S max ,R max )] (2)

[0056] Among them, (S min ,R min ) represents the parameter set consisting of the minimum scale parameter true value and the minimum rotation parameter true value, (S i ,R i ) represents the parameter set consisting of the true value of the i-th scale parameter and the true value of the i-th rotation parameter, (S max ,Rmax ) represents the parameter set consisting of the maximum scale parameter true value and the maximum rotation parameter true value.

[0057] S04: According to each set of data (S i ,R i ) Perform affine transformation on the feature map of the template image to obtain L

[0058] S05: L As a three-dimensional convolution kernel, S2 is used as a three-dimensional input signal, and Fourier convolution is performed to obtain L two-dimensional phase correlation graphs RS i , these phase correlation graphs are cascaded in the channel dimension to form the cost volume V. Mathematically expressed as:

[0059]

[0060] Among them, FFT means Fourier convolution and concat means cascade.

[0061] The cost volume should now be a four-dimensional tensor, with the four dimensions B, L, H, and W. B represents the batch size, L indicates the number of phase correlation maps, and H and W correspond to the height and width of the feature map. The cost volume is then normalized by computing the three-dimensional convolution.

[0062] The optimal SR parameter combination needs to be found. The value of each position in the cost volume is related to the matching probability. The greater the matching probability of a pixel, the greater the value of the pixel at the corresponding position in the cost volume. The argmax function is used to solve the position index of the maximum value in the cost volume, thereby solving the matching rotation, scale and translation parameters (R p ,S p ,tx p ,ty p ).

[0063] In this embodiment, the cross entropy loss function is used to supervise the cost body V, the optimization target is the true value of the rotation parameter R, the true value of the scale parameter S and the true value of the translation parameter (tx, ty), the loss is Loss, back propagation is performed, and the AdamW optimizer is used to update the parameters of the network, and the above process is repeated until the loss function converges.

[0064] More specifically, the loss function is:

[0065]

[0066] Among them, Loss is the loss function, V (R,S,tx,ty) is a set of the true values of the rotation parameters, scale parameters, and translation parameters of the template image during training. is the set of rotation parameters, scale parameters and translation parameters corresponding to the j-th phase correlation graph during training, j = 1, 2, ..., L.

[0067] In this embodiment, in the actual application stage, the general process is similar to the training process, except that the true values of the scale parameters, the rotation parameters, and the translation parameters are not available, and prediction is performed directly.

[0068] In a specific implementation, extracting the rotation parameter and scale parameter corresponding to a pixel includes: determining a corresponding phase correlation map based on the pixel; and selecting a corresponding predetermined rotation scale parameter based on the phase correlation map to obtain the rotation parameter and scale parameter. Extracting the translation parameter corresponding to the pixel includes: obtaining the coordinate value of the pixel in the corresponding phase correlation map, and using the coordinate value as the translation parameter.

[0069] In addition, in one embodiment, the method for generating the predetermined rotation scale parameter is: generating multiple predetermined rotation parameters according to a rotation parameter threshold, generating multiple predetermined scale parameters according to a scale parameter threshold; and combining any predetermined rotation parameter and predetermined scale parameter to obtain the predetermined rotation scale parameter.

[0070] More specifically, performing an affine transformation on the second feature map based on L groups of predetermined rotation scale parameters includes:

[0071]

[0072] Among them, (x, y) is the pixel in the second feature map, is the pixel corresponding to (x, y) in the third feature map, S i is the scale parameter in the i-th set of predetermined rotation scale parameters, R i is the rotation parameter in the i-th set of predetermined rotation scale parameters, i=1, 2, ..., L.

[0073] In order to describe the present invention more clearly, the method of the present invention is specifically described with reference to specific examples.

[0074] First, if Figure 2 As shown, the input optical image with a size of 256*256 is used as the reference image, that is, Figure 2 (b), the remote sensing image with a resolution of 192*192 is used as the template image, that is, Figure 2 (a) Data preprocessing is performed on the dataset. The remote sensing image is rotated by 0.8514 radians, scaled by 0.2782, translated by 145 pixels in the x-direction, and translated by 125 pixels in the y-direction. The affine transformation parameters (0.8514, 0.2782, 145, 125) are obtained.

[0075] like Figure 3As shown, the preprocessed template image is then input into the U-Net network to obtain a feature map S1 of size [1, 64, 48, 48], that is, Figure 3 (a); Input the preprocessed reference image into the U-Net network to obtain the feature map S2 of size [1, 64, 64, 64], that is, Figure 3 (b).

[0076] Then, the rotation and scale ranges are defined, and the rotation and scale parameters are combined. The combined parameter set is SR. There are a total of 288 parameter sets. The rotation parameters are divided into 12 groups from -0.3927 to 0.3927, and the scale parameters are divided into 24 groups from 0.75 to 1.5.

[0077] SR=[(-0.3927,0.75),...(-0.0357,0.75),...,(0.3927,1.25)] (6)

[0078] Then, according to the scale and rotation parameters (S i ,R i ) Perform affine transformation on the feature map of the template image to obtain 288 like Figure 4 As shown, Figure 4 (a) The corresponding parameters are R = -0.3927, S = 0.75, Figure 4 (b) The corresponding parameters are R = 0.1785, S = 0.837; Figure 4 (c) The corresponding parameters are R = 0.2499, S = 1.1196; Figure 4 (d) The corresponding parameters are R = 0.3927, S = 1.25.

[0079] 288 As a three-dimensional convolution kernel, S2 is used as a three-dimensional input signal, and Fourier convolution is performed to obtain 288 two-dimensional phase correlation graphs RS i , these phase correlation graphs are cascaded in the channel dimension to form the cost volume V.

[0080] The cost volume should now be a four-dimensional tensor with the four dimensions [1, 288, 65, 65]. The value of each position in the cost volume is related to the matching probability. The greater the matching probability of a pixel, the greater the value of the pixel at the corresponding position in the cost volume. Use the argmax function to solve the position index of the maximum value in the cost volume, and then solve the matching rotation, scale and translation parameters (0.8370, 0.2499, 144, 124). According to the parameters, the transformation is obtained, and the registration result is as follows Figure 5 shown.

[0081] In addition, the present invention also carried out other experiments, the specific schematic diagram is as follows Figure 6 From the result diagram in the figure, it can be seen that the method of the present invention is applicable to a variety of remote sensing scenes, such as urban areas, mountains, rivers, cultivated land, etc. Figure 6 (a) and Figure 6 (c) represents a mountain river scene, 6(b), Figure 6 (d) and Figure 6 (g) represents an urban scene, 6(e) represents a mountain scene, Figure 6 (f) represents the farmland scene.

[0082] Figure 6 (a1), Figure 6 (b1), Figure 6 (c1), Figure 6 (d1), Figure 6 (e1), Figure 6 (f1) and Figure 6 (g1) is the reference image of the corresponding scene, Figure 6 (a2), Figure 6 (b2), Figure 6 (c2), Figure 6 (d2), Figure 6 (e2), Figure 6 (f2) and Figure 6 (g2) is the template image of the scene, Figure 6 (a3), Figure 6 (b3) Figure 6 (c3), Figure 6 (d3), Figure 6 (e3), Figure 6 (f3) and Figure 6 (g3) is the image after affine transformation of the template image in this scene, Figure 6 (a4), Figure 6 (b4), Figure 6 (c4), Figure 6 (d4), Figure 6 (e4), Figure 6 (f4) and Figure 6 (g4) is the fused image of the scene.

[0083] The method of the present invention can achieve end-to-end training. The cost volume-based matching method used can handle scenes with large-scale affine transformations compared to traditional feature point matching methods. Compared with traditional phase correlation methods, this method has good generalization. At the same time, compared with existing neural network-based matching methods, the method of the present invention has higher efficiency and accuracy, and is highly efficient. The phase correlation map calculation in the registration parameter stage uses Fourier convolution, which greatly reduces the amount of calculation. At the same time, the method of the present invention adopts a data-driven approach, and the feature extraction network it uses uses a deeper U-Net network, which can handle different features caused by different sensors. Therefore, the network has good portability and is suitable for the registration of various multimodal images, including but not limited to bird's-eye view and drone image registration, multimodal medical images, multimodal natural images, etc.

[0084] In addition, the loss function proposed in the present invention has good trainability and generalization in this problem. The cross entropy loss function of the rotation scale and the cross entropy loss function of the translation parameter are jointly supervised, making the network more robust.

[0085] This cost volume-based approach constructs a cost volume by calculating multiple possible matching combinations, thereby transforming the matching problem into a detection problem of identifying the optimal matching location within the cost volume. Experiments have shown that the model trained by this method has very strong generalization and accuracy, and achieves high inference speed on devices that support parallel computing, such as GPUs and FPGAs.

[0086] Specifically, the image feature extraction module uses a deeper U-Net network with shared parameters. The network can simultaneously learn the features of SAR and optical images. Using a single network to simultaneously learn the parameters of both images can greatly reduce network parameters and enhance its transfer capabilities. Using a deeper U-Net network can extract deeper information. The cost volume-based method does not require the training and testing sizes to be consistent, and has better generalization properties than traditional registration methods and phase correlation registration methods. The cost volume can directly estimate the optimal rotation, translation, and scale parameters. Therefore, in the process of calculating the cross-entropy loss, it is only necessary to mix the rotation, translation, and scale parameters according to the dimensionality requirements of the cost volume. The loss function can be well integrated with the feature extraction and matching process, and has strong robustness and accuracy.

[0087] The present invention also discloses a cost volume-based heterogeneous image registration device, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the above-mentioned cost volume-based heterogeneous image registration method is implemented.

[0088] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0089] The device can be a computing device such as a desktop computer, laptop, PDA, radar, or cloud server. The device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the device may include more or fewer components, or a combination of certain components, or different components, such as input / output devices, network access devices, etc.

[0090] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0091] In some embodiments, the memory may be an internal storage unit of the extraction device, such as a hard disk or memory of the extraction device. In other embodiments, the memory may also be an external storage device of the extraction device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. equipped on the extraction device. Furthermore, the memory may also include both an internal storage unit of the extraction device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory may also be used to temporarily store data that has been output or is to be output.

Claims

1. A cost volume-based heterogeneous image registration method, characterized in that: The following steps are involved: Obtain a reference image and a template image; Extracting a first feature map of the reference image and a second feature map of the template image respectively through a U-Net network; Performing an affine transformation on the second feature map based on L groups of predetermined rotation scale parameters to obtain L third feature maps corresponding to the second feature map; where L is a positive integer; Performing Fourier convolution using the third feature map as a three-dimensional convolution kernel and the first feature map as a three-dimensional input signal to obtain L phase correlation maps; L phase correlation maps are cascaded in a predetermined channel dimension to generate a cost volume; a maximum matching probability in the cost volume is calculated; a pixel on the cost volume is determined based on the maximum matching probability; a corresponding phase correlation map is determined based on the pixel; and a corresponding predetermined rotation scale parameter is selected based on the phase correlation map to obtain a rotation parameter and a scale parameter.

2. The method for heterogeneous image registration based on cost volume according to claim 1, wherein: Extracting the translation parameter corresponding to the pixel includes: The coordinate value of the pixel in the corresponding phase correlation diagram is obtained, and the coordinate value is used as the translation parameter.

3. The cost volume-based heterogeneous image registration method according to claim 2, wherein: The U-Net network consists of three double convolutional layers, two maximum pooling layers and one 1×1 convolutional layer; The dual convolutional layer includes two convolutional layers, two batch normalization layers and two ReLU layers.

4. The method for heterogeneous image registration based on cost volume according to claim 3, wherein: The method for generating the predetermined rotation scale parameter is: generating a plurality of predetermined rotation parameters according to a rotation parameter threshold, and generating a plurality of predetermined scale parameters according to a scale parameter threshold; The predetermined rotation scale parameter is obtained by combining any of the predetermined rotation parameters and the predetermined scale parameter.

5. The method for heterogeneous image registration based on cost volume according to claim 4, wherein: The loss function of the U-Net network during training is: ， in, is the loss function, is a set of the true values of the rotation parameters, scale parameters, and translation parameters of the template image during training. is the set of rotation parameters, scale parameters and translation parameters corresponding to the j-th phase correlation graph during training, .

6. The method for heterogeneous image registration based on cost volume according to claim 4 or 5, characterized in that: Performing an affine transformation on the second feature map based on L groups of predetermined rotation scale parameters includes: ， in, is a pixel in the second feature map, is the third feature map corresponding to The corresponding pixels, is the scale parameter in the i-th set of predetermined rotation scale parameters, is the rotation parameter in the i-th set of predetermined rotation scale parameters, .

7. A cost volume-based heterogeneous image registration device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the cost volume-based heterogeneous image registration method according to any one of claims 1 to 6 is implemented.

Images