Multi-modal image registration method, system and computer device based on deformation adaptation

Abstract

The application discloses a kind of based on deformation adaptation multi-modal image registration method, system and computer equipment, comprising: acquisition multiple multi-modal images and carries out gray standardization, constructs diversified registration dataset;Build registration network model including pyramid coding module, deformation self-adapting module, cross-modal interaction module and registration parameter estimation module;Image pair is sequentially input into the above module, respectively extract basic feature mapping, deformation feature mapping and interactive enhanced feature mapping, finally output estimated conversion parameter matrix;Through preset loss function supervision training process, select optimal network parameter, obtain trained registration model;When practical application, the image pair to be registered is input into trained model, obtains conversion parameter matrix and completes image registration.Can effectively improve multi-modal image registration performance, especially in the case of severe geometric distortion and significant modal difference, still have good robustness and adaptability.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a deformation-adaptive multimodal image registration method, system, and computer device. Background Technology

[0002] The rapid development of multi-source imaging devices has led to the widespread application of optical, infrared, radar, and other novel sensors in remote sensing mapping, medical imaging, autonomous driving, and industrial inspection. Multimodal images acquired through different imaging mechanisms can present complementary structural and spectral information, providing a more comprehensive basis for environmental perception and data analysis. However, these images exhibit significant differences in brightness distribution, texture details, and spatial geometry, directly resulting in accurate registration of cross-modal images remaining a long-standing challenge.

[0003] Traditional registration methods typically determine spatial transformation relationships based on feature point matching or grayscale similarity. However, when faced with significant differences in brightness and structure between modalities, they often encounter problems such as difficulty in feature mapping and sensitivity to noise, limiting overall accuracy and robustness. Although deep learning methods have improved this problem to some extent, existing networks mostly focus on overall rigid or simple non-rigid transformations, and are still insufficient for adapting to severe geometric distortions.

[0004] To overcome the above challenges, this invention proposes a deformation-adaptive multimodal image registration method, which can effectively improve the performance of multimodal image registration, and still has good robustness and adaptability, especially in the presence of severe geometric distortion and significant modal differences. Summary of the Invention

[0005] To address the above technical problems, this invention provides a multimodal image registration method, system, and computer device based on deformation adaptation.

[0006] The technical solution adopted by this invention to solve its technical problem is:

[0007] A deformation-adaptive multimodal image registration method is described, comprising the following steps:

[0008] S1. Collect multiple sets of multimodal images and perform grayscale standardization. Divide the standardized images into reference images and images to be registered, and combine them into multiple image pairs to construct a registration dataset with diversity.

[0009] S2. Construct a multimodal image registration neural network model based on deformation adaptation. The model includes a pyramid encoding module, a deformation adaptation module, a cross-modal interaction module, and a registration parameter estimation module connected in sequence.

[0010] S3. Input the multimodal images in the dataset into the pyramid encoding module to obtain the basic feature mapping of the image pair, then input them into the deformation adaptation module to obtain the deformation feature mapping of the image pair, then input them into the cross-modal interaction module to generate the interaction enhancement feature mapping of the image pair, and finally input the interaction enhancement feature mapping of the image pair into the registration parameter estimation module to output the estimated transformation parameter matrix of the corresponding image pair.

[0011] S4. Based on the basic feature mapping and estimated transformation parameter matrix of the image pairs, the total loss function value of each image pair is calculated in combination with the preset loss function. The training process of the multimodal image registration neural network model is supervised. The network parameters with the smallest loss value are selected to update the multimodal image registration neural network model, and the trained multimodal image registration neural network model is obtained.

[0012] S5. Obtain the multimodal images to be registered in the actual scene and form a pair of images to be registered. Use the trained multimodal image registration neural network model to process the pair of multimodal images to be registered to obtain the image transformation parameter matrix. Based on the image transformation parameter matrix, transform the images to be registered in the image pair to obtain the registered images.

[0013] Preferably, in S1, multiple sets of multimodal images are acquired and grayscale normalized. The grayscale normalization process can be specifically expressed by the formula:

[0014] (1)

[0015] In the formula, and These represent the original image and the standardized image at different positions, respectively. pixel values, and These represent the maximum and minimum pixel values ​​in the original image, respectively.

[0016] Preferably, the pyramid encoding module includes a starting convolutional module and a three-level residual module connected in sequence. The starting convolutional module includes a convolutional layer, a normalization layer, and an activation function layer connected in sequence. The stride of the convolutional layer is 1, and the kernel size is 3. 3; The residual module consists of a first convolutional layer and a second convolutional layer connected in sequence. The stride of the first convolutional layer is 2, and the kernel size is 3. 3. The stride of the second convolutional layer is 1, and the kernel size is 3. 3, Interlayer passage 1 1. Convolution adjusts the dimensions to achieve residual connections; in S3, a pair of multimodal images from the dataset are sequentially input into the pyramid encoding module to obtain the basic feature mapping of the image pair, specifically:

[0017] (2)

[0018] In the formula, This indicates the pyramid coding module. and These represent the reference image and the image to be registered in the selected image pair, respectively. and These represent the basic feature maps of the reference image and the image to be registered in the selected image pair, respectively.

[0019] Preferably, the deformation adaptive module includes a deformable convolutional layer, a similarity calculation layer, a normalization layer, and a standard deviation layer connected in sequence, wherein the stride of the deformable convolutional layer is 1 and the kernel size is 3. 3; The similarity calculation layer calculates the cosine similarity between the center pixel and its neighboring pixels; the basic feature mapping of the image pair in S3 is then input into the deformation adaptation module to obtain the deformation feature mapping of the image pair, specifically:

[0020] (3)

[0021] In the formula, This indicates a deformation adaptive module. and These represent the basic feature maps of the reference image and the image to be registered, respectively. and These represent the deformation adaptation feature mappings of the reference image and the image to be registered in the selected image pair, respectively.

[0022] Preferably, the cross-modal interaction module includes a feature fusion layer, an attention weight calculation layer, and a dual-branch output convolutional layer connected in sequence, wherein the dual-branch output convolutional layer consists of two independent convolutional layers with a stride of 1 and a kernel size of 1. 1; In S3, the deformation feature mapping of the image pair is input into the cross-modal interaction module to generate the interaction enhancement feature mapping of the image pair, specifically:

[0023] (4)

[0024] In the formula, This indicates a cross-modal interaction module. and These represent the deformation adaptation feature mappings of the reference image and the image to be registered in the selected image pair, respectively. and These represent the interactive enhancement feature maps of the reference image and the image to be registered in the selected image pair, respectively.

[0025] Preferably, the registration parameter estimation module includes a neighborhood cost convolutional layer, a first convolutional layer, a second convolutional layer, a normalization layer, an activation function layer, a max pooling layer, and a third convolutional layer connected in sequence, wherein the stride of the first and third convolutional layers is 1 and the kernel size is 1. 1. The stride of the second convolutional layer is 1, and the kernel size is 3. 3; In S3, the interactive enhancement feature mapping of the image pair is fed into the registration parameter estimation module to output the transformation parameter matrix of the corresponding image pair, specifically:

[0026] (5)

[0027] In the formula, The image transformation parameter matrix represents the image pair. This indicates the registration parameter estimation module. and These represent the interactive enhancement feature maps of the reference image and the image to be registered in the selected image pair, respectively.

[0028] Preferably, the loss function in S4 includes a consistency loss function and a registration ground truth loss function. The consistency loss function value between the base feature map of the reference image and the base feature map of the image to be registered, and the registration ground truth loss function value between the estimated transformation parameter matrix and the ground truth transformation parameter matrix are calculated sequentially. Specifically, this can be expressed by the following formula:

[0029] (6)

[0030] (7)

[0031] (8)

[0032] In the formula, This represents the calculated value of the total loss function. This represents the calculated value of the consistency loss function. This represents the calculated value of the registration true loss function. and These are the weighting coefficients. This represents the image conversion process. The image transformation parameter matrix represents the image pair. Describing the Euclidean norm, and These represent the basic feature maps of the reference image and the image to be registered, respectively. Denotes the 1-norm. and These are the estimated transformation parameter matrix and the true transformation parameter matrix, respectively.

[0033] Preferably, in S4, the images to be registered in the image pair are transformed according to the image transformation parameter matrix, specifically as follows:

[0034] (9)

[0035] In the formula, This represents the image after registration of the image to be registered. This represents the image to be registered. This represents the image conversion process. The matrix represents the image transformation parameters of the image pair.

[0036] A deformation-adaptive multimodal image registration system includes a dataset determination module, a multimodal image registration neural network model building module, a module for determining the estimated transformation parameter matrix of the selected image pairs, a model training module, and an image registration module.

[0037] The dataset determination module is used to collect multiple sets of multimodal images and perform grayscale standardization. The standardized images are divided into reference images and images to be registered and combined into multiple image pairs to construct a registration dataset with diversity.

[0038] A multimodal image registration neural network model building module is used to build a deformation-adaptive multimodal image registration neural network model. The model includes a pyramid encoding module, a deformation adaptation module, a cross-modal interaction module, and a registration parameter estimation module connected in sequence.

[0039] The module for determining the estimated transformation parameter matrix of the selected image pair is used to input a pair of multimodal images in the dataset into the pyramid encoding module to obtain the basic feature mapping of the image pair, then input it into the deformation adaptation module to obtain the deformation feature mapping of the image pair, then input it into the cross-modal interaction module to generate the interaction enhancement feature mapping of the image pair, and finally input the interaction enhancement feature mapping of the image pair into the registration parameter estimation module to output the transformation parameter matrix of the corresponding image pair.

[0040] The model training module is used to calculate the total loss function value of each image pair based on the basic feature mapping and estimated transformation parameter matrix of the image pair, combined with the preset loss function, to supervise the training process of the multimodal image registration neural network model, select the network parameters with the smallest loss value to update the multimodal image registration neural network model, and obtain the trained multimodal image registration neural network model.

[0041] The image registration module is used to process the multimodal image to be registered using a trained multimodal image registration neural network model, obtain the image transformation parameter matrix, and transform the image to be registered in the image pair according to the image transformation parameter matrix to obtain the registered image.

[0042] A computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a deformation-adaptive multimodal image registration method.

[0043] The aforementioned deformation-adaptive multimodal image registration method and system acquires multiple sets of multimodal images and performs grayscale normalization. The normalized images are then divided into reference images and images to be registered, and combined into multiple image pairs to construct a diverse registration dataset. A deformation-adaptive multimodal image registration neural network model is built, and a loss function is designed to supervise the training process of the model. The trained model is used to process the images to be registered, obtaining an image transformation parameter matrix. Based on this matrix, the images to be registered in the image pairs are transformed to obtain the registered images. This method effectively improves multimodal image registration performance, exhibiting good robustness and adaptability, especially in cases of severe geometric distortion and significant modal differences. Attached Figure Description

[0044] Figure 1 This is a flowchart of a multimodal image registration method based on deformation adaptation in one embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the multimodal image registration process in a real-world scenario according to one embodiment of the present invention. Detailed Implementation

[0046] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0047] In one embodiment, such as Figure 1 As shown, a deformation-adaptive multimodal image registration method includes the following steps:

[0048] S1. Collect multiple sets of multimodal images and perform grayscale standardization. Divide the standardized images into reference images and images to be registered, and combine them into multiple image pairs to construct a registration dataset with diversity.

[0049] S2. Construct a multimodal image registration neural network model based on deformation adaptation. The model includes a pyramid encoding module, a deformation adaptation module, a cross-modal interaction module, and a registration parameter estimation module connected in sequence.

[0050] S3. Input the multimodal images in the dataset into the pyramid encoding module to obtain the basic feature mapping of the image pair, then input them into the deformation adaptation module to obtain the deformation feature mapping of the image pair, then input them into the cross-modal interaction module to generate the interaction enhancement feature mapping of the image pair, and finally input the interaction enhancement feature mapping of the image pair into the registration parameter estimation module to output the transformation parameter matrix of the corresponding image pair.

[0051] S4. Based on the basic feature mapping and estimated transformation parameter matrix of the image pairs, the total loss function value of each image pair is calculated in combination with the preset loss function. The training process of the multimodal image registration neural network model is supervised. The network parameters with the smallest loss value are selected to update the multimodal image registration neural network model, and the trained multimodal image registration neural network model is obtained.

[0052] S5. Obtain the multimodal images to be registered in the actual scene and form a pair of images to be registered. Use the trained multimodal image registration neural network model to process the pair of multimodal images to be registered to obtain the image transformation parameter matrix. Based on the image transformation parameter matrix, transform the images to be registered in the image pair to obtain the registered images.

[0053] In one embodiment, S1 acquires multiple sets of multimodal images and performs grayscale normalization. The grayscale normalization process can be specifically expressed by the formula:

[0054] (1)

[0055] In the formula, and These represent the original image and the standardized image at different positions, respectively. pixel values, and These represent the maximum and minimum pixel values ​​in the original image, respectively.

[0056] Specifically, grayscale normalization is used to map image pixel values ​​to a uniform range through linear transformation, eliminating interference caused by differences in intensity distribution between different modalities and improving data consistency.

[0057] In one embodiment, the pyramid encoding module includes a pre-convolutional module and a three-level residual module connected in sequence. The pre-convolutional module includes a convolutional layer, a normalization layer, and an activation function layer connected in sequence. The stride of the convolutional layer is 1, and the kernel size is 3. 3; The residual module consists of a first convolutional layer and a second convolutional layer connected in sequence. The stride of the first convolutional layer is 2, and the kernel size is 3. 3. The stride of the second convolutional layer is 1, and the kernel size is 3. 3, Interlayer passage 1 1. Convolution adjusts the dimensions to achieve residual connections; in S3, a pair of multimodal images from the dataset are sequentially input into the pyramid encoding module to obtain the basic feature mapping of the image pair, specifically:

[0058] (2)

[0059] In the formula, This indicates the pyramid coding module. and These represent the reference image and the image to be registered in the selected image pair, respectively. and These represent the basic feature maps of the reference image and the image to be registered in the selected image pair, respectively.

[0060] Specifically, the pyramid encoding module is used to extract key features from the input image step by step, providing high-quality feature representations for subsequent deformation adaptation, cross-modal interaction, and registration parameter estimation. The convolutional layer captures the basic structural information of the image by reducing the spatial dimension and increasing the number of feature channels. The normalization layer is used to stabilize the feature distribution, accelerate model convergence, and improve generalization ability. The activation function layer introduces nonlinear transformations to enhance the network's ability to model complex mapping relationships. The residual module is used to avoid gradient vanishing and gradient exploding problems during training through residual connections, while enhancing the expressive power of feature mapping.

[0061] In one embodiment, the deformation adaptive module includes a deformable convolutional layer, a similarity calculation layer, a normalization layer, and a standard deviation layer connected in sequence, wherein the stride of the deformable convolutional layer is 1 and the kernel size is 3. 3; The similarity calculation layer calculates the cosine similarity between the center pixel and its neighboring pixels; the basic feature mapping of the image pair in S3 is then input into the deformation adaptation module to obtain the deformation feature mapping of the image pair, specifically:

[0062] (3)

[0063] In the formula, This indicates a deformation adaptive module. and These represent the basic feature maps of the reference image and the image to be registered, respectively. and These represent the deformation adaptation feature mappings of the reference image and the image to be registered in the selected image pair, respectively.

[0064] Specifically, the deformation adaptation module is used to model and adapt to complex deformations in the input image, enabling the subsequent registration process to better handle geometric distortions between images of different modalities. The deformable convolutional layer introduces a learnable offset on the basis of conventional convolution, adaptively adjusting the convolution sampling position to flexibly perceive local geometric changes. The similarity calculation layer uses the center pixel as a reference to calculate its cosine similarity with neighboring pixels, which is used to measure the structural consistency of local regions and enhance effective features. The normalization layer standardizes the feature distribution, accelerates network convergence and improves overall stability. The standard deviation layer further evaluates the magnitude of feature changes in local regions, provides quantitative information on the degree of deformation, and enhances the model's adaptability to geometric distortions.

[0065] In one embodiment, the cross-modal interaction module includes a feature fusion layer, an attention weight calculation layer, and a dual-branch output convolutional layer connected in sequence, wherein the dual-branch output convolutional layer consists of two independent convolutional layers with a stride of 1 and a kernel size of 1. 1; In S3, the deformation feature mapping of the image pair is input into the cross-modal interaction module to generate the interaction enhancement feature mapping of the image pair, specifically:

[0066] (4)

[0067] In the formula, This indicates a cross-modal interaction module. and These represent the deformation adaptation feature mappings of the reference image and the image to be registered in the selected image pair, respectively. and These represent the interactive enhancement feature maps of the reference image and the image to be registered in the selected image pair, respectively.

[0068] Specifically, the cross-modal interaction module is used to fully integrate the key information of the reference image and the image to be registered, enhance the correlation and complementarity between different modal features, and provide more discriminative feature maps for the final registration parameter estimation. The feature fusion layer is used to concatenate the deformation adaptation feature maps of the reference image and the image to be registered in the channel dimension, and achieve the initial fusion of cross-modal information through convolution. The attention weight calculation layer adaptively allocates the weights of different modal features through a joint mechanism of channel attention and spatial attention. The dual-branch output convolutional layer generates the interactive enhancement feature maps corresponding to the reference image and the image to be registered, respectively.

[0069] In one embodiment, the registration parameter estimation module includes a neighborhood cost convolutional layer, a first convolutional layer, a second convolutional layer, a normalization layer, an activation function layer, a max pooling layer, and a third convolutional layer connected in sequence, wherein the stride of the first and third convolutional layers is 1 and the kernel size is 1. 1. The stride of the second convolutional layer is 1, and the kernel size is 3. 3; In S3, the interactive enhancement feature mapping of the image pair is fed into the registration parameter estimation module to output the transformation parameter matrix of the corresponding image pair, specifically:

[0070] (5)

[0071] In the formula, The image transformation parameter matrix represents the image pair. This indicates the registration parameter estimation module. and These represent the interactive enhancement feature maps of the reference image and the image to be registered in the selected image pair, respectively.

[0072] Specifically, the registration parameter estimation module is used to estimate the registration parameters between multimodal images based on the extracted interactive enhancement feature maps, thereby achieving accurate image alignment. The neighborhood cost convolution layer is used to calculate the local similarity of the feature maps and generate the neighborhood cost convolution. The first convolutional layer is used to adjust the feature channel dimension, reducing the computational cost while extracting key local features. The second convolutional layer is used to further capture the spatial relationships within a larger receptive field and enrich the feature representation. The normalization layer is used to accelerate the training process and improve the model's generalization ability. The activation function layer introduces nonlinear transformation. The max pooling layer is used to reduce the spatial dimension of the feature maps while retaining the most important feature information. The third convolutional layer is used to refine the feature maps and generate the transformation parameter matrix required for image registration.

[0073] In one embodiment, the loss function in S4 includes a consistency loss function and a registration ground truth loss function. The consistency loss function value between the base feature map of the reference image and the base feature map of the image to be registered, and the registration ground truth loss function value between the estimated transformation parameter matrix and the ground truth transformation parameter matrix are calculated sequentially. This can be specifically expressed by the following formula:

[0074] (6)

[0075] (7)

[0076] (8)

[0077] In the formula, This represents the calculated value of the total loss function. This represents the calculated value of the consistency loss function. This represents the calculated value of the registration true loss function. and These are the weighting coefficients. This represents the image conversion process. The image transformation parameter matrix represents the image pair. Denotes the Euclidean norm. and These represent the basic feature maps of the reference image and the image to be registered, respectively. Denotes the 1-norm. and These are the estimated transformation parameter matrix and the true transformation parameter matrix, respectively.

[0078] Specifically, the consistency loss function is used to constrain the consistency between the image to be registered and the reference image in the feature space, suppressing feature heterogeneity caused by modal differences and prompting the network to learn modality-invariant shared feature representations. The registration ground truth loss is used to directly evaluate the difference between the estimated transformation parameters and the true transformation parameters, guiding the model to optimize the accuracy of the registration parameters. The two types of losses work together to provide optimization objectives at both the feature alignment and parameter regression levels, effectively improving the registration accuracy and robustness of the model in multimodal scenarios.

[0079] When training a multimodal image registration neural network model, for each pair of images, the model generates image transformation parameters. And calculate the corresponding loss value. A smaller loss value indicates a better alignment between the transformed image to be matched and the reference image. The network weights are updated by selecting the network parameters corresponding to the minimum loss value, gradually optimizing the network performance. After training, the resulting multimodal image registration neural network model can accurately estimate the transformation parameter matrix of the image, thus achieving high-precision multimodal image registration.

[0080] In one embodiment, step S5 involves converting the images to be registered in the image pair according to the image conversion parameter matrix, specifically as follows:

[0081] (9)

[0082] In the formula, This represents the image after registration of the image to be registered. This represents the image to be registered. This represents the image conversion process. The matrix represents the image transformation parameters of the image pair.

[0083] Specifically, see Figure 2 , Figure 2 This is a schematic diagram of the multimodal image registration process in a real-world scenario according to one embodiment of the present invention.

[0084] Acquire multimodal images to be registered in a real-world scenario and assemble them into image pairs. and By processing the input image pairs to be registered into a deformation-adaptive multimodal image registration neural network model, the image transformation parameter matrix corresponding to the image pairs to be registered can be obtained. Specifically, referring to formula (2), the reference image in the image pair to be registered in the actual scene is used. and the image to be registered A general feature map can be obtained accordingly. and Then, referring to formula (3), the deformation adaptation feature mapping of the reference image and the image to be registered is obtained. and Then, referring to formula (4), we obtain the interactive enhancement feature mapping between the reference image and the image to be registered. and Then, referring to formula (5), the image transformation parameter matrix of the image pair to be registered is obtained. Then, according to formula (9), the registered image of the image to be registered in the actual scene is obtained.

[0085] A deformation-adaptive multimodal image registration method is proposed. This method involves acquiring and preprocessing several multimodal image datasets to obtain a diverse registration dataset. A deformation-adaptive multimodal image registration neural network model is then constructed. A dedicated loss function for multimodal image registration is designed, and the neural network model is trained under supervision to optimize the overall registration process. The fully trained model is then used to register the multimodal image pairs to be registered, outputting high-precision registration results. This method effectively improves multimodal image registration performance, exhibiting good robustness and adaptability, especially in cases with severe geometric distortion and significant modal differences.

[0086] In one embodiment, a deformation-adaptive multimodal image registration system is also provided, including a dataset determination module, a multimodal image registration neural network model building module, a module for determining the estimated transformation parameter matrix of the selected image pairs, a model training module, and an image registration module;

[0087] The dataset determination module is used to collect multiple sets of multimodal images and perform grayscale standardization. The standardized images are divided into reference images and images to be registered and combined into multiple image pairs to construct a registration dataset with diversity.

[0088] A multimodal image registration neural network model building module is used to build a deformation-adaptive multimodal image registration neural network model. The model includes a pyramid encoding module, a deformation adaptation module, a cross-modal interaction module, and a registration parameter estimation module connected in sequence.

[0089] The module for determining the estimated transformation parameter matrix of the selected image pair is used to input a pair of multimodal images in the dataset into the pyramid encoding module to obtain the basic feature mapping of the image pair, then input it into the deformation adaptation module to obtain the deformation feature mapping of the image pair, then input it into the cross-modal interaction module to generate the interaction enhancement feature mapping of the image pair, and finally input the interaction enhancement feature mapping of the image pair into the registration parameter estimation module to output the transformation parameter matrix of the corresponding image pair.

[0090] The model training module is used to calculate the total loss function value of each image pair based on the basic feature mapping and estimated transformation parameter matrix of the image pair, combined with the preset loss function, to supervise the training process of the multimodal image registration neural network model, select the network parameters with the smallest loss value to update the multimodal image registration neural network model, and obtain the trained multimodal image registration neural network model.

[0091] The image registration module is used to process the multimodal image to be registered using a trained multimodal image registration neural network model, obtain the image transformation parameter matrix, and transform the image to be registered in the image pair according to the image transformation parameter matrix to obtain the registered image.

[0092] For specific limitations regarding a deformation-adaptive multimodal image registration system, please refer to the limitations of a deformation-adaptive multimodal image registration method described above, which will not be repeated here. Each module in the aforementioned deformation-adaptive multimodal image registration system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0093] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a deformation-adaptive multimodal image registration method.

[0094] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0095] The above provides a detailed description of the deformation-adaptive multimodal image registration method, system, and computer device provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention; the descriptions of these embodiments are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A multimodal image registration method based on deformation adaptation, characterized in that, The method includes the following steps: S1. Collect multiple sets of multimodal images and perform grayscale standardization. Divide the standardized images into reference images and images to be registered, and combine them into multiple image pairs to construct a registration dataset with diversity. S2. Construct a multimodal image registration neural network model based on deformation adaptation. The model includes a pyramid encoding module, a deformation adaptation module, a cross-modal interaction module, and a registration parameter estimation module connected in sequence. S3. Input the multimodal images in the dataset into the pyramid encoding module to obtain the basic feature mapping of the image pair, then input them into the deformation adaptation module to obtain the deformation feature mapping of the image pair, then input them into the cross-modal interaction module to generate the interaction enhancement feature mapping of the image pair, and finally input the interaction enhancement feature mapping of the image pair into the registration parameter estimation module to output the estimated transformation parameter matrix of the corresponding image pair. The deformation adaptation module includes a deformable convolutional layer, a similarity calculation layer, a normalization layer and a standard deviation layer connected in sequence, wherein the stride of the deformable convolutional layer is 1 and the kernel size is 3. 3; The similarity calculation layer calculates the cosine similarity between the center pixel and its neighboring pixels; the basic feature mapping of the image pair in S3 is then input into the deformation adaptation module to obtain the deformation feature mapping of the image pair, specifically: （3） In the formula, This indicates a deformation adaptive module. and These represent the basic feature maps of the reference image and the image to be registered, respectively. and These represent the deformation adaptation feature mappings of the reference image and the image to be registered in the selected image pair, respectively; The cross-modal interaction module consists of a feature fusion layer, an attention weight calculation layer, and a dual-branch output convolutional layer connected in sequence. The dual-branch output convolutional layer is composed of two independent convolutional layers with a stride of 1 and a kernel size of 1. 1; In S3, the deformation feature mapping of the image pair is input into the cross-modal interaction module to generate the interaction enhancement feature mapping of the image pair, specifically: （4） In the formula, This indicates a cross-modal interaction module. and These represent the deformation adaptation feature mappings of the reference image and the image to be registered in the selected image pair, respectively. and These represent the interactive enhancement feature maps of the reference image and the image to be registered in the selected image pair, respectively; S4. Based on the basic feature mapping and estimated transformation parameter matrix of the image pairs, the total loss function value of each image pair is calculated in combination with the preset loss function. The training process of the multimodal image registration neural network model is supervised. The network parameters with the smallest loss value are selected to update the multimodal image registration neural network model, and the trained multimodal image registration neural network model is obtained. S5. Obtain the multimodal images to be registered in the actual scene and form a pair of images to be registered. Use the trained multimodal image registration neural network model to process the pair of multimodal images to be registered to obtain the image transformation parameter matrix. Based on the image transformation parameter matrix, transform the images to be registered in the image pair to obtain the registered images.

2. The multimodal image registration method based on deformation adaptation as described in claim 1, characterized in that, S1 acquires multiple sets of multimodal images and performs grayscale standardization. The specific grayscale standardization process can be expressed by the formula: （1） In the formula, and These represent the original image and the standardized image at different positions, respectively. pixel values, and These represent the maximum and minimum pixel values ​​in the original image, respectively.

3. The multimodal image registration method based on deformation adaptation as described in claim 2, characterized in that, The pyramid encoding module consists of a pre-convolutional module and a three-level residual module connected in sequence. The pre-convolutional module includes a convolutional layer, a normalization layer, and an activation function layer connected in sequence. The stride of the convolutional layer is 1, and the kernel size is 3. 3; The residual module consists of a first convolutional layer and a second convolutional layer connected in sequence. The stride of the first convolutional layer is 2, and the kernel size is 3.

3. The stride of the second convolutional layer is 1, and the kernel size is 3. 3, Interlayer passage 1 1. Convolution adjusts the dimensions to achieve residual connections; in S3, a pair of multimodal images from the dataset are sequentially input into the pyramid encoding module to obtain the basic feature mapping of the image pair, specifically: （2） In the formula, This indicates the pyramid coding module. and These represent the reference image and the image to be registered in the selected image pair, respectively. and These represent the basic feature maps of the reference image and the image to be registered in the selected image pair, respectively.

4. The multimodal image registration method based on deformation adaptation as described in claim 3, characterized in that, The registration parameter estimation module consists of a neighborhood cost convolutional layer, a first convolutional layer, a second convolutional layer, a normalization layer, an activation function layer, a max pooling layer, and a third convolutional layer connected in sequence. The stride of the first and third convolutional layers is 1, and the kernel size is 1.

1. The stride of the second convolutional layer is 1, and the kernel size is 3. 3; In S3, the interactive enhancement feature mapping of the image pair is fed into the registration parameter estimation module to output the transformation parameter matrix of the corresponding image pair, specifically: （5） In the formula, The image transformation parameter matrix represents the image pair. This indicates the registration parameter estimation module. and These represent the interactive enhancement feature maps of the reference image and the image to be registered in the selected image pair, respectively.

5. The multimodal image registration method based on deformation adaptation as described in claim 4, characterized in that, In S4, the loss functions include the consistency loss function and the registration ground truth loss function. The consistency loss function value between the base feature map of the reference image and the base feature map of the image to be registered, and the registration ground truth loss function value between the estimated transformation parameter matrix and the ground truth transformation parameter matrix are calculated sequentially. Specifically, this can be expressed by the following formula: （6） （7） （8） In the formula, This represents the calculated value of the total loss function. This represents the calculated value of the consistency loss function. This represents the calculated value of the registration true loss function. and These are the weighting coefficients. This represents the image conversion process. The image transformation parameter matrix represents the image pair. Denotes the Euclidean norm. and These represent the basic feature maps of the reference image and the image to be registered, respectively. Denotes the 1-norm. and These are the estimated transformation parameter matrix and the true transformation parameter matrix, respectively.

6. The multimodal image registration method based on deformation adaptation as described in claim 5, characterized in that, In S5, the images to be registered in the image pair are transformed according to the image transformation parameter matrix, specifically as follows: （9） In the formula, This represents the image after registration of the image to be registered. This represents the image to be registered. This represents the image conversion process. The matrix represents the image transformation parameters of the image pair.

7. A deformation-adaptive multimodal image registration system that performs the method described in any one of claims 1-6, characterized in that, It includes a dataset determination module, a multimodal image registration neural network model building module, a module for determining the estimated transformation parameter matrix of the selected image pairs, a model training module, and an image registration module; The dataset determination module is used to collect multiple sets of multimodal images and perform grayscale standardization. The standardized images are divided into reference images and images to be registered and combined into multiple image pairs to construct a registration dataset with diversity. A multimodal image registration neural network model building module is used to build a deformation-adaptive multimodal image registration neural network model. The model includes a pyramid encoding module, a deformation adaptation module, a cross-modal interaction module, and a registration parameter estimation module connected in sequence. The module for determining the estimated transformation parameter matrix of the selected image pair is used to input a pair of multimodal images in the dataset into the pyramid encoding module to obtain the basic feature mapping of the image pair, then input it into the deformation adaptation module to obtain the deformation feature mapping of the image pair, then input it into the cross-modal interaction module to generate the interaction enhancement feature mapping of the image pair, and finally input the interaction enhancement feature mapping of the image pair into the registration parameter estimation module to output the transformation parameter matrix of the corresponding image pair. The model training module is used to calculate the total loss function value of each image pair based on the basic feature mapping and estimated transformation parameter matrix of the image pair, combined with the preset loss function, to supervise the training process of the multimodal image registration neural network model, select the network parameters with the smallest loss value to update the multimodal image registration neural network model, and obtain the trained multimodal image registration neural network model. The image registration module is used to process the multimodal image to be registered using a trained multimodal image registration neural network model, obtain the image transformation parameter matrix, and transform the image to be registered in the image pair according to the image transformation parameter matrix to obtain the registered image.

8. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images