Method and apparatus for multi-modal registration based on medical images

Abstract

The present application relates to a medical image-based multi-modal registration method and device, the method comprising: obtaining a brain registration dataset and dividing it into a training set and a test set. A registration network is constructed based on a UNet network structure. An input image is selected and input into a dual architecture. The input image is subjected to one-way registration in both directions to obtain forward and reverse deformation fields and a one-way registration image. The deformation fields of the two dual parties are exchanged and applied to the one-way registration image for secondary registration to obtain a secondary registration image. A Laplace operator is called to extract structural features from the input image, the one-way registration image and the secondary registration image, respectively, and the registration network is constrained by structural similarity measurement. An end-to-end unsupervised image registration algorithm is trained by the training set and the test set, and is optimized according to the loss constraint. The registration image to be detected is input into the registration network, the optimal model weight is applied, and the registration result is output to ensure the uniformity and accuracy of the result.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and medical imaging technology, and in particular to a multimodal registration method and device based on medical images. Background Art

[0002] Two-dimensional deformable image registration has been an important research hotspot in the field of medical imaging and is crucial for preoperative planning, intraoperative guidance, disease diagnosis, and follow-up. Scene acquisition using different sensors is a common practice across various disciplines, from classic disciplines such as medical imaging and remote sensing to emerging tasks such as autonomous driving. Multimodal sensors allow for the collection of a wide range of physical properties, which in turn yields richer scene representations. For example, in radiation planning, multimodal data (e.g., T1-weighted and T2-weighted images in magnetic resonance imaging (MRI) provide different tissue contrasts, with T1 images emphasizing the anatomical boundaries of gray and white matter, while T2 images better highlight high-signal areas of cerebrospinal fluid and lesions) are used for more accurate tumor contours, thereby reducing the risk of damaging healthy tissue during radiotherapy.

[0003] In recent years, deep neural network-based registration techniques have become a powerful standard for large-scale medical image registration. During the training phase, deep registration models use the entire training set as the optimization target to train the network parameters, rather than using a single image pair. After training, these deep registration models typically only take a few seconds to infer the deformation field of an image pair. VoxelMorph is a basic image registration framework, proposed in a 2018 article published by MIT titled "An Unsupervised Learning Model for Deformable Medical Image Registration." It was originally designed for three-dimensional medical image registration problems, but can also be extended to two-dimensional registration scenarios. Its core idea is to directly predict the deformation field in an unsupervised learning manner by constructing a fully convolutional neural network (CNN), apply the deformation field to the input image, and output the registered result.

[0004] However, existing image registration methods often ignore the diffeomorphic properties of the deformation field. In medical image registration, these properties ensure that the geometric structure, differential properties, topological properties, and reversibility of the object are preserved during the registration process. Diffeomorphic transformations preserve the "smoothness" of images or anatomical structures, meaning that there is no noticeable distortion or structural damage during the registration process, thereby ensuring that the registered image accurately reflects the spatial relationships of the original objects. However, ignoring these properties can lead to discontinuities or extreme deformations in the deformation field, which can cause organs or tissues in the image to be unnaturally stretched, compressed, or distorted. In brain image registration, if the deformation field fails to preserve these properties, the boundaries between gray and white matter may be excessively stretched or even extremely compressed in local areas, making it impossible to accurately align corresponding regions in the two images. This results in unnatural shapes of organs and tissues, leading to distorted registration results. Diffeomorphic transformations ensure the reversibility of deformations, meaning that the deformed image can be restored to its original state through the inverse transformation of the deformation field. Furthermore, if the deformation field is not diffeomorphic, the transformation may not be reversible. This means that the registered image cannot be restored to its original geometry, or the deformation is too complex to be effectively reverse mapped. This poses a significant challenge for subsequent image analysis and cross-modality registration (such as T1 and T2 image registration), as the original image structure cannot be accurately reconstructed, further compromising the accuracy of image analysis.

[0005] Furthermore, existing registration algorithms often overlook the crucial step of extracting structural information when applying image constraints, instead directly applying the constraints using the entire image. Each component of a medical image has distinct structural features, and these local differences are particularly pronounced in complex organs like the brain and spine. For example, structures such as the brain's gray matter, white matter, cerebrospinal fluid, and blood vessels often require precise registration using local features. Directly applying global constraints to the entire image ignores local structural information and may overlook small but critical anatomical landmarks or lesions. Furthermore, this can lead to uneven registration results, meaning that some regions perform well while others fail to align accurately. For example, the boundary between white and gray matter in T1 and T2 images exhibits significant contrast differences. Applying constraints to the entire image can lead to less accurate registration of these regions, impacting the overall image alignment quality. Summary of the Invention

[0006] Based on this, it is necessary to provide a multimodal registration method and device based on medical images with high image analysis accuracy and good uniformity of registration results to address the above technical problems.

[0007] The present invention provides a multimodal registration method based on medical images, the method comprising:

[0008] Acquire a brain registration dataset, and divide case images in the brain registration dataset into a training set and a test set;

[0009] Based on the UNet network structure, the convolution step size is used in the encoding layer to halve the spatial dimension to the minimum layer, and upsampling, convolution and connection skipping are alternately used in the decoding stage to construct the registration network;

[0010] Selecting an input image from the brain registration dataset and inputting it into the dual architecture, performing bidirectional primary registration on the input image to obtain forward and reverse deformation fields and a primary registration image, and applying the dual exchange deformation fields to the primary registration image to perform secondary registration to obtain a secondary registration image;

[0011] Calling the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, respectively, and constraining the registration network through a structural similarity measure based on the structural features;

[0012] Training an end-to-end unsupervised image registration algorithm using the training set and the test set to optimize the end-to-end unsupervised image registration algorithm according to a loss constraint to obtain an optimal model weight;

[0013] Taking the image to be detected and registered as the input of the registration network, applying the optimal model weights, and outputting the registration result;

[0014] The brain registration dataset is a BraTS dataset, which includes anatomical imaging and pathological imaging in magnetic resonance imaging; the input image includes a moving image and a fixed image, and the forward and reverse deformation fields are respectively the deformation fields for registering from the moving image to the fixed image and the deformation fields for registering from the fixed image to the moving image.

[0015] In one embodiment, the step of selecting an input image from the brain registration dataset and inputting it into a dual architecture, performing a bidirectional primary registration on the input image to obtain forward and reverse deformation fields and a primary registration image, and applying the dual exchange deformation fields to the primary registration image to perform a secondary registration to obtain a secondary registration image comprises:

[0016] Calling the basic registration method VoxelMorph to perform a primary registration from the moving image to the fixed image, and establishing a network architecture for registration from the fixed image to the moving image that is dual to the primary registration image based on the primary registration image to obtain a deformation field for inverse registration;

[0017] The secondary registration image is obtained by subjecting the primary registration image to a deformation field with upper and lower exchanges through the duality of differential homeomorphism;

[0018] The duality of the differential homeomorphism includes push-forward and pull-back. The push-forward is used to describe the local movement direction of the image and the propagation of the image size in the deformation field. The pull-back is used to map the structural features in the image back to the source image to compare the structural features or similarity measurements.

[0019] In one embodiment, calling the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constraining the registration network through a structural similarity measure based on the structural features, includes:

[0020] The structural similarity between the registered image and the fixed image is calculated based on the structural features, and when the structural similarity reaches a maximum value, a similarity constraint is applied to the registration network. The expression of the similarity constraint is:

[0021] ;

[0022] Where, is the similarity constraint, is the feature image of the structural feature to be compared, It is the cross-correlation representation of the basic registration method VoxelMorph.

[0023] In one embodiment, calling the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constraining the registration network through a structural similarity measure based on the structural features, further includes:

[0024] Obtaining a feature image pixel corresponding to the input image and a set of all pixels of the input image, determining a plurality of adjacent pixels around the feature image pixel, and calculating a cross-correlation of the basic registration method VoxelMorph;

[0025] The cross-correlation expression of the basic registration method VoxelMorph is:

[0026] ;

[0027] Where, is the feature image pixel corresponding to the input image, is the set of all pixels of the input image, are multiple adjacent pixels around the feature image pixel, and are the average pixel intensities of the structural features to be compared.

[0028] In one embodiment, calling the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constraining the registration network through a structural similarity measure based on the structural features, further includes:

[0029] Calculating the smoothing loss of any deformation field of the input image based on the basic registration method VoxelMorph to smooth the deformation field;

[0030] The smoothing loss expression of the deformation field is:

[0031] ;

[0032] Where, is the feature image pixel corresponding to the input image, is the set of all pixels of the input image, is the gradient operator, For any deformation field, Represents the norm.

[0033] In one embodiment, calling the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constraining the registration network through a structural similarity measure based on the structural features, further includes:

[0034] Calling the Laplace operator to calculate pixels in the input image, the first registered image, and the second registered image by grayscale difference, and determining whether the grayscale value of the central pixel of the image is higher than the grayscale values ​​of other pixels around the central pixel;

[0035] When the grayscale value of the central pixel of the image is higher than the grayscale values ​​of other surrounding pixels, the grayscale of the central pixel of the image is increased, and when the grayscale value of the central pixel of the image is lower than the grayscale values ​​of other surrounding pixels, the grayscale of the central pixel of the image is reduced to extract image structural features.

[0036] In one embodiment, the calling of the Laplace operator extracts structural features from the input image, the first registered image, and the second registered image, and constrains the registration network through a structural similarity measure based on the structural features, and then further includes:

[0037] Calling the Laplace operator to calculate the gradient of the central pixel of the neighborhood in multiple directions, and summing the multiple gradients obtained to adjust the grayscale of the central pixel of the image according to the gradient operation result;

[0038] Moving the convolution kernel row by row on the input image, multiplying the value in the convolution kernel by the pixels coinciding with the convolution kernel, summing the values, and assigning the result to the pixel coinciding with the center of the convolution kernel to obtain a Laplacian result;

[0039] The Laplace result is the image feature structure as the input of the similarity constraint loss.

[0040] The present invention also provides a multimodal registration device based on medical images, the device comprising:

[0041] A data preparation module is used to obtain a brain registration dataset and divide the case images in the brain registration dataset into a training set and a test set;

[0042] A registration network building module for a UNet-based network structure that uses convolutional strides in the encoding layer to halve the spatial dimension to the minimum layer, and alternately uses upsampling, convolution, and connection skipping in the decoding stage to construct the registration network;

[0043] an image registration module, configured to select an input image from the brain registration dataset and input it into the dual architecture, perform a bidirectional primary registration on the input image to obtain forward and reverse deformation fields and a primary registered image, and perform a secondary registration on the primary registered image by applying the dual exchange deformation fields to obtain a secondary registered image;

[0044] A feature extraction module, configured to extract structural features from the input image, the first registered image, and the second registered image respectively by calling the Laplace operator, and constrain the registration network based on the structural features by using a structural similarity metric;

[0045] A model optimization module, configured to train an end-to-end unsupervised image registration algorithm using the training set and the test set, so as to optimize the end-to-end unsupervised image registration algorithm according to a loss constraint to obtain an optimal model weight;

[0046] An image registration detection module, configured to take the image to be detected as the input of the registration network, apply the optimal model weights, and output the registration result;

[0047] The brain registration dataset is a BraTS dataset, which includes anatomical imaging and pathological imaging in magnetic resonance imaging; the input image includes a moving image and a fixed image, and the forward and reverse deformation fields are respectively the deformation fields for registering from the moving image to the fixed image and the deformation fields for registering from the fixed image to the moving image.

[0048] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements any of the multimodal registration methods based on medical images described above.

[0049] The present invention also provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the multimodal registration methods based on medical images described above.

[0050] The multimodal registration method and device for medical images obtains a brain registration dataset and divides the case images in the dataset into a training set and a test set. Based on the UNet network architecture, the convolution stride is used in the encoding layer to halve the spatial dimension to the minimum layer. In the decoding stage, upsampling, convolution, and skip connections are alternately used to construct the registration network. Next, an input image from the brain registration dataset is input into the dual architecture. Bidirectional primary registration is performed on the input image, generating forward and reverse deformation fields and a primary registered image. The dual exchanged deformation field is then applied to the primary registered image for secondary registration, resulting in a secondary registered image. The Laplace operator is then used to extract structural features from the input image, the primary registered image, and the secondary registered image. Based on these structural features, the registration network is constrained using a structural similarity metric. The end-to-end unsupervised image registration algorithm is trained using the training and test sets, and optimized according to the loss constraint to obtain the optimal model weights. Finally, the image to be registered is fed into the registration network, and the optimal model weights are applied to output the registration result. This method utilizes a dual framework and a secondary registration based on deformation field exchange to ensure the diffeomorphic nature of the deformation field. The push-forward and pull-back operations of diffeomorphic duality ensure the inherent consistency between the forward and inverse deformation fields during registration, guaranteeing uniformity in the registration results. Furthermore, the Laplace structure operator is added to extract the structural features of the image that require structural information constraints. By applying similarity constraints to these structural features, more accurate registration results can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 A schematic diagram of the process of the multimodal registration method based on medical images provided by the present invention;

[0053] Figure 2 A block diagram of the VoxelMorph algorithm of the multimodal registration method based on medical images in a specific embodiment of the present invention;

[0054] Figure 3 A schematic diagram of the overall process of a multimodal registration method based on medical images in a specific embodiment provided by the present invention;

[0055] Figure 4 A schematic diagram of a registration algorithm flow of a multimodal registration method based on medical images in a specific embodiment of the present invention;

[0056] Figure 5 A schematic structural diagram of a multimodal registration device based on medical images provided by the present invention;

[0057] Figure 6 This is a diagram of the internal structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0059] The following combination Figures 1 to 6 The present invention describes a multimodal registration method and apparatus based on medical images.

[0060] like Figure 1 As shown, in one embodiment, a multimodal registration method based on medical images includes the following steps:

[0061] Step S110 , obtaining a brain registration dataset, and dividing the case images in the brain registration dataset into a training set and a test set.

[0062] Specifically, the server obtains a pre-configured brain registration dataset, which is a BraTS dataset, including anatomical imaging and pathological imaging in magnetic resonance imaging, and divides the case images in the obtained brain registration dataset into corresponding training sets and test sets according to a preset ratio.

[0063] In step S120, based on the UNet network structure, the spatial dimension is halved to the minimum layer using the convolution step in the encoding layer, and upsampling, convolution and connection skipping are alternately used in the decoding stage to construct the registration network.

[0064] Specifically, based on the UNet network structure, the server uses convolution steps in the encoding layer to halve the spatial dimension to the minimum layer, and alternately uses upsampling, convolution and connection jumps in the decoding stage to complete the construction of the registration network.

[0065] In step S130, an input image is selected from the brain registration dataset and input into the dual architecture, and a bidirectional primary registration is performed on the input image to obtain the forward and reverse deformation fields and the primary registration image, and the dual exchange deformation fields are applied to the primary registration image for secondary registration to obtain the secondary registration image.

[0066] The input image includes a moving image and a fixed image, and the forward and reverse deformation fields are respectively the deformation field for registering the moving image to the fixed image and the deformation field for registering the fixed image to the moving image.

[0067] Specifically, the server selects an input image from the brain registration dataset and inputs it into the dual architecture, performs bidirectional primary registration on the input image, obtains forward and reverse deformation fields and a primary registration image, and applies the dual exchange deformation field to the primary registration image for secondary registration to obtain a secondary registration image.

[0068] In some embodiments, the multimodal registration method based on medical images provided by the present invention selects an input image from a brain registration dataset and inputs it into a dual architecture, performs a bidirectional primary registration on the input image, obtains forward and reverse deformation fields and a primary registered image, and applies the dual exchange deformation field to the primary registered image for secondary registration to obtain a secondary registered image, specifically comprising the following steps:

[0069] In step S131, the basic registration method VoxelMorph is called to perform a registration from the moving image to the fixed image, and a network architecture for registration from the fixed image to the moving image is established based on the primary registration image, which is dual to the primary registration image, to obtain a deformation field for inverse registration.

[0070] In step S132 , the primary registration image is subjected to a deformation field with upper and lower exchanges through the duality of the diffeomorphism to obtain a secondary registration image.

[0071] Among them, the duality of differential homeomorphism includes push-forward and pull-back. Push-forward is used to describe the local movement direction of the image and the propagation of the image size in the deformation field, and pull-back is used to map the structural features in the image back to the source image to compare structural features or similarity measurements.

[0072] In step S140 , the Laplace operator is called to extract structural features from the input image, the first registered image, and the second registered image, and the registration network is constrained by a structural similarity metric based on the structural features.

[0073] Specifically, the server calls the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constrains the registration network through a structural similarity measure based on the extracted structural features.

[0074] In some embodiments, the multimodal registration method based on medical images provided by the present invention calls the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constrains the registration network based on the structural features through a structural similarity measure, specifically comprising the following steps:

[0075] In step S141, the structural similarity between the registered image and the fixed image is calculated based on the structural features, and when the structural similarity reaches the maximum value, the registration network is subjected to similarity constraint. The expression of the similarity constraint is:

[0076] ;

[0077] Where, is the similarity constraint, is the feature image of the structural feature to be compared, It is the cross-correlation representation of the basic registration method VoxelMorph.

[0078] In some embodiments, the multimodal registration method based on medical images provided by the present invention calls the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constrains the registration network based on the structural features through a structural similarity measure, and specifically further includes the following steps:

[0079] Step S142 , obtaining the feature image pixel corresponding to the input image and the entire pixel set of the input image, determining a plurality of adjacent pixels around the feature image pixel, and calculating the cross-correlation of the basic registration method VoxelMorph.

[0080] The cross-correlation expression of the basic registration method VoxelMorph is:

[0081] ;

[0082] Where, is the feature image pixel corresponding to the input image, is the set of all pixels of the input image, are multiple adjacent pixels around the feature image pixel, and are the average pixel intensities of the structural features to be compared.

[0083] In some embodiments, the multimodal registration method based on medical images provided by the present invention calls the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constrains the registration network based on the structural features through a structural similarity measure, and specifically further includes the following steps:

[0084] Step S143 : Calculate the smoothing loss of any deformation field of the input image based on the basic registration method VoxelMorph to smooth the deformation field.

[0085] The smoothing loss expression of the deformation field is:

[0086] ;

[0087] Where, is the feature image pixel corresponding to the input image, is the set of all pixels of the input image, is the gradient operator, For any deformation field, Represents the norm.

[0088] In some embodiments, the multimodal registration method based on medical images provided by the present invention calls the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constrains the registration network based on the structural features through a structural similarity measure, and specifically further includes the following steps:

[0089] Step S144 , calling the Laplace operator to calculate the pixels in the input image, the first registered image, and the second registered image by grayscale difference, and determining whether the grayscale value of the central pixel of the image is higher than the grayscale values ​​of other pixels around the central pixel.

[0090] Step S145, when the grayscale value of the central pixel of the image is higher than the grayscale values ​​of other surrounding pixels, the grayscale of the central pixel of the image is increased, and when the grayscale value of the central pixel of the image is lower than the grayscale values ​​of other surrounding pixels, the grayscale of the central pixel of the image is reduced to extract image structural features.

[0091] In some embodiments, the multimodal registration method based on medical images provided by the present invention calls the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constrains the registration network based on the structural features through a structural similarity measure, and then further includes the following steps:

[0092] In step S210 , the Laplace operator is called to calculate the gradients of the central pixel of the neighborhood in multiple directions, and the multiple gradients obtained are summed to adjust the grayscale of the central pixel of the image according to the gradient operation result.

[0093] In step S220, the convolution kernel is moved row by row on the input image, the values ​​in the convolution kernel and the pixels coinciding with the convolution kernel are multiplied and summed, and the result is assigned to the pixel coinciding with the center of the convolution kernel to obtain a Laplace result.

[0094] Among them, the Laplace result is the image feature structure as the input of the similarity constraint loss.

[0095] Step S150 , training the end-to-end unsupervised image registration algorithm using the training set and the test set, so as to optimize the end-to-end unsupervised image registration algorithm according to the loss constraint and obtain the optimal model weight.

[0096] Specifically, the server trains the end-to-end unsupervised image registration algorithm through the previously constructed training set and test to optimize the end-to-end unsupervised image registration algorithm according to the loss constraint, and obtains the optimal model weight when the loss is lowest and the model training is stable.

[0097] In step S160 , the image to be detected and registered is used as the input of the registration network, and the optimal model weight is applied to output the registration result.

[0098] Specifically, the server uses the image to be detected or the test set as the input of the registration network, combines the application of the optimal model weights, and outputs the final registration result.

[0099] The multimodal registration method based on medical images obtains a brain registration dataset and divides the case images in the dataset into a training set and a test set. Based on the UNet network architecture, the spatial dimension is halved to the minimum layer using convolution strides in the encoding layer. In the decoding stage, upsampling, convolution, and skip connections are alternately used to construct the registration network. Next, an input image from the brain registration dataset is input into the dual architecture. Bidirectional primary registration is performed on the input image, generating forward and inverse deformation fields and a primary registered image. The dual exchanged deformation fields are then applied to the primary registered image for secondary registration, resulting in a secondary registered image. The Laplace operator is then used to extract structural features from the input image, the primary registered image, and the secondary registered image. Based on these structural features, the registration network is constrained using a structural similarity metric. The end-to-end unsupervised image registration algorithm is trained using the training and test sets, and optimized according to the loss constraint to obtain the optimal model weights. Finally, the image to be registered is fed into the registration network, and the optimal model weights are applied to output the registration result. This method utilizes a dual framework and a secondary registration based on deformation field exchange to ensure the diffeomorphic nature of the deformation field. The push-forward and pull-back operations of diffeomorphic duality ensure the inherent consistency between the forward and inverse deformation fields during registration, guaranteeing uniformity in the registration results. Furthermore, the Laplace structure operator is added to extract the structural features of the image that require structural information constraints. By applying similarity constraints to these structural features, more accurate registration results can be achieved.

[0100] Combine Figures 2 to 4 As shown, in a specific embodiment, the multimodal registration method based on medical images provided by the present invention is implemented in Python programming under the Pytorch deep learning framework. The computer graphics card configuration is 3090 and the operating system is Ubuntu 20.04. The present invention first completes the network framework design for image registration, then feeds the training data into the neural network for training, and saves the tuned network weights. Finally, the trained network weights are used to perform image registration testing on medical images, specifically including steps 1 to 5:

[0101] Step 1, data preparation.

[0102] Specifically, the brain registration dataset used in the experiment is the BraTS dataset, which represents the two main modalities of magnetic resonance imaging (MRI), T1 and T2. The T1 modality is anatomical imaging, highlighting normal anatomical structures and providing clear tissue contrast and structural information. The T2 modality is pathological imaging, highlighting lesions with high water content and providing high-contrast information related to the lesions. The BraTS dataset contains 542 cases, including 285 training set cases, 66 validation set cases, and 191 test set cases. The image size is 256×256. During the training process, the training set is first subjected to certain data augmentation processing and then fed into the neural network for training.

[0103] Step 2: Construct the registration network.

[0104] Specifically, the registration network uses a UNet-like architecture, using convolutional strides in the encoding layer to halve the spatial dimension until the smallest layer is reached. The subsequent encoding operation on the coarse representation of the input image is similar to the image pyramid used in traditional image registration work. The decoding stage alternates between upsampling, convolution, and connection skipping. Successive layers of the decoder operate at finer spatial scales, achieving precise anatomical alignment.

[0105] Step 3: Design a multimodal image registration algorithm.

[0106] Specifically, the overall algorithm framework is a dual architecture. The input images are a moving image and a fixed image. First, a bidirectional primary registration is performed, resulting in the forward (moving image to fixed image) and reverse (fixed image to moving image) deformation fields and registered images, respectively. Then, the deformation fields are exchanged between the two dual images and applied to the primary registered image for secondary registration, yielding the secondary registered image. Finally, the Laplace operator is used to extract structural features from the moving image, fixed image, primary registered image, and secondary registered image, respectively. A structural similarity metric is then applied to constrain the images to optimize the registration network and improve system performance.

[0107] In this embodiment, the diffeomorphic nature of the deformation field is ensured through a dual framework and secondary registration based on deformation field exchange. Based on the VoxelMorph algorithm (registering a moving image M to a fixed image F), a dual network architecture is established (registering a fixed image F to a moving image M). This yields a deformation field for inverse registration. The registered image is then subjected to a top-down exchange of the deformation field to obtain a secondary registered image. This ensures the inherent consistency between the forward and inverse deformation fields during registration through the push-and-pull operations of diffeomorphic duality.

[0108] The duality of diffeomorphisms makes them a core tool for studying geometric transformations, as duality establishes a strict one-to-one correspondence between geometric objects. A push-forward describes the mapping of tangent vectors from one coordinate system to another through a deformation field, capturing the transformation of geometric shapes. In image registration, a push-forward describes how the direction and magnitude of local image movement propagate through the deformation field. A pull-back describes the pulling of covariates (such as gradients and densities) from the target coordinate system back to the original coordinate system, ensuring the consistency of the transformed information. In image registration, pull-back maps features (structural information) in the target image back to the source image for feature comparison or similarity measurement. In our algorithm, a secondary registration is performed by exchanging the deformation field to complete the push-forward and pull-back processes.

[0109] Since the modality of the registered image remains unchanged while its structure changes, the registered image from the moving image to the fixed image should theoretically have the same structure as the fixed image, and the registered image from the fixed image to the moving image should theoretically have the same structure as the moving image. Therefore, the registered image from the moving image to the fixed image is further deformed by the deformation field from the fixed image to the moving image to obtain the secondary registered image. Similarly, the registered image from the fixed image to the moving image is deformed by the deformation field from the fixed image to the moving image to obtain the secondary registered image. Finally, the primary and secondary registered images, which should theoretically have the same structural information, are constrained by structural similarity to complete the closed loop of the push and pull process.

[0110] In this embodiment, the similarity metric inherits the basic registration algorithm VoxelMorph mentioned above and uses cross-correlation CC to represent it. This similarity is measured by directly comparing the relationship between pixel intensities. The core goal of image registration optimization is to maximize the structural similarity between the registered image and the fixed image. Therefore, the similarity constraint expression is:

[0111] ;

[0112] Where, is the similarity constraint, is the feature image of the structural feature to be compared, It is the cross-correlation representation of the basic registration method VoxelMorph.

[0113] After that, the feature image pixels corresponding to the input image and the entire pixel set of the input image are obtained, and multiple adjacent pixels around the feature image pixels are determined, and the cross-correlation of the basic registration method VoxelMorph is calculated.

[0114] The cross-correlation expression of the basic registration method VoxelMorph is:

[0115] ;

[0116] Where, is the feature image pixel corresponding to the input image, is the set of all pixels of the input image, are multiple adjacent pixels around the feature image pixel, and are the average pixel intensities of the structural features to be compared.

[0117] In addition to the similarity metric loss, the loss_smooth loss is also inherited from the VoxelMorph algorithm and is used to smooth the deformation field so that it does not overlap as much as possible. Its expression is:

[0118] ;

[0119] Where, is the feature image pixel corresponding to the input image, is the set of all pixels of the input image, is the gradient operator, For any deformation field, Represents the norm.

[0120] In this embodiment, the Laplace structure operator is added to extract the structural features of the image that needs to be constrained by structural information, and similarity constraints are imposed on the structural features. The Laplace operator is a second-order differential operator in n-dimensional Euclidean space. It calculates the pixels in the neighborhood by grayscale difference. The basic process is: judge the grayscale value of the center pixel of the image and the grayscale values ​​of other pixels around it. If the grayscale of the center pixel is higher, then increase the grayscale of the center pixel; otherwise, reduce the grayscale of the center pixel, thereby realizing the image structural feature extraction operation. In the algorithm implementation process, the Laplace operator calculates the gradient of the center pixel in eight directions of the neighborhood, and then adds up the gradients to determine the relationship between the grayscale of the center pixel and the grayscale of other pixels in the neighborhood. Finally, the pixel grayscale is adjusted according to the result of the gradient operation. The differential form of the Laplace operator is:

[0121]

[0122] Where, is the image pixel, for The gray value at 、 is the grayscale value of the pixel in the left and right directions, 、 is the grayscale value of the pixel in the up and down directions, and the remaining four items are the grayscale values ​​of the four points of the pixel. The corresponding convolution kernel is:

[0123] ;

[0124] The convolution kernel is moved row by row across the image. The kernel's values ​​are multiplied by the pixels that coincide with its value, and the sum is assigned to the pixels coinciding with the kernel's center. Pixels in the first and last rows and columns of the image, where the operation cannot be performed, are assigned a value of zero. This yields the Laplace operation result. Finally, the Laplace operation result, i.e., the image's structural features, is used as the input to the loss_sim loss to constrain these structural features and achieve more accurate registration results.

[0125] Step 4: Train the neural network.

[0126] Specifically, this algorithm is an end-to-end unsupervised image registration algorithm. No human intervention is required in the intermediate process. The training and validation data sets are directly input for learning. The algorithm will automatically optimize the network through loss constraints. After the loss gradually decreases and stabilizes, the training ends and the optimal weights are saved.

[0127] Step 5: Multimodal image registration.

[0128] Specifically, the test set or other images to be registered are input into the network, and the optimal weights retained during the training process are applied to conduct experiments to obtain the final registration results and objective indicators.

[0129] Objective metrics include DICE, MI, PSNR, and SSIM. Comparison algorithms used include VoxelMorph, ASNet, Nemar, and TMI. The DICE coefficient measures the degree of overlap between two sets (usually binary images). Higher values ​​indicate greater overlap and better registration. MI mutual information is an information-theoretic metric that measures the correlation between two random variables and is particularly commonly used in multimodal image registration. It measures the amount of shared information between the two variables, with higher values ​​indicating greater information sharing. PSNR is an image quality metric used to compare the difference between the reconstructed image and the original image. It reflects image quality through the signal-to-noise ratio, with higher values ​​indicating better image quality. SSIM measures the structural similarity between two images, taking into account similarities in brightness, contrast, and structure. Higher values ​​indicate better image quality. Unlike PSNR, it better aligns with the perceptual characteristics of the human visual system. Table 1 shows the comparison results of objective indicators (DICE, MI, PSNR and SSIM) of this example (Ours) and the four comparison algorithms VoxelMorph, ASNet, Nemar and TMI:

[0130] Table 1

[0131]

[0132] As shown in the table, this example has good results in all indicators, and the image quality and registration performance are also good.

[0133] The following describes a multimodal registration device based on medical images provided by the present invention. The multimodal registration device based on medical images described below and the multimodal registration method based on medical images described above can refer to each other.

[0134] like Figure 5 As shown, in one embodiment, a multimodal registration device based on medical images includes a data preparation module 510, a registration network construction module 520, an image registration module 530, a feature extraction module 540, a model optimization module 550 and an image registration detection module 560.

[0135] The data preparation module 510 is used to obtain a brain registration dataset and divide the case images in the brain registration dataset into a training set and a test set.

[0136] The registration network construction module 520 is used for a UNet-based network structure, which uses convolution steps in the encoding layer to halve the spatial dimension to the minimum layer, and alternately uses upsampling, convolution and connection skipping in the decoding stage to construct the registration network.

[0137] The image registration module 530 is used to select an input image from the brain registration dataset and input it into the dual architecture, perform bidirectional primary registration on the input image to obtain forward and reverse deformation fields and a primary registration image, and apply the dual exchange deformation field to the primary registration image for secondary registration to obtain a secondary registration image.

[0138] The feature extraction module 540 is used to call the Laplace operator to extract structural features from the input image, the first registered image and the second registered image respectively, and constrain the registration network through a structural similarity measure based on the structural features.

[0139] The model optimization module 550 is used to train the end-to-end unsupervised image registration algorithm through a training set and a test set, so as to optimize the end-to-end unsupervised image registration algorithm according to the loss constraint and obtain the optimal model weight.

[0140] The image registration module 560 is used to take the image to be detected and registered as the input of the registration network, apply the optimal model weights, and output the registration result.

[0141] The BraTS dataset is used for brain registration, which includes anatomical and pathological images from magnetic resonance imaging. The input images include both moving and fixed images. The forward and inverse deformation fields are used to register the moving image to the fixed image and the fixed image to the moving image, respectively.

[0142] In this embodiment, the multimodal registration device based on medical images provided by the present invention, the image registration module 530 is specifically used to:

[0143] The basic registration method VoxelMorph is called to perform a registration from the moving image to the fixed image. Based on the once-registered image, a network architecture is established to perform registration from the fixed image to the moving image, which is dual to the once-registered image, to obtain the deformation field of the inverse registration.

[0144] Through the duality of differential homeomorphism, the primary registered image is transformed into a secondary registered image through the deformation field of upper and lower exchanges.

[0145] Among them, the duality of differential homeomorphism includes push-forward and pull-back. Push-forward is used to describe the local movement direction of the image and the propagation of the image size in the deformation field, and pull-back is used to map the structural features in the image back to the source image to compare structural features or similarity measurements.

[0146] In this embodiment, the multimodal registration device based on medical images provided by the present invention, the feature extraction module 540 is specifically used to:

[0147] The structural similarity between the registered image and the fixed image is calculated based on the structural features, and when the structural similarity reaches the maximum value, the registration network is constrained by similarity. The expression of the similarity constraint is:

[0148] ;

[0149] Where, is the similarity constraint, is the feature image of the structural feature to be compared, It is the cross-correlation representation of the basic registration method VoxelMorph.

[0150] In this embodiment, the multimodal registration device based on medical images provided by the present invention, the feature extraction module 540 is further configured to:

[0151] Obtain the feature image pixels corresponding to the input image and the entire pixel set of the input image, determine multiple adjacent pixels around the feature image pixels, and calculate the cross-correlation of the basic registration method VoxelMorph.

[0152] The cross-correlation expression of the basic registration method VoxelMorph is:

[0153] ;

[0154] Where, is the feature image pixel corresponding to the input image, is the set of all pixels of the input image, are multiple adjacent pixels around the feature image pixel, and are the average pixel intensities of the structural features to be compared.

[0155] In this embodiment, the multimodal registration device based on medical images provided by the present invention, the feature extraction module 540 is further configured to:

[0156] The smoothing loss of any deformation field of the input image is calculated based on the basic registration method VoxelMorph to smooth the deformation field.

[0157] The smoothing loss expression of the deformation field is:

[0158] ;

[0159] Where, is the feature image pixel corresponding to the input image, is the set of all pixels of the input image, is the gradient operator, For any deformation field, Represents the norm.

[0160] In this embodiment, the multimodal registration device based on medical images provided by the present invention, the feature extraction module 540 is further configured to:

[0161] Call the Laplace operator to calculate the pixels in the input image, the first registered image, and the second registered image through grayscale difference, and determine whether the grayscale value of the central pixel of the image is higher than the grayscale values ​​of other pixels around the central pixel.

[0162] When the grayscale value of the central pixel of the image is higher than the grayscale values ​​of other surrounding pixels, the grayscale of the central pixel of the image is increased; when the grayscale value of the central pixel of the image is lower than the grayscale values ​​of other surrounding pixels, the grayscale of the central pixel of the image is reduced to extract image structural features.

[0163] In this embodiment, the multimodal registration device based on medical images provided by the present invention further includes a Laplace constraint module for:

[0164] The Laplace operator is called to calculate the gradients in multiple directions of the central pixel of the neighborhood, and the multiple gradients obtained are summed to adjust the grayscale of the central pixel of the image through the gradient operation results.

[0165] The convolution kernel is moved row by row on the input image, the values ​​in the convolution kernel and the pixels coinciding with the convolution kernel are multiplied and summed, and the result is assigned to the pixel coinciding with the center of the convolution kernel to obtain the Laplacian result.

[0166] Among them, the Laplace result is the image feature structure as the input of the similarity constraint loss.

[0167] Figure 6 The following is a schematic diagram of the physical structure of an electronic device. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The electronic device includes a processor, an internal memory, and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a multimodal registration method based on medical images is implemented, which includes:

[0168] Obtain a brain registration dataset, and divide the case images in the brain registration dataset into a training set and a test set;

[0169] Based on the UNet network structure, the convolution step size is used in the encoding layer to halve the spatial dimension to the minimum layer, and upsampling, convolution and connection skipping are alternately used in the decoding stage to construct the registration network;

[0170] Select an input image from the brain registration dataset and input it into the dual architecture. Perform a bidirectional primary registration on the input image to obtain the forward and reverse deformation fields and the primary registration image. Then, apply the dual deformation fields of both sides to the primary registration image for secondary registration to obtain the secondary registration image.

[0171] The Laplace operator is called to extract structural features from the input image, the first registered image, and the second registered image, and the registration network is constrained by the structural similarity measure based on the structural features;

[0172] The end-to-end unsupervised image registration algorithm is trained through the training set and the test set to optimize the end-to-end unsupervised image registration algorithm according to the loss constraint to obtain the optimal model weight;

[0173] The image to be detected and registered is used as the input of the registration network, and the optimal model weight is applied to output the registration result;

[0174] Among them, the brain registration dataset is the BraTS dataset, which includes anatomical imaging and pathological imaging in magnetic resonance imaging; the input images include moving images and fixed images, and the forward and inverse deformation fields are the deformation fields for registering from moving images to fixed images and the deformation fields for registering from fixed images to moving images, respectively.

[0175] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the electronic device to which the solution of the present invention is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0176] In another aspect, the present invention further provides a computer storage medium storing a computer program, which, when executed by a processor, implements a multimodal registration method based on medical images, the method comprising:

[0177] Obtain a brain registration dataset, and divide the case images in the brain registration dataset into a training set and a test set;

[0178] Based on the UNet network structure, the convolution step size is used in the encoding layer to halve the spatial dimension to the minimum layer, and upsampling, convolution and connection skipping are alternately used in the decoding stage to construct the registration network;

[0179] Select an input image from the brain registration dataset and input it into the dual architecture. Perform a bidirectional primary registration on the input image to obtain the forward and reverse deformation fields and the primary registration image. Then, apply the dual deformation fields of both sides to the primary registration image for secondary registration to obtain the secondary registration image.

[0180] The Laplace operator is called to extract structural features from the input image, the first registered image, and the second registered image, and the registration network is constrained by the structural similarity measure based on the structural features;

[0181] The end-to-end unsupervised image registration algorithm is trained through the training set and the test set to optimize the end-to-end unsupervised image registration algorithm according to the loss constraint to obtain the optimal model weight;

[0182] The image to be detected and registered is used as the input of the registration network, and the optimal model weight is applied to output the registration result;

[0183] Among them, the brain registration dataset is the BraTS dataset, which includes anatomical imaging and pathological imaging in magnetic resonance imaging; the input images include moving images and fixed images, and the forward and inverse deformation fields are the deformation fields for registering from moving images to fixed images and the deformation fields for registering from fixed images to moving images, respectively.

[0184] In another aspect, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, implements a multimodal registration method based on medical images, the method comprising:

[0185] Obtain a brain registration dataset, and divide the case images in the brain registration dataset into a training set and a test set;

[0186] Based on the UNet network structure, the convolution step size is used in the encoding layer to halve the spatial dimension to the minimum layer, and upsampling, convolution and connection skipping are alternately used in the decoding stage to construct the registration network;

[0187] Select an input image from the brain registration dataset and input it into the dual architecture. Perform a bidirectional primary registration on the input image to obtain the forward and reverse deformation fields and the primary registration image. Then, apply the dual deformation fields of both sides to the primary registration image for secondary registration to obtain the secondary registration image.

[0188] The Laplace operator is called to extract structural features from the input image, the first registered image, and the second registered image, and the registration network is constrained by the structural similarity measure based on the structural features;

[0189] The end-to-end unsupervised image registration algorithm is trained through the training set and the test set to optimize the end-to-end unsupervised image registration algorithm according to the loss constraint to obtain the optimal model weight;

[0190] The image to be detected and registered is used as the input of the registration network, and the optimal model weight is applied to output the registration result;

[0191] Among them, the brain registration dataset is the BraTS dataset, which includes anatomical imaging and pathological imaging in magnetic resonance imaging; the input images include moving images and fixed images, and the forward and inverse deformation fields are the deformation fields for registering from moving images to fixed images and the deformation fields for registering from fixed images to moving images, respectively.

[0192] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0193] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0194] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0195] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A multimodal registration method based on medical images, characterized in that: The method comprises: Acquire a brain registration dataset, and divide case images in the brain registration dataset into a training set and a test set; Based on the UNet network structure, the convolution step size is used in the encoding layer to halve the spatial dimension to the minimum layer, and upsampling, convolution and connection skipping are alternately used in the decoding stage to construct the registration network; Selecting an input image from the brain registration dataset and inputting it into the dual architecture, performing bidirectional primary registration on the input image to obtain forward and reverse deformation fields and a primary registration image, and applying the dual exchange deformation fields to the primary registration image to perform secondary registration to obtain a secondary registration image; Calling the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, respectively, and constraining the registration network through a structural similarity measure based on the structural features; Training an end-to-end unsupervised image registration algorithm using the training set and the test set to optimize the end-to-end unsupervised image registration algorithm according to a loss constraint to obtain an optimal model weight; Taking the image to be detected and registered as the input of the registration network, applying the optimal model weights, and outputting the registration result; The brain registration dataset is a BraTS dataset, which includes anatomical imaging and pathological imaging in magnetic resonance imaging; the input image includes a moving image and a fixed image, and the forward and reverse deformation fields are respectively the deformation fields for registering from the moving image to the fixed image and the deformation fields for registering from the fixed image to the moving image.

2. The multimodal registration method based on medical images according to claim 1, characterized in that: The input image is selected from the brain registration dataset and input into the dual architecture, the input image is bidirectionally registered once to obtain forward and reverse deformation fields and a primary registration image, and the dual deformation fields are exchanged and applied to the primary registration image to perform secondary registration to obtain a secondary registration image, including: Calling the basic registration method VoxelMorph to perform a primary registration from the moving image to the fixed image, and establishing a network architecture for registration from the fixed image to the moving image that is dual to the primary registration image based on the primary registration image to obtain a deformation field for inverse registration; The secondary registration image is obtained by subjecting the primary registration image to a deformation field with upper and lower exchanges through the duality of differential homeomorphism; The duality of the differential homeomorphism includes push-forward and pull-back. The push-forward is used to describe the local movement direction of the image and the propagation of the image size in the deformation field. The pull-back is used to map the structural features in the image back to the source image to compare the structural features or similarity measurements.

3. The multimodal registration method based on medical images according to claim 1, characterized in that: The calling of the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constraining the registration network through a structural similarity measure based on the structural features, includes: The structural similarity between the registered image and the fixed image is calculated based on the structural features, and when the structural similarity reaches a maximum value, a similarity constraint is applied to the registration network. The expression of the similarity constraint is: ； Where, is the similarity constraint, is the feature image of the structural feature to be compared, It is the cross-correlation representation of the basic registration method VoxelMorph.

4. The multimodal registration method based on medical images according to claim 3, characterized in that: The calling of the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constraining the registration network through a structural similarity measure based on the structural features, further comprising: Obtaining a feature image pixel corresponding to the input image and a set of all pixels of the input image, determining a plurality of adjacent pixels around the feature image pixel, and calculating a cross-correlation of the basic registration method VoxelMorph; The cross-correlation expression of the basic registration method VoxelMorph is: ； Where, is the feature image pixel corresponding to the input image, is the set of all pixels of the input image, are multiple adjacent pixels around the feature image pixel, and are the average pixel intensities of the structural features to be compared.

5. The multimodal registration method based on medical images according to claim 4, characterized in that: The calling of the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constraining the registration network through a structural similarity measure based on the structural features, further comprising: Calculating the smoothing loss of any deformation field of the input image based on the basic registration method VoxelMorph to smooth the deformation field; The smoothing loss expression of the deformation field is: ； Where, is the feature image pixel corresponding to the input image, is the set of all pixels of the input image, is the gradient operator, For any deformation field, Represents the norm.

6. The multimodal registration method based on medical images according to claim 5, characterized in that: The calling of the Laplace operator to extract structural features from the input image, the first registered image, and the second registered image, and constraining the registration network through a structural similarity measure based on the structural features, further comprising: Calling the Laplace operator to calculate pixels in the input image, the first registered image, and the second registered image by grayscale difference, and determining whether the grayscale value of the central pixel of the image is higher than the grayscale values ​​of other pixels around the central pixel; When the grayscale value of the central pixel of the image is higher than the grayscale values ​​of other surrounding pixels, the grayscale of the central pixel of the image is increased, and when the grayscale value of the central pixel of the image is lower than the grayscale values ​​of other surrounding pixels, the grayscale of the central pixel of the image is reduced to extract image structural features.

7. The multimodal registration method based on medical images according to claim 6, characterized in that: The calling of the Laplace operator extracts structural features from the input image, the first registered image, and the second registered image respectively, and constrains the registration network through a structural similarity measure based on the structural features, and then further includes: Calling the Laplace operator to calculate the gradient of the central pixel of the neighborhood in multiple directions, and summing the multiple gradients obtained to adjust the grayscale of the central pixel of the image according to the gradient operation result; Moving the convolution kernel row by row on the input image, multiplying the value in the convolution kernel by the pixels coinciding with the convolution kernel, summing the values, and assigning the result to the pixel coinciding with the center of the convolution kernel to obtain a Laplacian result; The Laplace result is the image feature structure as the input of the similarity constraint loss.

8. A multimodal registration device based on medical images, characterized in that: The device comprises: A data preparation module is used to obtain a brain registration dataset and divide the case images in the brain registration dataset into a training set and a test set; A registration network building module for a UNet-based network structure that uses convolutional strides in the encoding layer to halve the spatial dimension to the minimum layer, and alternately uses upsampling, convolution, and connection skipping in the decoding stage to construct the registration network; an image registration module, configured to select an input image from the brain registration dataset and input it into the dual architecture, perform a bidirectional primary registration on the input image to obtain forward and reverse deformation fields and a primary registered image, and perform a secondary registration on the primary registered image by applying the dual exchange deformation fields to obtain a secondary registered image; A feature extraction module, configured to extract structural features from the input image, the first registered image, and the second registered image respectively by calling the Laplace operator, and constrain the registration network based on the structural features by using a structural similarity metric; A model optimization module, configured to train an end-to-end unsupervised image registration algorithm using the training set and the test set, so as to optimize the end-to-end unsupervised image registration algorithm according to a loss constraint to obtain an optimal model weight; An image registration detection module, configured to take the image to be detected as the input of the registration network, apply the optimal model weights, and output the registration result; The brain registration dataset is a BraTS dataset, which includes anatomical imaging and pathological imaging in magnetic resonance imaging; the input image includes a moving image and a fixed image, and the forward and reverse deformation fields are respectively the deformation fields for registering from the moving image to the fixed image and the deformation fields for registering from the fixed image to the moving image.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images