3D medical image registration method and device enhanced by contrastive learning of dual attention transformers

Abstract

The present invention discloses a 3D medical image registration method and device enhanced by contrastive learning of dual attention transformers. The registration method comprises: obtaining a 3D medical image data set and preprocessing it; then, downsampling a fixed image and a floating image to create an image pyramid; then, building a transformer-based registration network, and finally, using a loss function to calculate the loss function value between the fixed image and the registered image, and optimizing the network model by back propagation, thereby obtaining a trained neural network model. The present invention can fully capture local and global context information and improve the modeling ability of image details and overall structure by combining self-attention mechanism and cross-attention mechanism. And the contrast loss function is used to measure the distance between corresponding feature points between images. At the same time, the normalized total gradient and modality-independent neighborhood descriptor loss function are used to further calculate the similarity between images, thereby improving the accuracy and precision of medical image registration.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a 3D medical image registration method and device enhanced by dual attention transformer contrast learning. Background Art

[0002] Image registration is an important branch of computer vision, and is of great significance in the fields of disease diagnosis, image guidance, surgical navigation, and organ motion analysis. Image registration aims to construct a dense nonlinear transformation from a floating image to a fixed image, and medical image registration is a technology that aligns and matches medical images of different times, sources, or modalities. Compared with traditional image registration tasks, three-dimensional medical images have larger deformations, higher resolutions, and more complex structures. In addition, since the medical image data obtained by different imaging technologies will be affected by differences in body position and location, as well as differences in imaging equipment, the image data morphology and intensity vary greatly. These factors bring great challenges to three-dimensional medical image registration.

[0003] Traditional 3D medical image registration methods can be divided into two categories according to the type of geometric transformation, namely, registration methods based on rigid transformation and registration methods based on non-rigid transformation. These methods involve a large number of iterative optimizations, which consume a lot of time and resources, greatly limiting the real-time and effectiveness of medical image registration.

[0004] At present, the research on three-dimensional medical image registration methods based on deep learning has developed rapidly, which can be mainly divided into two aspects: 1) registration methods based on supervised learning; 2) registration methods based on unsupervised learning; 3) registration methods based on reinforcement learning. Representative works include: In the registration method based on supervised learning, Fan et al. published "Brain image registration using dual-supervised fully convolutional networks", Medical Image Analysis, 2019, pp. 193-206. This method introduces a dual supervision mechanism and a fully convolutional network architecture, and uses the registration results generated by traditional registration methods as true labels. However, it is often very difficult to obtain labels for anatomical structures or specific organs in medical images. Registration methods based on supervised learning are easily restricted by segmentation labels, which seriously limits the application of such methods in medical images. Among the registration methods based on unsupervised learning, Balakrishnan et al. published "An unsupervised learning model for deformable medical image registration", IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 9252-9260. This method is an unsupervised registration method similar to the U-Net architecture, which is used to estimate dense spatial deformation fields. However, this method constructs a single-scale deformable registration network by stacking convolutional blocks, which is difficult to solve problems such as large displacement and complex deformation in medical images. Wang et al. published "End-to-end unsupervised deformable image registration with convolutional neural networks", IEEE Transactions on Medical Imaging, 2019, pp. 610-620. This article proposes an end-to-end unsupervised registration model, namely DIRNet (Deformable Image Registration Network).The model uses ConvNet to analyze image pairs and output spatial transformer parameters to generate a displacement vector field to distort floating images into fixed images. DIRNet performs unsupervised optimization training through similarity metrics and can be applied to unseen image pairs. However, the spatial transformer used by DIRNet can only capture linear or locally nonlinear image deformations, and it may not provide accurate registration results for more complex nonlinear image deformations. Although the registration method based on unsupervised learning is not limited by labels, this type of method mainly relies on the statistical characteristics of the image itself for registration, which may not accurately reflect the anatomical structure and physiological characteristics of medical images. Patent application CN118314175A of Yu Guorong et al. is based on an unsupervised deformable three-dimensional medical image registration method and device, using deep learning technology and unsupervised learning to achieve global affine registration by fusing a dilated convolutional visual transformer and a normalized total gradient. This method introduces global connectivity and locality into the visual transformer, and at the same time has the ability to resist radiation distortion that may occur in different medical imaging modes. Although this method shows good performance on a specific dataset, it may need to be adjusted under different datasets to obtain the best performance. Among the registration methods based on reinforcement learning, patent application CN110009669A by Yang Bo et al. discloses a 3D / 2D medical image registration method based on deep reinforcement learning. The method first pre-processes the 2D / 3D medical images to obtain multi-resolution fixed images and floating images, and then uses a reinforcement learning network to project the 3D images to obtain DRR images. The similarity between each DRR image and the 2D reference image is calculated, and the DRR image with the highest similarity measure is selected. Finally, the DRL network is used for image registration, but no experimental results have been provided yet.

[0005] In summary, the above methods still have some limitations. The main difficulties are: 1) The registration model is too complex and requires a lot of time and resources; 2) It is difficult to see effectiveness in the registration tasks of complex medical images, especially those with large deformation; 3) The existing public medical image registration methods have strong assumptions and lack practical applications. Summary of the invention

[0006] The present invention provides a 3D medical image registration method and device enhanced by dual attention transformer contrast learning, which are used to solve or at least partially solve the technical problem of poor registration effect existing in the prior art.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] The first aspect provides a 3D medical image registration method enhanced by dual attention transformer contrast learning, comprising:

[0009] Acquire 3D medical image data and divide it into training set and test set;

[0010] Preprocessing the divided training set and test set, and selecting fixed images and floating images from the preprocessed training set;

[0011] Downsampling each pair of fixed and floating images to create an image pyramid, and concatenating the fixed and floating images at each level of the pyramid;

[0012] The concatenated image is input into a pre-built transformer-based registration network to obtain a deformation field of the pyramid, wherein the transformer-based registration network includes three stages, each stage uses the same module, each stage includes an image block embedding layer, a dual attention transformer and a deformable registration head, the image block embedding layer is used to process the concatenated image to obtain a block embedding matrix, the dual attention transformer includes a dual attention module and a feedforward module, the dual attention module is used to process the block embedding matrix using an inter-layer attention mechanism and a self-attention mechanism to obtain a feature matrix, the feedforward module is based on dilated convolution, and is used to obtain feature embedding according to the feature matrix, and the deformable registration head is used to obtain the final deformation field according to the feature embedding;

[0013] The floating image is interpolated using the final deformation field to obtain the registered floating image, and the correlation between the registered floating image and the fixed image is obtained by point registration. The corresponding feature points of the group are calculated using contrast loss, and the similarity between the feature points is calculated using the normalized total gradient and the modality-independent neighborhood descriptor. The objective function is established by the similarity between the corresponding feature points of the groups and the similarity between the registered floating image and the fixed image. The transformer-based registration network is back-propagated and optimized through the objective function to obtain a trained neural network model.

[0014] The test set is input into the trained neural network model to obtain the optimal deformation field, and the floating image is deformably registered using the optimal deformation field to obtain the final registered image.

[0015] In one embodiment, preprocessing the divided training set and test set, and selecting fixed images and floating images from the preprocessed training set, includes:

[0016] The images in the training set and test set are cropped, resampled, normalized, and motion aligned to adjust the images to a uniform size and resolution.

[0017] For the training set, fixed images and floating images are selected by cross selection;

[0018] For the test set, one image is randomly selected as the fixed image and the remaining images are used as floating images.

[0019] In one embodiment, each pair of fixed image and floating image is downsampled to create an image pyramid, and the fixed image and floating image of each layer are connected, including:

[0020] The fixed image is interpolated by trilinear interpolation. and floating images Downsample to obtain and ,in, For the Fixed image of the layer, For the Floating image of the layer;

[0021] Based on the fixed image and floating image of each layer, an L-layer image pyramid is constructed, where the fixed image and floating image of the last layer of the pyramid are , ;

[0022] Will and Connection, where The concatenated image is used as the transformer-based registration network The input image of the stage.

[0023] In one implementation, the concatenated images are input into a pre-built transformer-based registration network to obtain a deformation field of the pyramid, including:

[0024] The image transformation step is used to transform and The concatenated image is fed into a pre-built transformer-based registration network to obtain the Deformation field of the stage ;

[0025] The spatial transformation and connection steps are used to transform With Floating image of the layer image pyramid Perform spatial transformation to obtain the image after spatial transformation , then connect and , as the first The input image of the stage, For the pyramid Fixed image of the layer;

[0026] No. The first stage is obtained by performing the operations of the image transformation step and the space transformation and connection step. Deformation field of the stage and the input image of the next stage, repeating this operation until the deformation field of the Lth stage is obtained , as the final deformation field.

[0027] In one embodiment, the image transformation step includes:

[0028] The transformer-based registration network The input image of the stage is represented as ,in , and express height, width and depth, is the number of channels, and the image block embedding layer uses convolution operation to Convert to image patch embedding map , and then Flattened into a series of block embeddings ,in, Used to aggregate into block embedding matrix , i An index representing the number of image blocks, , and Indicates i The height, width and depth of the image block, represents the embedding dimension, Indicates i The number of image blocks;

[0029] After adding the position embedding to the block embedding matrix, it is input into the dual attention transformer of the same stage, where each dual attention module contains a self-attention mechanism and A cross-attention mechanism;

[0030] The block embedding matrix after adding the position embedding first passes through the linear projection matrix , , The transformation of ,key Sum The matrix is ​​then calculated through the self-attention mechanism, where the The calculation formula of the self-attention mechanism in the stage is:

[0031] (1)

[0032] for The characteristic dimension of Indicates The block embedding matrix after adding the position embedding of the stage, Indicates the total number of stages;

[0033] The cross attention calculation is performed based on the output of the self-attention mechanism. The calculation formula of the cross attention mechanism is:

[0034] (2)

[0035] , Represents the two input matrices of the cross attention mechanism, Indicates the number of a stage. represents the query matrix obtained after the first stage of self-attention mechanism, Represents the value matrix and key matrix obtained after the second or third stage of self-attention or cross-attention mechanism. After the self-attention mechanism and cross-attention mechanism of the stage, the feature matrix is ​​obtained ;

[0036] Input the feature matrix into the feed-forward module to generate feature embedding , where the feedforward module consists of a two-hidden layer Block, add three dilated convolutions between hidden layers, feature matrix After the first hidden layer and activation function processing, the processing result is then sent to the dilated convolution layer to obtain multi-scale feature representation and finally generate feature embedding ;

[0037] Embedding features Send it to the deformable registration head and get the Deformation field of the stage , where the deformable registration head is implemented based on a 3×3×3 convolutional layer.

[0038] In one embodiment, The first stage is obtained by performing the operations of the image transformation step and the space transformation and connection step. Deformation field of the stage and the input image for the next stage, including:

[0039] Will Increment to Then repeat the image transformation steps to get The block embedding matrix of the stage , Compared with the feature embedding in the previous stage After addition, the corresponding dual attention transformer is input. In the dual attention transformer, the output of the attention layer of the previous stage is used as the first The input of the cross attention calculation of the stage is the feature matrix output by the dual attention module Sequentially sent to the feedforward module and deformable registration head at the same stage, the Feature embedding of the stage and Temporary deformation field of the stage ;

[0040] Will Increment to Then repeat the spatial transformation and connection steps to transform the deformation field of the previous stage Upsample and Add up to get The final deformation field of the stage .

[0041] In one embodiment, the calculation of the objective function includes:

[0042] Get the registered floating image With fixed image The normalized total gradient between:

[0043] (3)

[0044] in, Indicates along The derivative of the direction , The difference image between the registered floating image and the fixed image;

[0045] Get the registered floating image With fixed image The modality-independent neighborhood descriptor between:

[0046] (4)

[0047] in, , yes , The number of pixels, Compute the modality-independent neighborhood descriptor of the image;

[0048] Get the registered floating image With fixed image between The contrast loss of the corresponding feature points of the group:

[0049] (5)

[0050] in, Represents the index number of k groups of feature points and for and The corresponding registration point pairs between and for and The temperature coefficient of the non-corresponding registration point pairs It is a hyperparameter used to balance the convergence speed and stability of the model;

[0051] Deformation Field The smoothing regularization term is:

[0052] (6)

[0053] in, represents the gradient of the deformation field;

[0054] The objective function is calculated based on the normalized total gradient, the modality-independent neighborhood descriptor, the contrast loss, and the smooth regularization term of the deformation field:

[0055] (7)

[0056] in, is the weight of the normalized total gradient, is the weight of the modality-independent neighborhood descriptor, is the weight of the contrast loss, is the weight of the smooth regularization term of the deformation field.

[0057] Based on the same inventive concept, the second aspect of the present invention provides a 3D medical image registration device enhanced by contrastive learning using dual attention transformers, comprising:

[0058] The data set acquisition and division module is used to acquire 3D medical image data and divide it into training sets and test sets;

[0059] A data preprocessing module is used to preprocess the divided training set and test set, and select fixed images and floating images from the preprocessed training set;

[0060] An image pyramid building module, for downsampling each pair of fixed image and floating image to create an image pyramid, and connecting the fixed image and floating image at each layer;

[0061] An image transformation module is used to input the connected image into a pre-built transformer-based registration network to obtain a deformation field of the pyramid, wherein the transformer-based registration network includes three stages, each stage uses the same module, each stage includes an image block embedding layer, a dual attention transformer and a deformable registration head, the image block embedding layer is used to process the connected image to obtain a block embedding matrix, the dual attention transformer includes a dual attention module and a feedforward module, the dual attention module is used to process the block embedding matrix using an inter-layer attention mechanism and a self-attention mechanism to obtain a feature matrix, the feedforward module is based on dilated convolution, and is used to obtain feature embedding according to the feature matrix, and the deformable registration head is used to obtain a final deformation field according to the feature embedding;

[0062] The model training module is used to interpolate the floating image using the final deformation field to obtain the registered floating image, and obtain the correlation between the registered floating image and the fixed image through point registration. The corresponding feature points of the group are calculated using contrast loss, and the similarity between the feature points is calculated using the normalized total gradient and the modality-independent neighborhood descriptor. The objective function is established by the similarity between the corresponding feature points of the groups and the similarity between the registered floating image and the fixed image. The transformer-based registration network is back-propagated and optimized through the objective function to obtain a trained neural network model.

[0063] The image registration module is used to input the test set into the trained neural network model to obtain the optimal deformation field, and use the optimal deformation field to perform deformable registration on the floating image to obtain the final registered image.

[0064] Based on the same inventive concept, the third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the 3D medical image registration method enhanced by dual attention transformer contrast learning as described in the first aspect.

[0065] Based on the same inventive concept, the fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the 3D medical image registration method enhanced by contrastive learning of dual attention transformers as described in the first aspect is implemented.

[0066] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0067] The present invention proposes a dual-attention transformer contrast learning enhanced 3D medical image registration method, which adopts a transformer-based registration network, which includes a dual-attention transformer, which designs a self-attention mechanism and a cross-attention mechanism, and can better balance the acquisition of local and global information, thereby improving the modeling ability of image details and overall structure, and the feedforward module combined with the dilated convolution can embed multi-scale information and local features into the image block, further introducing locality into the transformer. The present invention also introduces the normalized total gradient, the modality-independent neighborhood descriptor, the contrast loss and the smoothing regularization term as the loss function of the network model, and designs it as the total loss function for end-to-end learning. Compared with the commonly used cross-correlation coefficient, mutual information, mean square error and other measurement methods, the normalized total gradient can capture the gradient information of the registration image and the reference image, and by normalizing the gradient vector, the difference in image scale and intensity is eliminated, which makes it more sensitive to local deformation and local changes of the image. The modality-independent neighborhood descriptor provides a modality-independent image feature extraction method, resists intensity changes, captures multidimensional features and provides local similarity information. Point registration and contrast loss calculate the similarity of corresponding feature points between the fixed image and the registered image, accelerating the registration training process. The smooth regularization term constrains the deformation field to maintain the continuity and naturalness of the deformation field. These newly added loss functions and descriptors enhance the performance of the registration network model, making it more effective and robust in the task of medical image registration. In addition, the present invention introduces a gradually optimized deformation field to flexibly capture nonlinear transformations in the image, better retain the detailed information of the image, and improve the accuracy and adaptability of the registration. Compared with the prior art, the transformer model based on dual attention in the present invention effectively establishes the global and local relationships between adjacent image blocks, improves the spatial consistency of the registration method and the utilization of contextual information. At the same time, it has better robustness for complex multimodal medical image registration, improves the registration effect of medical images, and can provide important technical support for robotic surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0069] Figure 1 A flowchart of a 3D medical image registration method enhanced by dual attention transformer contrast learning provided by an embodiment of the present invention;

[0070] Figure 2 A schematic diagram of the structure of a dual attention converter used in an embodiment of the present invention;

[0071] Figure 3 Schematic diagram of normalized total gradient according to an embodiment of the present invention;

[0072] Figure 4 is an image example of a three-dimensional medical image dataset in an embodiment of the present invention;

[0073] Figure 5 is a comparison result diagram of the medical image registration method of the present invention and the benchmark method;

[0074] Figure 6 A module diagram of a 3D medical image registration device enhanced by dual attention transformer contrast learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The published patent application CN118314175A is based on an unsupervised deformable three-dimensional medical image registration method and device, using deep learning technology and unsupervised learning, and realizing global affine registration by fusing a visual transformer of dilated convolution and a normalized total gradient. This method introduces global connectivity and locality into the visual transformer, and at the same time has the ability to resist radiation distortion that may occur in different medical imaging modes. However, there are complex changes in brightness, contrast, etc. in medical images, and relying solely on the normalized total gradient as a loss function may not be able to fully solve these problems. In addition, this method uses an affine matrix to transform the image, and the affine matrix can usually only capture linear transformations. For medical images with nonlinear problems, the affine transformation may not provide sufficient flexibility and accuracy. Although this method shows good performance on a specific dataset (OASIS), it needs to be adjusted under different datasets to obtain the best performance.

[0076] The main differences between the present invention and patent application CN118314175A include:

[0077] 1. The registration network based on the transformer is different, including: the dual attention module and the feedforward module are different. CN118314175A only uses the self-attention mechanism, and the present invention adopts a mechanism that combines inter-layer attention with self-attention to increase inter-layer attention; in each stage, the visual transformer of CN118314175A has 4 identical encoders (attention mechanism + feedforward layer), and the visual transformer of each stage of the present invention adopts a dual attention transformer, with only 1 encoder (self-attention + cross attention + feedforward layer). The feedforward module uses a different way of adding dilated convolutions. CN118314175A adds a dilated convolution layer between the two hidden layers in the MLP of the visual transformer. This solution adds three dilated convolutions to the MLP of the visual transformer, effectively expanding the receptive field range of the local descriptor.

[0078] 2. The image transformation model is different. The transformation model in the CN118314175A method is a 3×4 affine transformation matrix, and the matrix of the i-th layer is directly deformed with the floating image of the i+1th layer. The transformation model of this scheme is a deformation field. The deformation field of the i-th layer is upsampled and deformed with the floating image of the i+1th layer, and the final deformation field of the i+1th layer is obtained by adding it to the temporary deformation field of the i+1th layer.

[0079] 3. Different objective functions. The deformable medical image registration method uses the normalized total gradient to establish the objective function. This scheme combines the normalized total gradient, the modality-independent neighborhood descriptor, the smoothing loss term, and the contrast loss function to establish the objective function, and performs back-propagation optimization on the network model. This scheme performs point registration between the registered image and the fixed image, and uses the contrast loss function to measure the similarity of the corresponding registration points to further optimize the registration result. This is not available in the deformable medical image registration method.

[0080] In general, the registration models of CN118314175A and this solution have three stages, and the scales of the input images in different stages are different. CN118314175A uses the traditional self-attention mechanism to capture the dependencies between adjacent image blocks of the same scale, but the image information obtained by the attention mechanism of other scales cannot be effectively acquired and transmitted. This solution introduces inter-layer cross attention in the visual transformer, which can not only capture the image features of the same scale, but also cross different scales to capture information between different scales, thereby improving the modeling ability of image details and overall structure. Although there is only one encoder in the dual attention transformer in the present invention, through the hierarchical design and the optimization of the cross attention mechanism, it can still effectively capture multi-scale features, while improving the parameter efficiency, information transmission ability and feature representation quality of the model. If four identical encoders are still used, it will be detrimental to the efficiency of the model. In addition, considering that the affine matrix can usually only capture linear transformations, the present invention introduces a gradually optimized deformation field to flexibly capture nonlinear transformations in the image, better retain the detailed information of the image, and improve the accuracy and adaptability of the registration. The loss function has been newly added with modality-independent neighborhood descriptor, contrast loss and smooth regularization term. The modality-independent neighborhood descriptor is a modality-independent local descriptor that can effectively handle feature extraction of images of different modalities, resist intensity changes, capture multi-dimensional features and provide local similarity information. Combining point registration and contrast loss to calculate the similarity between feature points at corresponding positions of the fixed image and the registered image helps to evaluate the accuracy and quality of registration, and then assists in adjusting and optimizing the registration process. The smooth regularization term can constrain the deformation field during the registration process to maintain the continuity and naturalness of the deformation field. The newly added loss function not only enhances the sensitivity and extraction ability of feature changes of the registration network model, but also effectively resists radiation distortion and intensity changes in medical images, enhancing the performance of the registration network model in many aspects, making it more effective and robust in medical image registration tasks.

[0081] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] Embodiment 1

[0083] The present invention discloses a 3D medical image registration method enhanced by dual attention transformer contrast learning, comprising:

[0084] S1: Acquire 3D medical image data and divide it into training set and test set;

[0085] S2: preprocessing the divided training set and test set, and selecting fixed images and floating images from the preprocessed training set;

[0086] S3: downsample each pair of fixed and floating images to create an image pyramid, and connect the fixed and floating images at each level of the pyramid;

[0087] S4: Input the concatenated image into a pre-built transformer-based registration network to obtain a deformation field of the pyramid, wherein the transformer-based registration network includes three stages, each stage uses the same module, each stage includes an image block embedding layer, a dual attention transformer and a deformable registration head, the image block embedding layer is used to process the concatenated image to obtain a block embedding matrix, the dual attention transformer includes a dual attention module and a feedforward module, the dual attention module is used to process the block embedding matrix using an inter-layer attention mechanism and a self-attention mechanism to obtain a feature matrix, the feedforward module is based on dilated convolution, and is used to obtain feature embedding according to the feature matrix, and the deformable registration head is used to obtain the final deformation field according to the feature embedding;

[0088] S5: Use the final deformation field to interpolate the floating image to obtain the registered floating image, and obtain the value between the registered floating image and the fixed image through point registration. The corresponding feature points of the group are calculated using contrast loss, and the similarity between the feature points is calculated using the normalized total gradient and the modality-independent neighborhood descriptor. The objective function is established by the similarity between the corresponding feature points of the groups and the similarity between the registered floating image and the fixed image. The transformer-based registration network is back-propagated and optimized through the objective function to obtain a trained neural network model.

[0089] S6: Input the test set into the trained neural network model to obtain the optimal deformation field, and use the optimal deformation field to perform deformable registration on the floating image to obtain the final registered image.

[0090] First, the marks involved in the technical solution adopted by the present invention are uniformly described as follows:

[0091] Tanh represents the hyperbolic tangent activation function,

[0092] MLP stands for Multi-layer Perceptron.

[0093] Vision Transformer stands for Visual Transformer.

[0094] OASIS stands for Open Access Series of Imaging Studies, a public medical imaging dataset.

[0095] LPBA stands for the public medical image dataset LONI Probabilistic Brain Atlas.

[0096] See also Figure 1 , which is a flow chart of a 3D medical image registration method enhanced by dual attention transformer contrast learning provided in an embodiment of the present invention.

[0097] Specifically, S1 is the acquisition and division of the data set, S2 is the data preprocessing and the selection of fixed images and floating images, S3 is the construction of an image pyramid and the connection of fixed images and floating images at each layer to obtain the input image of the corresponding stage of the registration network, S4 is the use of the pre-built transformer-based registration network to transform and process the input image to obtain the deformation field of the pyramid, S5 is the use of the deformation field obtained by S4 to interpolate the floating image, and then construct the objective function based on the normalized total gradient, modality-independent neighborhood descriptor, contrast loss, etc., and the transformer-based registration network is back-propagated and optimized through the objective function to obtain a trained neural network model, and S6 is the use of the trained model to perform deformable registration on the floating images in the test set to obtain the final registered image. The pyramid refers to the creation of three input images of different scales, commonly known as the image pyramid.

[0098] In the specific implementation process, the 3D medical image data obtained in S1 comes from the public datasets OASIS and LPBA, and then randomly divided into training sets and test sets according to a certain ratio. S3 connects the fixed image and floating image of the corresponding layer as the input image of the corresponding stage of the registration network. In S4, the dual attention transformer includes a dual attention module and a feedforward module. The dual attention module is used to process the block embedding matrix using the inter-layer attention mechanism and the self-attention mechanism, that is, the inter-layer cross attention is introduced in the visual transformer, which can not only capture the image features of the same scale, but also cross different scales and capture information between different scales, thereby improving the modeling ability of image details and overall structure. The objective function in S5 considers the normalized total gradient, the modality-independent neighborhood descriptor, and the contrast loss. By normalizing the gradient vector, the differences in image scale and intensity are eliminated, which makes it more sensitive to local deformation and local changes of the image. The modality-independent neighborhood descriptor provides a modality-independent image feature extraction method that resists intensity changes, captures multi-dimensional features, and provides local similarity information. Point registration and contrast loss calculate the similarity of corresponding feature points between the fixed image and the registered image, accelerating the registration training process. The smooth regularization term constrains the deformation field to maintain the continuity and naturalness of the deformation field. These new loss functions and descriptors enhance the performance of the registration network model, making it more effective and robust in medical image registration tasks.

[0099] In one embodiment, S2 includes:

[0100] The images in the training set and test set are cropped, resampled, normalized, and motion aligned to adjust the images to a uniform size and resolution.

[0101] For the training set, fixed images and floating images are selected by cross selection;

[0102] For the test set, one image is randomly selected as the fixed image and the remaining images are used as floating images.

[0103] In one embodiment, S3 includes:

[0104] The fixed image is interpolated by trilinear interpolation. and floating images Downsample to obtain and ,in, For the Fixed image of the layer, For the Floating image of the layer;

[0105] Based on the fixed image and floating image of each layer, an L-layer image pyramid is constructed, where the fixed image and floating image of the last layer of the pyramid are , ;

[0106] Will and connection, as a transformer-based registration network The input image of the stage.

[0107] In the specific implementation process, The downsampling scale factor of the layer pyramid is .

[0108] In one embodiment, S4 includes:

[0109] The image transformation step is used to transform and The concatenated image is fed into a pre-built transformer-based registration network to obtain the Deformation field of the stage ;

[0110] The spatial transformation and connection steps are used to transform With Floating image of the layer image pyramid Perform spatial transformation to obtain the image after spatial transformation , then connect and , as the first The input image of the stage, For the Fixed image of the layer;

[0111] No. The first stage is obtained by performing the operations of the image transformation step and the space transformation and connection step. Deformation field of the stage and the input image of the next stage, repeating this operation until the deformation field of the Lth stage is obtained , as the final deformation field.

[0112] In the specific implementation process, there is a one-to-one correspondence between the layers and the stages, that is, the first layer( ) Fixed image and The image obtained after concatenating the layer floating images is used as the model (transformer-based registration network) The input image of the stage.

[0113] Each stage of the image transformation step and the space transformation and connection step is executed cyclically. The image transformation step obtains the registration result of the current stage, that is, the deformation field of the current stage. The space transformation and connection step combines the deformation field of the current stage with the floating image of the image pyramid of the next layer. Perform spatial transformation to obtain a floating image after spatial transformation, and then connect the floating image after spatial transformation and the fixed image of the image pyramid of the next layer as the input image of the next stage. Among them, for the formation of the deformation field and feature embedding of the first stage, since the first layer has no previous layer (0th layer), there is only the next layer. Therefore, for the first stage, there is no step of adding the feature embedding and deformation field of the previous layer. In addition, since the deformation field of the first layer does not need to be combined with other information, the deformation field generated is the final deformation field of the layer (not temporary). The temporary deformation field means that it is not the final deformation field of the layer, and it needs to be added to the final deformation field of the previous layer.

[0114] For the second stage to the Lth stage, the image transformation step and the spatial transformation and connection step are performed to obtain the first Deformation field of the stage and the input image of the next stage, repeating this operation until the deformation field of the Lth stage is obtained , as the final deformation field.

[0115] See also Figure 1 , fixed image and floating images After downsampling, we get and ,Will and Connect to get the input image of the first stage of the transformer-based registration network. and Connect to get the input image of the second stage of the transformer-based registration network, and so on, to get the input image of the third stage of the transformer-based registration network. The input image of each stage is input to the image block embedding layer of the corresponding stage of the transformer-based registration network for processing.

[0116] Will and The concatenated image is fed into a pre-built transformer-based registration network to obtain the deformation field of the first stage. ;Will Floating image with level 2 image pyramid Perform spatial transformation to obtain the image after spatial transformation , then connect and , as the input image for the 2nd stage.

[0117] In the second stage, the image transformation steps are followed to obtain the Deformation field of the stage Then, the input image of the third stage is obtained through the operation of spatial transformation and connection steps, and this operation is repeated until the deformation field of the third layer of pyramid is obtained. ,Will As the final deformation field. The present invention uses three input images of different scales to perform three stages of registration. The transformer-based registration network uses three continuously stacked image registration structures (each image registration structure includes an image block embedding layer, a dual attention transformer and a deformable registration head), so as to achieve better registration effect.

[0118] In general, the overall concept of the present invention is: first, use the input image of the minimum size ( and The connected image) is registered in the first stage and the deformation field is output , then, the medium-sized input image ( and The concatenated image) and the deformation field of the first stage After fusion, the second stage of registration is performed to output the intermediate deformation field , and then convert the image to the same size as the original input image ( and The connected image) is fused with the deformation field of the second stage and then the third stage of registration is performed to output the final deformation field Then, based on the spatial transformation module and the final deformation field, a registered image is generated.

[0119] In one embodiment, the image transformation step includes:

[0120] The transformer-based registration network The input image of the stage is represented as ,in , and express height, width and depth, is the number of channels, and the image block embedding layer uses convolution operation to Convert to image patch embedding map , and then Flattened into a series of block embeddings ,in, Used to aggregate into block embedding matrix , i An index representing the number of image blocks, , and Indicates iThe height, width and depth of the image block, represents the embedding dimension, Indicates i The number of image blocks;

[0121] In the block embedding matrix After adding the position embedding, it is input into the dual attention transformer of the same stage, where each dual attention module of the stage contains a self-attention mechanism and A cross-attention mechanism;

[0122] The block embedding matrix after adding the position embedding first passes through the linear projection matrix , , The transformation of ,key Sum The matrix is ​​then calculated through the self-attention mechanism, where the The calculation formula of the self-attention mechanism in the stage is:

[0123] (1)

[0124] for The characteristic dimension of Indicates The block embedding matrix after adding the position embedding of the stage;

[0125] The cross attention calculation is performed based on the output of the self-attention mechanism. The calculation formula of the cross attention mechanism is:

[0126] (2)

[0127] , Represents the two input matrices of the cross attention mechanism, Indicates the number of a stage. represents the query matrix obtained after the first stage of self-attention mechanism, Represents the value matrix and key matrix obtained after the self-attention or cross-attention mechanism of the jth stage. After the self-attention mechanism and cross-attention mechanism of the stage, the feature matrix is ​​obtained ;

[0128] Input the feature matrix into the feed-forward module to generate feature embedding , where the feedforward module consists of a two-hidden layer Block, add three dilated convolutions between hidden layers, feature matrix After the first hidden layer and activation function processing, the processing result is then sent to the dilated convolution layer to obtain multi-scale feature representation and finally generate feature embedding ;

[0129] Embedding features Send it to the deformable registration head and get the Deformation field of the stage , where the deformable registration head is implemented based on a 3×3×3 convolutional layer.

[0130] See also Figure 2 , is a schematic diagram of the dual attention converter structure used in an embodiment of the present invention.

[0131] Specifically, First, through the linear projection matrix , , The transformation results in different queries ,key Sum The matrix is: Then the self-attention mechanism is used for calculation, and the output of the self-attention mechanism is used as the input of the cross-attention mechanism. Finally, the feature matrix is ​​obtained through the cross-attention mechanism. In the first stage, one self-attention calculation is performed, in the second stage, one self-attention calculation and one cross-attention calculation are performed, and in the third stage, one self-attention and two cross-attention calculations are performed.

[0132] Indicates the number of the attention mechanism of a stage, that is, it indicates the number of attention mechanisms of a stage. If it is stage 1, then Can only be 1, indicating the first attention mechanism of this stage, that is, the self-attention mechanism. If it is the second stage, then The value of is 1 or 2. = 1, it indicates the first attention mechanism in the second stage, namely the self-attention mechanism. =2, it indicates the first attention mechanism in the second stage, i.e., the cross attention mechanism; if it is the third stage, then The value of is 1, 2 or 3. = 1, it indicates the first attention mechanism in the third stage, namely the self-attention mechanism. =2, it indicates the second attention mechanism in the third stage, that is, the first cross attention mechanism in this stage. =3, it indicates the third attention mechanism in the third stage, that is, the second cross attention mechanism in this stage.

[0133] Among them, the specific process of the cross attention mechanism is as follows:

[0134] The output of the self-attention mechanism is used as input for the The calculation formula of the s-th cross attention mechanism in the stage is:

[0135]

[0136] in, Respectively represent Phase The input of the attention mechanism and the Phase The input of the attention mechanism. hour, is the input of the self-attention mechanism; otherwise is the input of the cross attention mechanism. The stage goes through a self-attention mechanism and After the cross attention mechanism, we get The characteristic matrix of the stage .

[0137] More specifically, when , it means that in the first stage, one self-attention calculation is performed and the result is when , indicating that in the second stage, a self-attention calculation is first performed to obtain the result Then the output of this stage when s=1 As the cross-attention K and V and the output of the previous stage As Q, we can perform cross attention calculation and get ,when , indicating that in the third stage, a self-attention calculation is first performed to obtain the result Then the output at this stage when s=1 is As the cross attention K and V and the output of the first stage As Q, we can perform cross attention calculation and get , and then As the K and V of the cross attention and the output of the second stage As Q, we can perform cross attention calculation and get , as the first The characteristic matrix of the stage .

[0138] In one embodiment, The first stage is obtained by performing the operations of the image transformation step and the space transformation and connection step. Deformation field of the stage and the input image for the next stage, including:

[0139] Will Increment to Then repeat the image transformation steps to get The block embedding matrix of the stage , Compared with the feature embedding in the previous stage After addition, the corresponding dual attention transformer is input. In the dual attention transformer, the output of the attention layer of the previous stage is used as the first The input of the cross attention calculation of the stage is the feature matrix output by the dual attention module Sequentially sent to the feedforward module and deformable registration head at the same stage, the Feature embedding of the stage and Temporary deformation field of the stage ;

[0140] Will Increment to Then repeat the spatial transformation and connection steps to transform the deformation field of the previous stage Upsample and Add up to get The final deformation field of the stage .

[0141] It should be noted that It refers to the block embedding matrix after adding position embedding.

[0142] In one embodiment, the calculation of the objective function includes:

[0143] Get the registered floating image With fixed image The normalized total gradient between:

[0144] (3)

[0145] in, Indicates along The derivative of the direction , The difference image between the registered floating image and the fixed image;

[0146] Get the registered floating image With fixed image The modality-independent neighborhood descriptor between:

[0147] (4)

[0148] in, , yes , The number of pixels, Compute the modality-independent neighborhood descriptor of the image;

[0149] Get the registered floating image With fixed image between The contrast loss of the corresponding feature points of the group:

[0150] (5)

[0151] in, Represents the index number of the k-group feature point, k is set to 12, and for and The corresponding registration point pairs between and for and The temperature coefficient of the non-corresponding registration point pairs It is a hyperparameter used to balance the convergence speed and stability of the model;

[0152] Deformation Field The smoothing regularization term is:

[0153] (6)

[0154] in, Represents the deformation field The gradient of

[0155] The objective function is calculated based on the normalized total gradient, the modality-independent neighborhood descriptor, the contrast loss, and the smooth regularization term of the deformation field:

[0156] (7)

[0157] in, is the weight of the normalized total gradient, is the weight of the modality-independent neighborhood descriptor, is the weight of the contrast loss, is the weight of the smooth regularization term of the deformation field.

[0158] See also Figure 3 , which is a schematic diagram of the normalized total gradient according to an embodiment of the present invention.

[0159] In the specific implementation process, the network model is back-propagated and optimized through the objective function until the calculated objective function value no longer decreases or the network training reaches the preset number of training iterations, the neural network training is completed, and the trained neural network model and the optimal deformation field are obtained. .

[0160] The 3D medical image registration method enhanced by dual attention transformer contrast learning provided by the present invention is described below through a specific example.

[0161] The embodiment uses the public OASIS and LPBA three-dimensional medical images as data sets, such as Figure 4 As shown in the figure, the OASIS dataset contains brain images and segmentation maps of 414 normal elderly people and patients with different degrees of Alzheimer's disease, and the LPBA dataset collects brain images and manual segmentation maps of 40 healthy adults. The specific steps are as follows:

[0162] Step 1: Image acquisition and data set division

[0163] Firstly, the public dataset OASIS was divided into a training set of 331 cases and a test set of 83 cases in a ratio of 8:2. Then, random and non-repetitive pairing was performed in the training set, test set and LPBA dataset of OASIS. 4000 data pairs to be registered were generated in the OASIS training set for training, and 1000 data pairs to be registered were generated in the OASIS test set and LPBA dataset for testing respectively.

[0164] Step 2: Image preprocessing

[0165] Step S2.1: Perform standard preprocessing on the divided datasets. The resolution of each image is kept at 1 mm × 1 mm × 1 mm. For the OASIS dataset, the image size is cropped to 128 × 128 × 128. For the LPBA dataset, the image size is cropped to 160 × 192 × 160. The images are normalized and affine aligned during training and testing.

[0166] Step S2.2: For the training set, select pairs of images by cross selection and use them as fixed images and floating images ;

[0167] Step S2.3: For the test set, arbitrarily select an image as a fixed image , and the rest as floating images .

[0168] Step 3: Fix each pair of images and floating images Downsample to obtain and , to construct an L-layer image pyramid, connecting and , as the registration network The input image of the stage.

[0169] Step S3.1: Fix the image by trilinear interpolation and float Down-sampling is performed to construct an L-layer image pyramid, where Set to 3, The downsampling scale factor of the layer pyramid is , the fixed image and floating image of the last layer of pyramid are , .

[0170] Step 4: Input the image into The image block embedding layer of the stage obtains the block embedding matrix ,Will Input dual attention transformer to get feature embedding ,Will Input into the deformable registration head and get Deformation field of the stage .

[0171] Step 4.1: The input image of the stage is represented as ,in , and express height, width and depth, is the number of channels, and the image block embedding layer uses convolution operation to Convert to image patch embedding map , and then Flattened into a series of block embeddings ,These Can be aggregated into block embedding matrices , where i represents the index of the image block number, , and represents the height, width and depth of the i-th image block, represents the embedding dimension, represents the number of blocks of the i-th image block. For the OASIS dataset, , , , , and They are set to 128, 128, 128, 3, 4096 and 256 respectively. For the LPBA dataset, , , , , and Set to 160, 192, 160, 3, 4096 and 256 respectively;

[0172] Step 4.2: In the block embedding matrix After adding the position embedding, it is input into the dual attention transformer of the same stage, which consists of a dual attention module and a feedforward module;

[0173] Step 4.3: Calculate the corresponding output through the self-attention mechanism;

[0174] Step 4.4: Use the output of the self-attention mechanism as the input of the cross-attention mechanism to obtain the output of the cross-attention mechanism; After the dual attention module, the feature matrix is ​​obtained ;

[0175] Step 4.5: Generate feature embeddings through the feed-forward module ;

[0176] Step 4.6: Embedding features This is fed into a deformable registration head to obtain the deformation field, which is implemented by a 3×3×3 convolutional layer.

[0177] Step 5: With Floating image of the layer image pyramid Perform spatial transformation to obtain , then connect and , as the first The input image of the stage.

[0178] Step 6: According to steps 4 and 5, we get Deformation field of the stage and the input image for the next stage.

[0179] Step 7, repeat step 6 until the deformation field of stage L is obtained .

[0180] Step 8, Utilize For floating images Interpolate to get the registered floating image , obtained by point registration and Between The corresponding feature points of the group are calculated using contrast loss, and the normalized total gradient and modality-independent neighborhood descriptor are used to calculate With fixed image The similarity between , and Establish an objective function, and use the objective function to perform back propagation optimization on the network model to obtain a trained neural network model. Use the objective function to perform back propagation optimization on the network model until the calculated objective function value no longer decreases or the network training reaches the preset number of training iterations, the neural network training is completed, and a trained neural network model and the optimal deformation field are obtained. , where the batch size is set to 1, the initial learning rate is 0.0001, the maximum number of iterations is 160000, and the optimization algorithm of the model is Adam optimizer.

[0181] Step 9: Input the test set into the trained neural network model to obtain the optimal deformation field ,use For floating images The deformable registration is performed to obtain the final registered image. The comprehensive qualitative results of the present invention and the benchmark method are as follows: Figure 5 shown.

[0182] In order to verify the advantages of the position-aware deformable medical image registration method proposed in the present invention, the OASIS and LPBA three-dimensional medical image datasets are used to analyze and compare this example method with other methods, as shown in Table 1.

[0183] Table 1 Comparison of registration results of different methods

[0184]

[0185] Among them, the voxel morphology method can be found in the paper "Learning Framework for Deformable Medical Image Registration" published by Balakrishnan et al. (IEEE Transactions on Medical Imaging, 2019, pp.1788–1800); the self-supervised anatomical embedding enhancement method can be found in the paper "Deformable Image Registration Based on Self-Supervised Anatomical Embedding" published by Liu et al. (International Association for Medical Image Computing and Computer-Assisted Intervention, 2021, pp.87-97); the coarse-to-fine transformer method can be found in the paper "Affine Medical Image Registration Based on Coarse-Fine Visual Transformation" published by Mok et al. (International Conference on Computer Vision and Pattern Recognition, 2022, pp.20835-20844); the deformable image registration method can be found in the paper "Unsupervised Deformable Image Registration for Missing Correspondence in Preoperative and Postoperative Recurrent Brain Tumor MRI Scans" published by Mok et al. (International Conference on Medical Image Computing and Computer-Assisted Intervention, 2022, pp.25-35);

[0186] It can be seen from the experimental results in Table 1 that the method of the present invention shows the best registration performance on both the OASIS dataset and the LPBA dataset. This is due to the novelty of the method of the present invention in feature extraction, global dependency capture and loss function design. Compared with the convolutional neural network structure of voxel morphology and deformable image registration, the method of the present invention adopts a transformer structure based on dual attention dilated convolution, which can better capture local and global feature information, learn long-range dependencies, and handle more complex deformations of human body parts. In the experiments of the two datasets, the method of the present invention has better results in Dice similarity coefficient and Hausdorff distance indicators compared with voxel morphology and deformable image registration methods; compared with self-supervised anatomical embedding enhancement, the registration performance is significantly improved; compared with the coarse-to-fine transformer based on normalized cross-correlation, the method of the present invention adopts normalized total gradient to more comprehensively consider the gradient information of the image, reduce the influence of factors such as noise on the registration method, capture more details and nonlinear features, and achieve better registration effect. It further illustrates that the method of the present invention has a strong perception ability for local and global information of medical images and can effectively register three-dimensional medical images.

[0187] Embodiment 2

[0188] Based on the same inventive concept, this embodiment discloses a 3D medical image registration device enhanced by contrastive learning of dual attention transformers, see Figure 6 ,include:

[0189] The data set acquisition and division module 601 is used to acquire a 3D medical image data set and divide it into a training set and a test set;

[0190] A data preprocessing module 602 is used to preprocess the divided training set and test set, and select fixed images and floating images from the preprocessed training set;

[0191] An image pyramid building module 603 is used to downsample each pair of fixed image and floating image to create an image pyramid, and connect the fixed image and floating image of each layer in the pyramid;

[0192] An image transformation module 604 is used to input the connected image into a pre-built transformer-based registration network to obtain a deformation field of the pyramid, wherein the transformer-based registration network includes three stages, each stage uses the same module, each stage includes an image block embedding layer, a dual attention transformer and a deformable registration head, the image block embedding layer is used to process the connected image to obtain a block embedding matrix, the dual attention transformer includes a dual attention module and a feedforward module, the dual attention module is used to process the block embedding matrix using an inter-layer attention mechanism and a self-attention mechanism to obtain a feature matrix, the feedforward module is based on dilated convolution, and is used to obtain feature embedding according to the feature matrix, and the deformable registration head is used to obtain a final deformation field according to the feature embedding;

[0193] The model training module 605 is used to interpolate the floating image using the final deformation field to obtain the registered floating image, and obtain the value between the registered floating image and the fixed image through point registration. The corresponding feature points of the group are calculated using contrast loss, and the similarity between the feature points is calculated using the normalized total gradient and the modality-independent neighborhood descriptor. The objective function is established by the similarity between the corresponding feature points of the groups and the similarity between the registered floating image and the fixed image. The transformer-based registration network is back-propagated and optimized through the objective function to obtain a trained neural network model.

[0194] The image registration module 606 is used to input the test set into the trained neural network model to obtain the optimal deformation field, and use the optimal deformation field to perform deformable registration on the floating image to obtain the final registered image.

[0195] Since the device introduced in the second embodiment of the present invention is a device used to implement the 3D medical image registration method enhanced by contrastive learning of dual attention transformers in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the device, so it is not repeated here. All devices used in the method in the first embodiment of the present invention belong to the scope of protection of the present invention.

[0196] Embodiment 3

[0197] Based on the same inventive concept, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the first embodiment is implemented.

[0198] Since the computer-readable storage medium introduced in the third embodiment of the present invention is a computer-readable storage medium used to implement the 3D medical image registration method enhanced by contrastive learning of dual attention transformers in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the computer-readable storage medium, so it is not repeated here. All computer-readable storage media used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0199] Embodiment 4

[0200] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in Embodiment 1 when executing the program.

[0201] Since the computer device introduced in the fourth embodiment of the present invention is a computer device used to implement the 3D medical image registration method enhanced by contrastive learning of dual attention transformers in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the computer device, so it is not repeated here. All computer devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0202] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic creative concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A 3D medical image registration method enhanced by contrastive learning with dual attention transformers, characterized in that: include: Acquire 3D medical image data and divide it into training set and test set; Preprocessing the divided training set and test set, and selecting fixed images and floating images from the preprocessed training set; Downsampling each pair of fixed and floating images to create an image pyramid, and concatenating the fixed and floating images at each level of the pyramid; The concatenated image is input into a pre-built transformer-based registration network to obtain a deformation field of the pyramid, wherein the transformer-based registration network includes three stages, each stage uses the same module, each stage includes an image block embedding layer, a dual attention transformer and a deformable registration head, the image block embedding layer is used to process the concatenated image to obtain a block embedding matrix, the dual attention transformer includes a dual attention module and a feedforward module, the dual attention module is used to process the block embedding matrix using an inter-layer attention mechanism and a self-attention mechanism to obtain a feature matrix, the feedforward module is based on dilated convolution, and is used to obtain feature embedding according to the feature matrix, and the deformable registration head is used to obtain the final deformation field according to the feature embedding; The floating image is interpolated using the final deformation field to obtain the registered floating image, and the correlation between the registered floating image and the fixed image is obtained by point registration. The corresponding feature points of the group are calculated using contrast loss, and the similarity between the feature points is calculated using the normalized total gradient and the modality-independent neighborhood descriptor. The objective function is established by the similarity between the corresponding feature points of the groups and the similarity between the registered floating image and the fixed image. The transformer-based registration network is back-propagated and optimized through the objective function to obtain a trained neural network model. The test set is input into the trained neural network model to obtain the optimal deformation field, and the floating image is deformably registered using the optimal deformation field to obtain the final registered image.

2. The 3D medical image registration method enhanced by dual attention transformer contrast learning as claimed in claim 1, characterized in that: The divided training set and test set are preprocessed, and fixed images and floating images are selected from the preprocessed training set, including: The images in the training set and test set are cropped, resampled, normalized, and motion aligned to adjust the images to a uniform size and resolution. For the training set, fixed images and floating images are selected by cross selection; For the test set, one image is randomly selected as the fixed image and the remaining images are used as floating images.

3. The 3D medical image registration method enhanced by dual attention transformer contrast learning as claimed in claim 2, characterized in that: Each pair of fixed and floating images is downsampled to create an image pyramid, and the fixed and floating images at each level are concatenated, including: The fixed image is interpolated by trilinear interpolation. and floating images Downsample to obtain and ,in, For the Fixed image of the layer, For the Floating image of the layer; Based on the fixed image and floating image of each layer, an L-layer image pyramid is constructed, where the fixed image and floating image of the last layer of the pyramid are , ; Will and Connection, where The concatenated image is used as the transformer-based registration network The input image of the stage.

4. The 3D medical image registration method enhanced by dual attention transformer contrast learning as claimed in claim 3, characterized in that: The concatenated images are fed into a pre-built transformer-based registration network to obtain the pyramid deformation field, including: The image transformation step is used to transform and The concatenated image is fed into a pre-built transformer-based registration network to obtain the Deformation field of the stage ; The spatial transformation and connection steps are used to transform With Floating image of the layer image pyramid Perform spatial transformation to obtain the image after spatial transformation , then connect and , as the first The input image of the stage, For the pyramid Fixed image of the layer; No. The first stage is obtained by performing the operations of the image transformation step and the space transformation and connection step. Deformation field of the stage and the input image of the next stage, repeating this operation until the deformation field of the Lth stage is obtained , as the final deformation field.

5. The 3D medical image registration method enhanced by dual attention transformer contrast learning as claimed in claim 4, characterized in that: The image transformation steps include: The transformer-based registration network The input image of the stage is represented as ,in , and express height, width and depth, is the number of channels, and the image block embedding layer uses convolution operation to Convert to image patch embedding map , and then Flattened into a series of block embeddings ,in, Used to aggregate into block embedding matrix , i The index representing the number of image blocks, , and Indicates i The height, width and depth of the image block, represents the embedding dimension, Indicates i The number of image blocks; After adding the position embedding to the block embedding matrix, it is input into the dual attention transformer of the same stage, where each dual attention module contains a self-attention mechanism and A cross-attention mechanism; The block embedding matrix after adding the position embedding first passes through the linear projection matrix , , The transformation of ,key Sum The matrix is ​​then calculated through the self-attention mechanism, where the The calculation formula of the self-attention mechanism in the stage is: （1） for The characteristic dimension of Indicates The block embedding matrix after adding the position embedding of the stage, Indicates the total number of stages; The cross attention calculation is performed based on the output of the self-attention mechanism. The calculation formula of the cross attention mechanism is: （2） , Represents the two input matrices of the cross-attention mechanism, Indicates the number of a stage. represents the query matrix obtained after the first stage of self-attention mechanism, Represents the value matrix and key matrix obtained after the second or third stage of self-attention or cross-attention mechanism. After the self-attention mechanism and cross-attention mechanism of the stage, the feature matrix is ​​obtained ; Input the feature matrix into the feed-forward module to generate feature embedding , where the feedforward module consists of a two-hidden layer Block, add three dilated convolutions between hidden layers, feature matrix After being processed by the first hidden layer and activation function, the processed result is then sent to the dilated convolution layer to obtain multi-scale feature representation and finally generate feature embedding ; Embedding features Send it to the deformable registration head and get the Deformation field of the stage , where the deformable registration head is implemented based on a 3×3×3 convolutional layer.

6. The 3D medical image registration method enhanced by dual attention transformer contrast learning as claimed in claim 4, characterized in that: No. The first stage is obtained by performing the operations of the image transformation step and the space transformation and connection step. Deformation field of the stage and the input image for the next stage, including: Will Increment to Then repeat the image transformation steps to get The block embedding matrix of the stage , Compared with the feature embedding in the previous stage After addition, the corresponding dual attention transformer is input. In the dual attention transformer, the output of the attention layer of the previous stage is used as the first The input of the cross attention calculation of the stage is the feature matrix output by the dual attention module Sequentially sent to the feedforward module and deformable registration head at the same stage, the Feature embedding of the stage and Temporary deformation field of the stage ; Will Increment to Then repeat the spatial transformation and connection steps to transform the deformation field of the previous stage Upsample and Add up to get The final deformation field of the stage .

7. The 3D medical image registration method enhanced by dual attention transformer contrast learning as claimed in claim 1, characterized in that: The calculation of the objective function includes: Get the registered floating image With fixed image The normalized total gradient between: （3） in, Indicates along The derivative of the direction , The difference image between the registered floating image and the fixed image; Get the registered floating image With fixed image The modality-independent neighborhood descriptor between: （4） in, , yes , The number of pixels, Compute the modality-independent neighborhood descriptor of the image; Get the registered floating image With fixed image between The contrast loss of the corresponding feature points of the group: （5） in, Represents the index number of k groups of feature points and for and The corresponding registration point pairs between and for and The temperature coefficient of the non-corresponding registration point pairs It is a hyperparameter used to balance the convergence speed and stability of the model; Deformation Field The smoothing regularization term is: （6） in, represents the gradient of the deformation field; The objective function is calculated based on the normalized total gradient, the modality-independent neighborhood descriptor, the contrast loss, and the smooth regularization term of the deformation field: （7） in, is the weight of the normalized total gradient, is the weight of the modality-independent neighborhood descriptor, is the weight of the contrast loss, is the weight of the smooth regularization term of the deformation field.

8. A 3D medical image registration device enhanced by dual attention transformer contrast learning, characterized in that: include: The data set acquisition and division module is used to acquire 3D medical image data and divide it into training sets and test sets; A data preprocessing module is used to preprocess the divided training set and test set, and select fixed images and floating images from the preprocessed training set; An image pyramid building module, for downsampling each pair of fixed image and floating image to create an image pyramid, and connecting the fixed image and floating image at each layer; An image transformation module is used to input the connected image into a pre-built transformer-based registration network to obtain a deformation field of the pyramid, wherein the transformer-based registration network includes three stages, each stage uses the same module, each stage includes an image block embedding layer, a dual attention transformer and a deformable registration head, the image block embedding layer is used to process the connected image to obtain a block embedding matrix, the dual attention transformer includes a dual attention module and a feedforward module, the dual attention module is used to process the block embedding matrix using an inter-layer attention mechanism and a self-attention mechanism to obtain a feature matrix, the feedforward module is based on dilated convolution, and is used to obtain feature embedding according to the feature matrix, and the deformable registration head is used to obtain a final deformation field according to the feature embedding; The model training module is used to interpolate the floating image using the final deformation field to obtain the registered floating image, and obtain the correlation between the registered floating image and the fixed image through point registration. The corresponding feature points of the group are calculated using contrast loss, and the similarity between the feature points is calculated using the normalized total gradient and the modality-independent neighborhood descriptor. The objective function is established by the similarity between the corresponding feature points of the groups and the similarity between the registered floating image and the fixed image. The transformer-based registration network is back-propagated and optimized through the objective function to obtain a trained neural network model. The image registration module is used to input the test set into the trained neural network model to obtain the optimal deformation field, and use the optimal deformation field to perform deformable registration on the floating image to obtain the final registered image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the 3D medical image registration method enhanced by dual attention transformer contrast learning as described in any one of claims 1 to 7 is implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the 3D medical image registration method enhanced by dual attention transformer contrast learning is implemented as described in any one of claims 1 to 7.

Images