A registration method for three-dimensional nuclear magnetic resonance images

Abstract

The application discloses a registration method for three-dimensional MRI images, which is characterized in that a two-way self-attention network is used to model the relationship between different regions in the image and between images, a registration network model is constructed based on a symmetrical encoding-decoding network 3D U-net with long residual connection, and the two-way self-attention network is used to enhance the feature embedding of three-dimensional images, and the local features of the convolutional neural network and the global feature context relationship across images are used for enhancement. The method can enhance the modeling and identification of the semantic correspondence of anatomical structures in the differential homeomorphism registration by building a registration network model, and improve the registration accuracy of three-dimensional MRI images.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a registration method for three-dimensional magnetic resonance imaging (MRI) images. Background Technology

[0002] Deformable image registration establishes nonlinear voxel correspondences between images and is fundamental to many medical image analysis tasks, such as anatomical structure annotation based on atlas images, statistical shape analysis, and anatomical structure comparison. Identifying the semantic correspondences between fine structures in images is a key issue in deformable registration. Due to the differences in position, pose, size, and shape of the images, establishing reliable correspondences between anatomical structures remains a very challenging task. Since traditional registration methods rely on time-consuming large-scale parameter iteration optimization, convolutional neural networks have been used in deformable image registration to predict the deformation field or differential homeomorphism velocity field in an end-to-end manner. However, these methods only utilize the local embedding capability of convolution operations, establishing contextual relationships of different sizes within the image through the concatenation of multiple convolutional layers, lacking processing of global relationships between images, and thus may generate unreasonable voxel correspondences. To alleviate registration getting trapped in local extrema, preprocessing for affine registration or using the distance of keypoint annotations as a loss function during registration is required.

[0003] Since their inception, deep self-attention transform networks (Transformers) have demonstrated remarkable performance in applications including natural language processing, text translation, and question answering. Recent research has also incorporated deep self-attention transform networks into computer vision tasks such as object detection, image recognition, and image segmentation. Deep self-attention transform networks facilitate the global modeling of relationships between different regions in an image. In image processing, attention mechanisms enhance task-critical features and suppress irrelevant features. In medical image registration, related work uses attention-driven hierarchical registration strategies and greedy supervised methods to address rigid CT image registration problems. In X-ray and CT image registration, automatic attention mechanisms are introduced across multiple regions to find reliable visual cues. However, existing attention mechanisms in these methods only address dependencies within a single image or solve rigid registration problems, failing to address cross-image semantic correspondences in deformable registration problems. Current techniques still struggle to address how to leverage global dependencies between images to enhance the accuracy of anatomical structure correspondences. Summary of the Invention

[0004] This invention provides a registration method for three-dimensional magnetic resonance imaging (MRI) images, aiming to overcome the above-mentioned defects of the prior art and achieve efficient and accurate correspondence solving and registration of three-dimensional images.

[0005] To achieve the above objectives, this invention establishes a differential homeomorphic registration frame (registration network) with a dual-path self-attention network. The proposed method uses the self-attention mechanism to explicitly model intra-image and cross-image contextual relationships. The dual-path self-attention network performs relationship modeling and feature enhancement on two image embedding methods to find semantic correspondences of anatomical structures. The established registration network includes a learnable image embedding module, a cross-image relationship learning module, and a deformation field inference module. The combination of the two image embedding methods utilizes both low-level local spatial features and high-level global correlation features of the network to infer the deformation field of the registration. The registration network established in this invention is trained in an unsupervised learning manner to solve the deformable registration problem. The difficulty of unsupervised deformable registration lies in identifying the semantic correspondences between anatomical structures in images. The registration network of this invention handles cross-image and global contextual correlations, improving the discriminative ability of image feature embedding in voxel correspondence solving. The main innovation of this invention lies in proposing a novel dual-path deep self-attention network for differential homeomorphic registration of 3D images, and promoting the semantic correspondence of anatomical structures in images through global relationship modeling across images. The image feature embedding leverages both the ability of convolutional networks to extract local features and global and cross-image correlations based on attention mechanisms.

[0006] In this invention, the following terms are used to define the following: Magnetic Resonance Imaging (MRI) and Deep Self-Attention Network (Transformer).

[0007] The technical solution provided by this invention is:

[0008] A registration method for three-dimensional brain MRI images is proposed, which creates a registration network model and uses a dual-path self-attention network to explicitly model different regions within the image and the relationships between images. This enhances the ability to identify and model the semantic correspondence of anatomical structures in differential homeomorphic registration, thereby improving registration accuracy. The method includes the following steps:

[0009] 1) Registration network model construction, including the following process:

[0010] 11) A registration network is constructed based on a symmetric encoder-decoder network with long residual connections. A dual-path self-attention network is used to enhance the feature embedding of the 3D image. Simultaneously, local features from a convolutional neural network and global feature contextual relationships across images are utilized for enhancement. Given the network input, a floating image V... m and fixed image V fThe goal is to estimate the differential homeomorphic registration field ψ to establish a dense voxel correspondence between images. The registration framework of this invention consists of a learnable image embedding module, a dual-path cross-image relation learning module, and a deformation field inference module.

[0011] 12) Create an image feature embedding module:

[0012] 13) Create a dual-path self-attention feature enhancement module:

[0013] 14) Create a differential homeomorphism registration module:

[0014] Differential homeomorphisms are differentiable and invertible, which gives differential homeomorphic registration the properties of one-to-one mapping and topological preservation. Differential homeomorphic registration predicts the static velocity field v. t Where t∈[0,1], the velocity field v t and deformation field ψ t satisfy:

[0015]

[0016] In Formula 1, t is the time parameter, t∈[0,1], v t For the velocity field, ψ t For deformation field, symbol The combination of mappings is represented by ψ0 = I, which is the identity transformation.

[0017] This invention uses a convolutional decoder network φ r Receive enhanced image embedding and The differential homeomorphic velocity field v is predicted, and the velocity field v is integrated using a scaling-square layer, ψ1=exp(v), to obtain the final registration deformation field ψ1.

[0018] 2) Training the registration network

[0019] 3) Perform online registration of the 3D MRI images to be registered:

[0020] During the online registration stage, any pair of floating images V m and fixed image V f Input the registration network trained in step 2), output the deformation field ψ, and deform the floating image to obtain the registered image. V′ m With V f The more similar the two pairs, the better the registration result.

[0021] The above steps enable efficient online registration of brain MRI images.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] The registration method for three-dimensional brain MRI images provided by this invention clearly models the cross-image anatomical structure dependencies in the registration network, thereby improving the registration accuracy and achieving efficient online registration of three-dimensional brain MRI images. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the structural framework of the registration network model corresponding to this invention;

[0025] Among them, V m For floating images; V f For a fixed image; φ s This is the first branch of the convolutional neural network encoder; φ c For the second branch of the convolutional neural network; F m For floating image V m Individual feature embeddings; F f For fixed image V f Individual feature embeddings; F c For V m and V f Image stitching is used as the input feature embedding; T s This is the first branch of a deep self-attention network; T c This is the second deep self-attention network branch; Image feature embedding for enhancement; v is the velocity field; ψ is the registration deformation field; L reg L is the loss function for the regularization term of the velocity field v; sim Image similarity measurement loss function

[0026] Figure 2 This is a schematic diagram of the dual-path self-attention feature enhancement module in the registration network model created in this invention.

[0027] Among them, F f For fixed image V f Individual feature embeddings; F m For floating image V m Individual feature embeddings; F c For V m and V f Image stitching is used as the input feature embedding; For (F) f ,F m Enhanced feature embedding; For F c Enhanced feature embedding; Embedding for final enhanced image features

[0028] Figure 3This is a schematic diagram of the registration result and registration deformation field visualization of a pair of three-dimensional brain MRI images in an embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.

[0030] This invention provides a registration method for three-dimensional MRI images. It utilizes a dual-path self-attention network to explicitly model different regions within the image and the relationships between images, creating a registration network model. This enhances the ability to identify and model the semantic correspondence of anatomical structures in differential homeomorphic registration, thereby improving registration accuracy.

[0031] The following examples demonstrate the registration and quantitative verification of three-dimensional MRI images of the brain, including the following steps:

[0032] 1) Registration network model construction, including the following process:

[0033] 11) The structure of the registration network model is shown in the attached figure. Figure 1 As shown, this invention constructs a registration network based on a symmetric encoder-decoder network with long residual connections, and utilizes a dual-path self-attention network to enhance the feature embedding of 3D images. It also leverages local features from convolutional neural networks and enhances global feature context relationships across images. Given the network input, a floating image V... m and fixed image V f The goal is to estimate the differential homeomorphic registration field ψ to establish a dense voxel correspondence between images. The registration network model framework of this invention consists of a learnable image feature embedding module, a dual-path cross-image relation learning module (dual-path self-attention feature enhancement module), and a deformation field inference module (differential homeomorphic registration module).

[0034] 12) Create an image feature embedding module:

[0035] The registration network of this invention uses 3D U-net (a symmetric encoder-decoder network with long residual connections) to embed features of individual and stitched 3D images, thereby creating an image feature embedding module. For example... Figure 1 As shown, the image feature embedding module created by this invention includes a first convolutional neural network encoder branch and a second convolutional neural network branch. The first convolutional neural network encoder branch φ s It receives a single-channel image as input and outputs a floating image V. m Individual feature embedding F m and fixed image V f Individual feature embedding F f In addition to single-channel 3D image embedding, a second convolutional neural network branch φ is used.c Extract the feature embedding F from the concatenation of the floating and fixed images as input. c .

[0036] 13) Create a dual-path self-attention feature enhancement module:

[0037] This invention creates a dual-path self-attention feature enhancement module to model cross-image dependencies and achieve feature enhancement for image embedding. It uses a self-attention network with an encoder-decoder structure; each of the two deep self-attention networks consists of three encoder modules and two decoder modules to model intra-image and inter-image relationships. The computational process is shown in the appendix. Figure 2 As shown. The input to the self-attention network is in the form of a sequence, therefore the image features F extracted by the convolutional neural network {m,f} and F c The spatial dimension will be reconstructed into a feature sequence. as well as As the input sequence, the spatial dimensions of the image features are reconstructed. To preserve spatial information, learnable positional encodings are added to the feature sequence. The first self-attention encoder-decoder network T... s T handles the internal features of individual image embeddings and the dependencies between images. s Feature sequences applied to individual fixed or floating images, i.e. and T s take over And learnable location encoding, output as enhanced features For separate embeddings of fixed and floating images, in order to distinguish whether the feature sequence comes from a fixed or floating image, T s The input also includes image index encoding to identify fixed and floating images. s It models both intra-image and inter-image dependencies. The output feature embeddings... It enhances the features extracted by convolutional neural networks, enabling larger receptive fields and modeling of global dependencies.

[0038] In the second branch of the feature enhancement stage, the second self-attention encoder-decoder network T c Modeling volume image feature embedding F using image stitching as input via self-attention mechanism c Global correlation. T c The input is And learnable positional encoding. Output enhanced features. 13) The final enhanced image feature embedding Right now and The channel splicing.

[0039] 14) Create a differential homeomorphism registration module:

[0040] Reversibility and topology preservation are properties that medical image registration should possess. To ensure that the deformation field predicted by the registration network provided in this invention meets these requirements, the registration network predicts a differential homeomorphic velocity field and uses an integral operation layer with no learnable parameters to transform the velocity field into the final registration deformation field. Differential homeomorphisms are differentiable and invertible, giving differential homeomorphic registration the properties of one-to-one mapping and topology preservation. Differential homeomorphic registration predicts the static velocity field v. t Where t∈[0,1], the velocity field v t and deformation field ψ t satisfy:

[0041]

[0042] In Formula 1, t is the time parameter, t∈[0,1], v t For the velocity field, ψ t For deformation field, symbol The combination of mappings is represented by ψ0 = I, which is the identity transformation.

[0043] This invention uses a convolutional decoder network φ r Receive enhanced image embedding and The differential homeomorphic velocity field v is predicted, and the velocity field v is integrated using an integral operation layer, ψ1=exp(v), to obtain the final registration deformation field ψ1.

[0044] 2) Training the registration network model to obtain a trained registration network model;

[0045] The proposed registration network model uses unsupervised learning to optimize network parameters by aligning images in a similarity measure space. Given a floating image V... m and fixed image V f Registration is performed to predict the registration deformation field ψ. A spatial transformation network is used to apply the deformation field to the floating image to deform it, resulting in the deformed floating image. symbol Represents image distortion. Image similarity measurement loss function L. sim Use the L1 norm of image differences.

[0046] L sim =‖V′ m -V f ‖1 (Formula 2)

[0047] In Formula 2, V′ m For floating image V mThe deformed image is obtained by registering the deformation field ψ, V f For a fixed image, ||·||1 is the L1 norm.

[0048] For the regularization term L of the velocity field reg Its smoothness is constrained using the Frobenius norm:

[0049]

[0050] In Equation 3, v is the differential homeomorphic velocity field output by the registration network. Denotes the first-order gradient, ||·|| F It is the Frobenius norm.

[0051] The final loss function L is:

[0052] L = L sim +αL reg (Formula 4)

[0053] In Formula 4, L sim L is the similarity measurement loss function. reg The velocity field regularization term is defined by α, and the hyperparameter α is used to balance the image similarity and the smoothness regularization term of the velocity field. The proposed dual-path self-attention registration network is optimized by minimizing the loss function L.

[0054] 3) Perform online registration of the 3D MRI images to be registered:

[0055] During the online registration stage, any pair of floating images V m and fixed image V f Input the trained registration network model from step 2), output the deformation field ψ, and deform the floating image to obtain the registered image. V′ m With V f The more similar the two pairs, the better the registration result.

[0056] The above steps enable efficient online registration of brain MRI images, improving the accuracy of MRI image registration.

[0057] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A registration method for three-dimensional MRI images, characterized in that, A dual-path self-attention network is used to model the relationships between different regions within an image and between images themselves, thus building a registration network model. This enhances the modeling and recognition of the semantic correspondence of anatomical structures in differential homeomorphic registration, thereby improving registration accuracy. The steps include: 1) Build the registration network model, including the following steps: 11) A registration network model is constructed based on the symmetric encoder-decoder network 3D U-net with long residual connections. The feature embedding of the three-dimensional image is enhanced by using a dual-path self-attention network. At the same time, the local features of the convolutional neural network and the global feature context relationship across the image are enhanced. The registration network model includes a learnable image feature embedding module; a dual-path self-attention feature enhancement module for dual-path cross-image relation learning; and a deformation field inference module, also known as a differential homeomorphic registration module. The input to the registration network model is a floating image V. m and fixed image V f The goal is to estimate the differential homeomorphic registration field ψ to establish a dense voxel correspondence between images; 12) Create an image feature embedding module: Feature embedding of individual and stitched 3D images is performed using 3D U-net, thereby creating an image feature embedding module; the image feature embedding module includes a first convolutional neural network encoder branch and a second convolutional neural network branch; the first convolutional neural network encoder branch φ s Receives a single-channel image as input and outputs a floating image V. m Individual feature embedding F m and fixed image V f Individual feature embedding F f ; Use the second convolutional neural network branch φ c Extract the feature embedding F from the concatenation of the floating and fixed images as input. c ; 13) Create a dual-path self-attention feature enhancement module to model cross-image dependencies and achieve feature enhancement for image embedding; Two self-attention networks with encoder-decoder structures are used, each consisting of three encoder modules and two decoder modules; the image features F extracted by the convolutional neural network are... {m，f} and F c Spatial dimensions reconstructed into feature sequences as well as As the input sequence for a self-attention network; Learnable positional codes are added to the feature sequence after spatial dimension reconstruction of image features to preserve spatial information; the first self-attention encoding-decoding network T s T handles the internal features of individual image embeddings and the dependencies between images. s Feature sequences applied to individual fixed or floating images, i.e. and T s take over And learnable location encoding, output as enhanced features Second self-attention encoding-decoding network T c Using a self-attention mechanism, a volume image feature embedding F is modeled with image stitching as input. c Global dependence; T c The input is And learnable location encoding; output as enhanced features Will and The channels are concatenated to obtain the final enhanced image feature embedding. 14) Create a differential homeomorphism registration module; The differential homeomorphism registration module predicts the differential homeomorphic velocity field and uses an integration layer with no learnable parameters to transform the velocity field into the final registration deformation field. By employing differential homeomorphism mapping, differential homeomorphism registration possesses the properties of one-to-one mapping and topology preservation. The static velocity field v... t and deformation field ψ t The expression is represented by Formula 1: In Formula 1, t is the time parameter, t∈[0,1], v t For the velocity field, ψ t For deformation field, symbol The combination of mappings is represented by ψ0 = I, which is the identity transformation; 2) Train the registration network model to obtain a trained registration network model; Training the registration network model specifically uses unsupervised learning to optimize network parameters by aligning images in a similarity measure space; Given a floating image V m and fixed image V f Registration is performed to predict the registration deformation field ψ. A spatial transformation network is then used to apply the deformation field to the floating image to deform it, resulting in the deformed floating image. symbol Represents image distortion; Image similarity measurement loss function L sim Using the L1 norm of image differences, it is expressed as Equation 2: L sim =||V′ m -V f ||1 (Formula 2) Among them, V′ m For floating image V m The deformed image is obtained by registering the deformation field ψ, V f For a fixed image, ||·||1 is the L1 norm; Using the Frobenius norm to constrain smoothness, the regularization term L of the velocity field... reg Represented as Formula 3: Where v is the differential homeomorphic velocity field output by the registration network. Denotes the first-order gradient, ||·|| F It is the Frobenius norm; The final loss function L is expressed as: L = L sim +αL reg (Formula 4) Among them, L sim L is the similarity measurement loss function. reg The hyperparameter α is used to balance the image similarity and the smoothness regularization of the velocity field. The proposed dual-path self-attention registration network is optimized by minimizing the loss function L to obtain the trained registration network model. 3) Perform online registration of the 3D MRI images to be registered: Given any pair of floating images V m and fixed image V f Input a pre-trained registration network model, output a deformation field ψ, and deform the floating image to obtain the registered image. The above steps enable efficient online registration of brain MRI images, improving the accuracy of MRI image registration.

2. The registration method for three-dimensional MRI images as described in claim 1, characterized in that, In step 13), for the separate embedding of the fixed image and the floating image, T s The input also includes image index encoding to identify fixed and floating images, thereby distinguishing whether the feature sequence comes from a fixed or floating image; T s Simultaneously, it models the dependencies within and between images; the output feature embedding Enhance the features extracted by convolutional neural networks.

3. The registration method for three-dimensional MRI images as described in claim 1, characterized in that, In step 14), a convolutional decoder network φ is used. r Receive enhanced image embedding and The differential homeomorphic velocity field v is predicted, and then the velocity field v is integrated using an integration layer with no learnable parameters, ψ1 = exp(v), to obtain the final registration deformation field ψ1; where the convolutional decoder network φ r It consists of multiple upsampling-convolution modules.

4. The registration method for three-dimensional MRI images as described in claim 1, characterized in that, In step 3), V′ m With V f The more similar the two numbers are, the better the registration result.

5. The registration method for three-dimensional MRI images as described in claim 1, characterized in that, The three-dimensional MRI image specifically refers to a three-dimensional brain MRI image.

Images