Multi-organ registration method based on cross-modal attention mechanism and vector fusion

Abstract

This invention discloses a multi-organ registration method based on cross-modal attention mechanisms and vector fusion, primarily addressing the poor multi-organ registration results of existing technologies for CT and MRI images of the abdomen and head and neck. The implementation scheme involves: preprocessing multi-organ multimodal data from the abdomen and head and neck; establishing a multimodal information encoder composed of several cascaded downsampling convolutional blocks and several cascaded cross-modal attention mechanism modules; extracting feature maps from different modalities and concatenating and convolving them with the original feature maps extracted by different downsampling convolutional blocks to obtain fused feature maps; establishing a multi-branch single-organ deformation map decoder to generate deformation maps for each organ; and establishing a vector gradient fusion module to reduce conflicts during multi-organ deformation map fusion, thus achieving cross-modal multi-organ registration. This invention preserves the morphology and structure of each organ, improving the accuracy of multi-organ registration. It can be used by doctors for surgical planning, radiotherapy, and auxiliary diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a multi-organ cross-modal registration method, which can be used for CT and MRI image registration of the human abdomen and head and neck. It provides assistance to doctors in medical diagnosis, surgical planning, radiotherapy, etc., allowing doctors to perform more accurate analysis and providing a basis for doctors to formulate more precise treatment plans. Background Technology

[0002] CT and MRI scans are the most commonly used imaging techniques in the medical field. Each has its advantages in clinical application. CT scans are fast, making them highly advantageous for situations requiring rapid diagnosis. CT also shows lesions in areas such as the lungs and bones effectively. MRI, on the other hand, is better at showing lesions in soft tissues and the nervous system, and is particularly valuable for the diagnosis and differential diagnosis of diseases such as tumors and inflammation. CT and MRI have complementary advantages in their imaging principles; therefore, combining the observation of both types of images can improve diagnostic accuracy and provide a more reliable basis for developing treatment plans.

[0003] However, doctors rely on spatial imagination and subjective experience when observing and analyzing different images, which carries a certain risk of error. This can be addressed through medical image registration. Medical image registration technology aligns image data from different modalities or time points, allowing doctors to more clearly observe the morphology, location, and extent of lesions.

[0004] In the field of multi-organ, multimodal medical image registration, there are two major challenges: 1. the problem of deformation field interference when fusing registration results from different single organs; and 2. how to enable the model to learn the spatial correspondence between images of different modalities. Furthermore, multi-organ registration in the abdomen and head and neck region is particularly challenging, with limited research in this area. The distribution of organs in the abdomen is complex. Because the abdomen contains numerous organs, each occupying a relatively small proportion of the image (typically only 10%–30%), registration algorithms need to process the features of each organ more precisely. Moreover, the deformation patterns of multiple organs in the abdomen differ significantly from those in multi-region registration problems such as lung parenchyma and interstitium. The latter typically exhibit smoother and more dynamic deformation characteristics, while deformation of a single abdominal organ leads to complex, non-smooth deformation in the surrounding area. In head and neck registration, the high elasticity of the soft tissues in the head and neck region often results in adhesion and occlusion between organs, leading to insufficient smoothness in the overall registration result. Furthermore, the numerous and complex organs in the head and neck region often make it difficult for traditional registration methods to provide valuable deformation information.

[0005] Patent document CN202311107895.4 discloses a method for registering multiple organs in the abdomen based on depth probability maps and vector fusion. It uses several parallel VoxelMorphs to generate deformation fields and global probability maps of different organs, and introduces a VFF vector field fusion module to fuse the deformation fields of each organ using an attention mechanism. Although this method can overcome the deformation field interference problem during the fusion of multiple organ deformation fields and reduce the fusion error, the deformation field of each organ is generated independently, and there is a lack of information exchange between different organs and modalities when generating the deformation fields of each single organ. Therefore, it limits the ability to capture the spatial correspondence between different organs and modalities, resulting in insufficient utilization of multimodal and multi-organ information fusion in the registration results.

[0006] Patent document CN202310944659.1 discloses a method for abdominal multi-organ registration based on an adaptive multi-gated hybrid expert model. It first fuses information from two modal images using a VoxelMorph encoder. Then, it modifies the VoxelMorph network's decoder into three parallel multi-branch networks, generating feature maps corresponding to different organs using different branches. The adaptive multi-gated hybrid expert model is then used to fuse the feature maps of all organs, resulting in a fused feature map containing information from different modalities and organs. Next, three parallel upsampling branch networks generate single-organ deformation maps of each organ based on the fused feature map. Finally, the single-organ deformation maps of all organs are summed to obtain the overall multi-organ deformation map, from which the registration result is obtained. Although this method fuses information from multiple modalities and organs, the fusion method is relatively crude and has the following two shortcomings:

[0007] First, when fusing multimodal information, a simple CNN encoder is used to fuse images of different modalities by downsampling. This fusion method cannot learn the spatial correspondence between images of different modalities well, thus affecting the final registration effect.

[0008] Secondly, when fusing information from multiple organs, the deformation fields of different single organs are simply added together to obtain the multi-organ deformation field. This method of fusing single-organ deformation fields is too simplistic and lacks the ability to explore and utilize the relationships between the deformation fields of each single organ. This limits the ability to learn the correspondence between different organs, resulting in insufficient utilization of multi-organ fusion information in the registration results. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of the prior art by proposing a multi-organ registration method based on cross-modal attention mechanism and vector fusion, so as to better learn the spatial correspondence between images of different modalities, reduce the interference between the registration deformation fields of each organ, reduce the registration error of each organ, and improve the overall registration effect.

[0010] To achieve the above objectives, the technical approach of this invention is as follows: In cross-modal image registration, a cross-modal attention mechanism is used to learn the spatial correspondence between features of CT and MRI images, which are two different modalities, and the deep features extracted from these two modalities are mapped to the same feature to achieve cross-modal feature fusion; In multi-organ registration, the deformation fields of each organ are generated by using features that fuse information from different modalities, and the registration deformation fields of each organ are optimized by returning the loss of each organ. Then, the deformation fields are fused using a vector gradient fusion module to reduce the direct conflict of multi-organ deformation field fusion, and finally, cross-modal multi-organ registration is achieved.

[0011] Based on the above technical concept, the implementation steps of the present invention include the following:

[0012] (1) Obtain abdominal and head and neck CT and MRI data, filter and preprocess them, and divide the abdominal and head and neck CT and MRI image data into training set, validation set and test set respectively;

[0013] (2) Construction of a multimodal, multi-organ registration network CAVF-MMR:

[0014] (2a) A multimodal information encoder U1 is constructed, consisting of several cascaded downsampling convolutional blocks and several cascaded cross-modal attention mechanism modules, to extract feature maps fea from CT and MRI images. cross_0 ,fe across_1 ;

[0015] (2b) Establish a single-organ deformation map decoder U2 with multiple branches to decode the single-organ deformation map of each organ;

[0016] (2c) The existing U-Net network is selected as the global probabilistic graph generation network u3 to generate the global probabilistic graph g. a ;

[0017] (2d) Establish a vector fusion module U8, which includes a multi-head attention mechanism module U4, a Poisson fusion module u5, a multi-organ deformation map decoding network u6, and a spatial transformation module U7, to fuse all single-organ deformation maps;

[0018] (2e) Connect the multimodal information encoder u1 to the single organ deformable map decoder u2, and then cascade the single organ deformable map decoder U2, the probabilistic map generation network U3 and the vector fusion module U8 to form a multimodal multi-organ registration network CAVF-MMR based on cross-modal attention mechanism and vector fusion.

[0019] (3) Training the multimodal multi-organ registration network CAVF-MMR:

[0020] (3a) Define the global loss function L of the multimodal multi-organ registration network CAVF-MMR:

[0021] L=∑L i +L a +L mse +L sm

[0022] Where L i It is the Dice loss of the i-th organ in reference image A and registered image B, L a It is the Dice loss of the union of all organs a in reference image A and registered image B, L mse L is the MSE loss of the union of all organs a in reference image A and registered image B. sm It is the smoothing loss of the registered image B;

[0023] (3b) Input a batch of data from the training set into the multimodal multi-organ registration network, and use the reference image A from the training set as a reference. t and its label l At Image B (Registered) t and its registration label l Bt The global loss value L of the training set is calculated using the global loss function. t ;

[0024] (3c) Global loss L for the training set t Perform backpropagation to iteratively update network parameters;

[0025] (3d) Based on the reference image A of a certain batch of data in the validation set v and its label l At Image B (Registered) v and its registration label l Bv The global loss L of the validation set is calculated. v This is used to evaluate the registration performance of the current registration network;

[0026] (3e) Repeat (3b) to (3c) until the global loss of the training set L t The network converges, resulting in a well-trained registration network.

[0027] (4) Input the images to be registered in the test set into the trained registration network model to obtain the multi-organ registration results of the test set.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] 1. The present invention uses a cross-modal attention mechanism in the established multimodal information encoder U1 to input information from different modalities into the attention mechanism layer, thereby obtaining positional attention information of one modal information to another modal information. Thus, it can learn the spatial positional correspondence across modalities and improve the accuracy of image registration.

[0030] 2. The global loss function L defined in this invention not only considers the overlap of organ registration, but also the structural restoration degree of organ registration and the smoothness of the registered organ edges. This makes the defined global loss function L rich in information and able to accurately and comprehensively guide the registration network to generate accurate and smooth registration results.

[0031] 3. In the single-organ deformable image decoder U2, the feature maps at different levels extracted by the multimodal information encoder U1 are fused to obtain a fused feature map containing multimodal image structural information and cross-modal spatial relationship information. This fused feature map is then used as the input to each branch network in the single-organ deformable image decoder U2, enabling the registration network to better understand and utilize the complementary information between multiple modalities, thereby enhancing the multimodal registration capability of the registration network.

[0032] 4. In the vector fusion module U8, this invention improves the Poisson fusion strategy by utilizing a multi-head attention mechanism and a custom Laplacian operator. This effectively integrates multiple single-organ vector fields, reduces the mutual interference between organ correspondence problems and deformation maps in the multi-organ registration process, and enhances the multi-organ registration capability of the registration network. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the implementation process of the present invention;

[0034] Figure 2 This is a schematic diagram of the overall structure of the multimodal multi-organ registration neural network model constructed in this invention;

[0035] Figure 3 This is a structural diagram of the cross-modal attention mechanism module established in this invention;

[0036] Figure 4 This is a structural diagram of the vector fusion module established in this invention;

[0037] Figure 5 This is a comparison chart showing the effects of the present invention and five existing registration methods on the registration of abdominal organs such as the bladder, cervix, and rectum.

[0038] Figure 6 This is a comparison diagram of the registration effects of the present invention and five existing registration methods on the head and neck oral cavity, left parotid gland, right parotid gland and brainstem organs. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only a part of the present invention, and not all of the embodiments. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should all fall within the protection scope of the present invention.

[0040] Reference Figure 1 The implementation steps of this example include the following:

[0041] Step 1: Obtain the data, preprocess it, and divide it into training set, validation set, and test set.

[0042] (1.1.1) Obtain abdominal data, preprocess it, and classify it:

[0043] Abdominal data was obtained from the medical department, and 2D image slices that simultaneously possessed both CT and MRI modalities and included the three abdominal organs (bladder, cervix, and rectum) and their labels were selected. These slices were then preprocessed by grayscale normalization and standardization to obtain the final 2D abdominal CT and MRI image data.

[0044] (1.1.2) The final abdominal data is divided into an abdominal training set, an abdominal validation set, and an abdominal test set in a ratio of 8:1:1;

[0045] (1.2) Obtain head and neck data, and perform preprocessing and classification:

[0046] (1.2.1) Obtain head and neck data from public datasets, filter out 2D image slices that have both CT and MRI modal images and include four organs and labels of the head and neck, oral cavity, left parotid gland, right parotid gland and brainstem, and then rigidly register the MRI and CT images in each pair of data, and perform normalization and standardization preprocessing in sequence to obtain the final 2D head and neck CT and MRI image data.

[0047] (1.2.2) The final head and neck data are divided into head and neck training set, head and neck validation set and head and neck test set in a ratio of 8:1:1.

[0048] Step 2: Construct a multimodal multi-organ registration network CAVF-MMR based on cross-modal attention mechanism and vector fusion.

[0049] Reference Figure 2 This step includes the following:

[0050] (2.1) A multimodal information encoder U1 is established, consisting of several cascaded downsampling convolutional blocks and several cascaded cross-modal attention mechanism modules, to extract feature maps fea from CT and MRI images. cross_0 fea cross_1 ;

[0051] (2.1.2) Constructing the downsampling convolutional block C d :

[0052] Establish multiple downsampling convolutional layers consisting of convolutional kernels, batch normalization, and ReLU activation connections;

[0053] Each pair of downsampling convolutional layers is connected to a max pooling layer to form a downsampling convolutional block C. d ;

[0054] (2.1.3) Constructing the cross-modal attention mechanism module M A :

[0055] Reference Figure 3 The implementation of this step includes the following:

[0056] The three convolutional kernels are cascaded with three flattened layers to form three convolutional layers (Conv). q Conv k and Conv v ;

[0057] Three Conv q Conv k Conv v Connected to the query input, key input, and value input of the attention mechanism layer respectively, they form the cross-modal attention mechanism module M. A ;

[0058] (2.1.4) Convert the three downsampled convolutional blocks C d With 3 cross-modal attention mechanism modules M A Cascade them into a branch, and connect the two branches in parallel to form a multimodal information encoder U1.

[0059] (2.2) Establish a single-organ deformation map decoder U2 with multiple branches to decode the single-organ deformation map g of each organ. i , where i represents the i-th organ;

[0060] (2.2.1) Construct a deconvolution block consisting of two convolutional layers and one deconvolutional layer cascaded together. Each convolutional layer consists of a convolutional kernel, batch normalization and LeakyReLU activation, and the deconvolutional layer consists of a deconvolutional kernel, batch normalization and LeakyReLU function connected together.

[0061] (2.2.2) Construct a convolutional block consisting of two concatenated convolutional layers, wherein each convolutional layer consists of a convolutional kernel, batch normalization, and LeakyReLU activation connections;

[0062] (2.2.3) Concatenate two deconvolution blocks and one convolution block to form a single-organ deformation graph decoding branch. i Decode n single-organ deformation maps using branching. i Parallel connections form a multi-branched single-organ deformation map decoder U2, where n is the total number of organs and i represents the i-th organ;

[0063] (2.3) The existing U-Net network is selected as the global probabilistic graph generation network U3 to generate the global probabilistic graph g. a ;

[0064] (2.4) Establish a vector fusion module U8, which includes a multi-head attention mechanism module U4, a Poisson fusion module U5, a multi-organ deformation map decoding network U6, and a spatial transformation module U7, to fuse all single-organ deformation maps;

[0065] Reference Figure 4 This step includes the following:

[0066] (2.4.1) Constructing the multi-head attention mechanism module U4

[0067] A linear layer is concatenated with a normalized layer to form the first linear network Linear1.

[0068] The three linear layers are concatenated with the three normalized layers respectively to form the second linear network Linear2, the third linear network Linear3, and the fourth linear network Linear4. These three linear networks Linear2, Linear3, and Linear4 are then connected in parallel to form the merged linear network Linaer. a ;

[0069] The first linear network Linear1 and the merged linear network Linear a All inputs are fed into a multi-head attention layer, where multi-head attention interaction operations are performed to obtain the attention weights w between images of different modalities. iThis constitutes the multi-head attention mechanism module U4, and uses the output of the first linear network Linear1 as the key and value inputs for multi-head attention, while merging the linear networks flow... a The output of is used as the query input for multi-head attention, where i is the i-th organ.

[0070] (2.4.2) Constructing the Poisson fusion module U5

[0071] Create a 3×3 convolution kernel and initialize the kernel parameters by setting the values ​​in the first row of the kernel to 0, 1, 0; the values ​​in the second row to 1, -4, 1; and the values ​​in the third row to 0, 1, 0, thus obtaining the Laplacian operator Δ for the 2D image.

[0072] The attention weight w output by the multi-head attention mechanism module U4 is used as the weighting weight, and the Δg output by the Laplacian operator Δ is used as the weighting weight. i The Laplacian operator result Δg is obtained by performing a weighted summation operation on the multi-organ deformation map g:

[0073]

[0074] Among them, w i It is the attention weight of the i-th organ, Δg i This is a single-organ deformation diagram of the i-th organ, g. i The result of the Laplace operator; i = 1, 2, ..., n, where n is the total number of organs;

[0075] (2.4.3) Establish a multi-organ deformation map decoding network U6, which includes an MLP layer and a normalization layer. The output of the MLP layer is connected to the input of the normalization layer, and the input residual is connected to the output of the normalization layer.

[0076] (2.4.4) Construct a spatial transformation module U7 that includes deformation coordinate calculation and bilinear interpolation. It takes a multi-organ deformation map g and the image to be registered as input. First, it calculates the deformation coordinates based on the multi-organ deformation map g. Then, it uses bilinear interpolation to resample the image to be registered and the labels of each organ into the deformation coordinates, thus obtaining the final registered image B and the registration labels l of each organ. B This enables multi-organ, multi-modal registration of the image to be registered to the reference image A.

[0077] (2.4.5) Connect the multi-head attention mechanism module U4 and the Poisson fusion module U5 in parallel, and then cascade them with the multi-organ deformation decoding network graph U6 and the spatial transformation module U7 to form the vector fusion module U8.

[0078] (2.5) Connect the multimodal information encoder U1 to the single organ deformation map decoder U2, and then cascade the single organ deformation map decoder U2, the probabilistic map generation network U3 and the vector fusion module U8 to form a multimodal multi-organ registration network CAVF-MMR based on cross-modal attention mechanism and vector fusion.

[0079] Step 3: Train the multimodal multi-organ registration network CAVF-MMR.

[0080] (3.1) Define the global loss function L of the multimodal multi-organ registration network CAVF-MMR:

[0081] (3.1.1) Define the Dice loss L for each individual organ in the reference image A and the registered image B. i :

[0082]

[0083] The Dice similarity coefficient is a metric used to assess the degree of overlap between two samples. i This represents the number of pixels in the image for the i-th organ. This represents the k-th organ in the i-th organ of reference image A. i 1 pixel, This represents the k-th organ in the registered image B. i 1 pixel, k i =1,2,…,Y i i = 1, 2, ..., n, where n is the total number of organs;

[0084] (3.1.2) Define the Dice loss L for the union of all organs in the reference image A and the registered image B. a :

[0085]

[0086] The Dice similarity coefficient is a metric used to assess the similarity or overlap between two samples. 'a' represents the union of all organs, and 'Y' represents the similarity. a This represents the number of pixels in the image where the union of all organs is represented. The k-th element represents the union of all organs in reference image A. a 1 pixel, The k-th element represents the union of all organs in the registered image B. a 1 pixel, k a =1,2,…,Y a ;

[0087] (3.1.3) Define the total MSE loss L for all organs in reference image A and registered image B.mse :

[0088]

[0089] w and h represent the width and height of the reference image A, respectively; x represents the pixel index in the horizontal direction of the image; and y represents the pixel index in the vertical direction.

[0090] (3.1.4) Define the smoothing loss L of the registered image B. sm :

[0091]

[0092] w and h represent the width and height of the reference image A, respectively; x represents the pixel index in the horizontal direction of the image; and y represents the pixel index in the vertical direction.

[0093] (3.1.5) Based on the loss functions defined in (3.1.1) to (3.1.4), the global loss function L of the multimodal multi-organ registration network CAVF-MMR is obtained:

[0094] L=∑L i +L a +L mse +L sm .

[0095] (3.2) Input a batch of data from the training set into the multimodal multi-organ registration network, and use the reference image A of that batch of data from the training set. t and its label l At Image B (Registered) t and its registration label l Bt The global loss value L for this batch of data in the training set is calculated using the global loss function. t ;

[0096] (3.3) Global loss L for the training set t Perform backpropagation to iteratively update network parameters;

[0097] (3.3.1) Set the hyperparameters for backpropagation: Set the initial learning rate to 0.0001 and the batch size to 1;

[0098] (3.3.2) SGD is used as the optimizer for training the multimodal multi-organ registration network CAVF-MMR, and the global loss L of the training set is... t The input is fed into the SGD optimizer for backpropagation to update the parameters of the CAVF-MMR network;

[0099] (3.4) Based on the reference image A of a certain batch of data in the validation set v and its label l At Image B (Registered)v and its registration label l Bv The global loss L of the validation set is calculated. v This is used to evaluate the registration performance of the current registration network;

[0100] (3.5) Repeat (3.2) to (3.3) until the global loss of the training set L t The network converges, resulting in a well-trained registration network.

[0101] Step 4: Use the trained registration network to obtain the registration result of the image to be registered.

[0102] 4.1) The image to be registered and its organ labels, and the reference image and its organ labels are input into the trained CAVF-MMR network. The network uses the multimodal information encoder U1 to calculate the feature maps fea of ​​the CT and MRI modalities. cross_0 fea cross_1 This feature map is processed by the single-organ deformation map decoder U2 in the network to obtain the single-organ deformation map g of each organ. i ;

[0103] 4.2) Input the organ labels of the image to be registered and the organ labels of the reference image into the global probabilistic map generation network U3 to obtain the global probabilistic map g. a ;

[0104] 4.3) Single organ deformation diagram of all organs g i and global probability graph g a The final registration result is obtained through the vector fusion module U8.

[0105] The effects of the present invention can be further illustrated by the following simulation.

[0106] 1. Simulation conditions:

[0107] The simulation platform for this experiment was a computer with an Intel Core i7-8700K CPU and 32GB of memory, running Windows 10. The PyTorch 3.8 deep learning framework was used to build and train the CAVF-MMR network. NVIDIA GeForce GTX1080Ti GPU and CUDA 11.2 were used to accelerate the training of the CAVF-MMR network and the generation of registration images.

[0108] The registration performance evaluation metrics used in the simulation include the Destination Similarity Coefficient (DSC), Mutual Information (MI), and Average Surface Distance (ASD), and their specific calculation formulas are as follows:

[0109]

[0110] MI(A,B)=H(A)+H(B)-H(A,B)

[0111]

[0112] A represents the organ label of the reference image, B represents the organ label of the registered image, H(A) and H(B) represent the information entropy of label A and label B respectively, H(A,B) is the joint entropy of label A and label B, S(A) represents the surface pixels of the organ label of the reference image, S(B) represents the surface pixels of the organ label of the registered image, and d(s A d(s) represents the shortest distance from any pixel of the organ label in the reference image to the surface pixel of the organ label in the registered image. B S(A)) represents the shortest distance from any pixel in the predicted result to a pixel on the actual label surface.

[0113] 2. Simulation Content and Results

[0114] Simulation 1: Under the above simulation conditions, cross-modal multi-organ registration was performed on abdominal organs using the present invention and five existing registration methods: SyN, demos, VoxelMorph, DFMIR, and PoissonNet. The evaluation indices for each method on different organs were calculated: Destine similarity coefficient (DSC), mutual information (MI), and average surface distance (ASD). The results are shown in Table 1.

[0115] Table 1. Results of comparative experiments on the abdomen.

[0116]

[0117] As shown in Table 1, while the Demons algorithm considers the smoothness of the deformation field and is suitable for cross-modal registration, it cannot accurately register to each organ. SyN, due to noise interference from different modalities, is even less capable of completing the registration task. The deep learning-based VoxelMorph registration method cannot learn the features of each modality. While the DFMIR registration method is suitable for cross-modal registration, it cannot resolve the deformation field conflict between organs, weakening its deformation capability. The PoissonNet registration method is suitable for multi-organ registration, but it cannot handle cross-modal issues well. These existing registration methods have essentially failed to complete the cross-modal multi-organ registration task. This invention, by employing a cross-modal attention mechanism for splicing and a deformation field conflict fusion strategy, achieves the best results in DICE, ASD, and MI metrics compared to existing technologies.

[0118] Simulation 2: Under the above simulation conditions, multimodal multi-organ registration of the abdominal bladder, cervix, and rectum was performed using the present invention and five existing registration methods: SyN, demos, VoxelMorph, DFMIR, and PoissonNet. The registration results are as follows: Figure 5 As shown.

[0119] from Figure 5 It is readily apparent that in multimodal, multi-organ registration with large deformations, the Syn algorithm is essentially incapable of completing the registration. While the Demons algorithm can address the problem of large deformations across modalities, it fails to accurately focus on each individual organ, leading to completely erroneous registration results. The DFMIR method can effectively focus on features suitable for cross-modal organ registration, but it cannot complete multi-organ registration. Although PoissonNet can resolve conflicts in multi-organ registration, it cannot achieve cross-modal registration, resulting in suboptimal registration results for each organ. In contrast, the multimodal, multi-organ registration results of this invention preserve the morphology and structure of each organ, exhibiting clear organ edges and accurate organ morphology.

[0120] Simulation 3: Under the above simulation conditions, cross-modal multi-organ registration was performed on head and neck organs using the present invention and five existing registration methods: SyN, demos, VoxelMorph, DFMIR, and PoissonNet. The results were evaluated using the DSC similarity coefficient, MI mutual information, and ASD mean surface distance. The results are shown in Table 2.

[0121] Table 2 shows the results of the comparative experiments on the head and neck.

[0122]

[0123] As can be observed from Table 2, for the four tiny soft tissues of the head and neck, neither traditional algorithms nor deep learning registration methods such as Demons, SyN, voxelmorph, and DFMIR can complete the cross-modal multi-organ registration task. This is because the organs are too small for these registration methods to achieve proper organ registration. While PoissonNet can basically achieve multi-organ registration, it cannot handle cross-modal images well. In contrast, the multi-modal multi-organ registration results of this invention maintain the morphology and structure of each organ, with clear organ edges and accurate organ morphology. This is due to the use of cross-modal attention mechanisms, deformation map fusion strategies, and Poisson image editing methods.

[0124] Simulation 4: Under the above simulation conditions, multimodal multi-organ registration was performed on four organs in the head and neck region—the oral cavity, left parotid gland, right parotid gland, and brainstem—using the present invention and five existing registration methods: SyN, demos, VoxelMorph, DFMIR, and PoissonNet. The results are as follows: Figure 6 As shown.

[0125] from Figure 6 The deformation field clearly shows that the deformation field generated by the registration method CAVF-MMR proposed in this invention is mainly concentrated in the relevant regions of the four organs. Other registration methods simply cannot accurately register to the four key organs. Although PoissonNet is slightly better than other methods, it cannot handle cross-modal registration, which prevents it from learning relevant features, and therefore it still cannot accurately register the four organs.

[0126] The above description is merely a specific example of the present invention and does not constitute any limitation on the present invention. Obviously, those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0127] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

Claims

1. A multi-organ registration method based on cross-modal attention mechanism and vector fusion, characterized in that, Includes the following steps: (1) Obtain abdominal and head and neck CT and MRI data, filter and preprocess them, and divide the abdominal and head and neck CT and MRI image data into training set, validation set and test set respectively; (2) Constructing a multimodal multi-organ registration network CAVF-MMR based on cross-modal attention mechanism and vector fusion: (2a) A multimodal information encoder U1 is constructed, consisting of several cascaded downsampling convolutional blocks and several cascaded cross-modal attention mechanism modules, to extract feature maps fea from CT and MRI images. cross_0 fea cross_1 ; (2b) Establish a single-organ deformation map decoder U2 with multiple branches to decode the single-organ deformation map of each organ; (2c) Select the existing U-Net network as the global probabilistic graph generation network U3 to generate the global probabilistic graph g. a ; (2d) Establish a vector fusion module U8, which includes a multi-head attention mechanism module U4, a Poisson fusion module U5, a multi-organ deformation map decoding network U6, and a spatial transformation module U7, to fuse all single-organ deformation maps; (2e) Connect the multimodal information encoder U1 to the single organ deformable map decoder U2, and then cascade the single organ deformable map decoder U2, the probabilistic map generation network U3 and the vector fusion module U8 to form a multimodal multi-organ registration network CAVF-MMR based on cross-modal attention mechanism and vector fusion. (3) Training the multimodal multi-organ registration network CAVF-MMR: (3a) Define the global loss function L of the multimodal multi-organ registration network CAVF-MMR: L=∑L i +L a +L mse +L sm L i It is the Dice loss of the i-th organ in reference image A and registered image B, L a It is the Dice loss of the union of all organs a in reference image A and registered image B, L mse L is the MSE loss of the union of all organs a in reference image A and registered image B. sm It is the smoothing loss of the registered image B; (3b) Input a batch of data from the training set into the multimodal multi-organ registration network, and use the reference image A from the training set as a reference. t and its label l At Image B (Registered) t and its registration label l Bt The global loss value L of the training set is calculated using the global loss function. t ; (3c) Global loss L for the training set t Perform backpropagation to iteratively update network parameters; (3d) Based on the reference image A of a certain batch of data in the validation set v and its label l At Image B (Registered) v and its registration label l Bv The global loss L of the validation set is calculated. v This is used to evaluate the registration performance of the current registration network; (3e) Repeat (3b) to (3c) until the global loss of the training set L t The network converges, resulting in a well-trained registration network. (4) Input the images to be registered in the test set into the trained registration network model to obtain the multi-organ registration results of the test set.

2. The method according to claim 1, characterized in that, In step (1), the acquired abdominal and head and neck CT and MRI data are screened, preprocessed, and divided into training, validation, and test sets, as follows: (1a) 2D image slices with both CT and MRI modal images, and containing the three organs of the abdomen, bladder, cervix and rectum, and labels, are selected from the abdominal data obtained from the medical department. The slices are then subjected to grayscale normalization and standardization preprocessing to obtain the final 2D abdominal CT and MRI image data. (1b) Divide the final abdominal data into an abdominal training set, an abdominal validation set, and an abdominal test set in a ratio of 8:1:1; (1c) Select 2D image slices from the head and neck data obtained from the public dataset that have both CT and MRI modal images and include the four organs of the head and neck, oral cavity, left parotid gland, right parotid gland and brainstem, and labels. Then, rigidly register the MRI and CT images in each pair of data and perform normalization and standardization preprocessing in sequence to obtain the final 2D head and neck CT and MRI image data. (1d) The final head and neck data are divided into head and neck training set, head and neck validation set and head and neck test set in a ratio of 8:1:

1.

3. The method according to claim 1, characterized in that, In step (2a), a multimodal information encoder U1 is established, consisting of several cascaded downsampling convolutional blocks and several cascaded cross-modal attention mechanism modules, as follows: (2a1) Construct downsampling convolutional blocks: (2a1.1) Construct a downsampling convolutional layer consisting of convolutional kernels, batch normalization, and ReLU activation connections; (2a1.2) Connect every two downsampling convolutional layers with a max pooling layer to form a downsampling convolutional block; (2a2) Construct a cross-modal attention mechanism module: (2a2.1) Concatenate the three convolutional kernels with the three flattened layers respectively to form three convolutional layers Conv q Conv k and Conv v ; (2a2.2) Combine the three Conv q Conv k Conv v These are connected to the query input, key input, and value input of the attention mechanism layer, respectively, to form a cross-modal attention mechanism module; (2a3) Concatenate three downsampled convolutional blocks with three cross-modal attention mechanism modules into a branch, and connect two branches in parallel to form a multimodal information encoder U1.

4. The method according to claim 1, characterized in that, Step (2b) establishes a single-organ deformation map decoder U2 with multiple branches, as follows: (2b1) Construct a deconvolution block consisting of two convolutional layers and one deconvolutional layer cascaded together. Each convolutional layer consists of a convolutional kernel, batch normalization and LeakyReLU activation, and the deconvolutional layer consists of a deconvolutional kernel, batch normalization and LeakyReLU function connected together. (2b2) ​​Construct a convolutional block consisting of two concatenated convolutional layers, wherein each convolutional layer consists of a convolutional kernel, batch normalization, and LeakyReLU activation connections; (2b3) Concatenate two deconvolution blocks and one convolution block to form a branch of the single-organ deformation map decoder U2. i , where i represents the i-th organ. The single-organ deformation image decoder U2 with multiple branches has a total of n branches, where n is the total number of organs.

5. The method according to claim 1, characterized in that, The global probability graph generation network U3 in step (2c) consists of a cascaded downsampling network, a bottleneck layer and an upsampling network. The downsampling network is connected to the corresponding network layers of the upsampling network at different network layers, realizing information exchange between features of different scales, thereby generating a global probability graph that takes into account features of various scales.

6. The method according to claim 1, characterized in that, Step (2d) establishes a vector fusion module U8, which includes a multi-head attention mechanism module U4, a Poisson fusion module U5, a multi-organ deformation map decoding network U6, and a spatial transformation module U7, as follows: (2d1) Constructing the multi-head attention mechanism module U4 A linear layer is concatenated with a normalized layer to form the first linear network, Linear1. The three linear layers are concatenated with the three normalized layers respectively to form the second linear network Linear2, the third linear network Linear3, and the fourth linear network Linear4. These three linear networks Linear2, Linear3, and Linear4 are then connected in parallel to form the merged linear network Linear4. a ; The first linear network Linear1 and the merged linear network Linear a All inputs are fed into a multi-head attention layer for multi-head attention interaction computation, forming the multi-head attention mechanism module U4. The output of the first linear network, Linear1, serves as the key and value for multi-head attention, and the outputs of the merged linear networks are combined. a The output is used as a query for multi-head attention; (2d2) Construct the Poisson fusion module U5 Create a 3×3 convolution kernel and initialize the kernel parameters by setting the values ​​in the first row of the kernel to 0, 1, 0; the values ​​in the second row to 1, -4, 1; and the values ​​in the third row to 0, 1, 0, thus obtaining the Laplacian operator Δ for the 2D image. The attention weights w output by the multi-head attention mechanism module U4 i As a weighting factor, the output Δg of the Laplacian operator Δ i The Laplacian operator result Δg is obtained by performing a weighted summation operation on the multi-organ deformation map g: w i It is the attention weight of the i-th organ, Δg i This is a single-organ deformation diagram of the i-th organ, g. i The result of the Laplace operator; i = 1, 2, ..., n, where n is the total number of organs; (2d3) Establish a multi-organ deformation map decoding network U6 consisting of an MLP layer and a normalization layer. The output of the MLP layer is connected to the input of the normalization layer, and the input residual is connected to the output of the normalization layer. (2d4) Construct a spatial transformation module U7 that includes deformation coordinate calculation and bilinear interpolation. It takes a multi-organ deformation map g and the image to be registered as input. First, it calculates the deformation coordinates based on the multi-organ deformation map g. Then, it uses bilinear interpolation to resample the image to be registered and the labels of each organ into the deformation coordinates, thus obtaining the final registered image B and the registration labels l of each organ. B This enables multi-organ, multi-modal registration of the image to be registered to the reference image A. (2d5) Connect the multi-head attention mechanism module U4 and the Poisson fusion module U5 in parallel, and then cascade them with the multi-organ deformation decoding network graph U6 and the spatial transformation module U7 to form the vector fusion module U8.

7. The method according to claim 1, characterized in that, Step (3a) Define the different losses L involved in the global loss function L. i L a L mse L sm The calculation formulas are as follows: (3a1) Dice loss L of each individual organ in reference image A and registered image B i : The Dice similarity coefficient is a metric used to assess the degree of overlap between two samples. i This represents the number of pixels in the image for the i-th organ. This represents the k-th organ in the i-th organ of reference image A. i 1 pixel, This represents the k-th organ in the registered image B. i 1 pixel, k i =1,2,…,Y i i = 1, 2, ..., n, where n is the total number of organs; (3a2) Dice loss L for the union of all organs in reference image A and registered image B a : The Dice similarity coefficient is a metric used to assess the similarity or overlap between two samples. 'a' represents the union of all organs, and 'Y' represents the similarity. a This represents the number of pixels in the image where the union of all organs is represented. The k-th element represents the union of all organs in reference image A. a 1 pixel, The k-th element represents the union of all organs in the registered image B. a 1 pixel, k a =1,2,…,T a ; (3a3) Total MSE loss of all organs in reference image A and registered image B mse : w and h represent the width and height of the reference image A, respectively; x represents the pixel index in the horizontal direction of the image; and y represents the pixel index in the vertical direction. (3a4) Smoothing loss L of registered image B sm : w and h represent the width and height of the reference image A, respectively; x represents the horizontal pixel index in the image; and y represents the vertical pixel index.

8. The method according to claim 1, characterized in that, In step (3c), the global loss L of the training set is... t Perform backpropagation to iteratively update network parameters, as follows: (3c1) Set the hyperparameters for backpropagation: Set the initial learning rate to 0.0001 and the batch size to 1; (3c2) SGD is used as the optimizer for training the multimodal multi-organ registration network CAVF-MMR, and the global loss of the training set L is... t The parameters are fed into the SGD optimizer for backpropagation, iteratively updating the parameters of the CAVF-MMR network.

Images