Multi-organ registration method based on cross-modal attention mechanism and vector fusion

Abstract

The invention discloses a multi-organ registration method based on a cross-modal attention mechanism and vector fusion, and mainly solves the problem of poor multi-modal image multi-organ registration effect of CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) of the abdomen, the head and the neck in the prior art. According to the implementation scheme, the method comprises the following steps: preprocessing multi-organ multi-modal data of the abdomen and the head and neck; establishing a multi-modal information encoder composed of a plurality of cascaded down-sampling convolution blocks and a plurality of cascaded cross-modal attention mechanism modules; feature maps of different modal images are extracted, and the feature maps and original image feature maps extracted by different down-sampling convolution blocks are spliced and convolved to obtain a fused feature map; establishing a multi-branch single organ deformation graph decoder to generate a deformation graph of each organ; and a vector gradient fusion module is established to reduce conflicts during multi-organ deformation graph fusion, and cross-modal multi-organ registration is realized. According to the invention, the form and structure of each organ can be maintained, and the accuracy of multi-organ registration is improved. The system can be used by doctors for making operation plans, radiotherapy and auxiliary diagnosis.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and particularly relates to a multi-organ cross-modality registration method, which can be used for CT and MRI image registration of the human abdomen and head and neck, to assist doctors in medical diagnosis, surgical planning, radiotherapy, etc., to enable doctors to conduct more accurate analysis, and to provide a basis for doctors to formulate more precise treatment plans. Background Art

[0002] CT and MRI are the most commonly used imaging examination methods in the medical field. CT and MRI each have their own advantages in clinical applications. CT examination is fast, which is a great advantage for situations that require rapid diagnosis. At the same time, CT has a better display effect on lesions in the lungs, bones and other parts. MRI has a better display effect on lesions in soft tissues, nervous system and other parts, especially for the diagnosis and differentiation of diseases such as tumors and inflammation. CT and MRI each have their own advantages in imaging principles and are complementary. Therefore, combining these two types of images for observation can improve the accuracy of diagnosis and provide a more reliable basis for formulating treatment plans.

[0003] However, when doctors observe and analyze different images, they need to rely on spatial imagination and subjective experience, which may lead to certain errors. This can be solved by medical image registration. Medical image registration technology is to align image data of different modalities or different time points, so that doctors can observe the morphology, location and range of lesions more clearly.

[0004] In the field of multi-organ multi-modal medical image registration, there are two major difficulties: 1. The problem of deformation field interference when fusing the registration results of different single organs, and 2. How to make the model learn the spatial correspondence between different modal images. In addition, the registration of multiple organs in the abdomen and head and neck is even more challenging, and there are few studies in this area. The distribution of abdominal organs is complex. Since the abdomen contains many organs and each organ occupies a small proportion in the image, usually only 10% to 30% of the entire image, this requires the registration algorithm to process the characteristics of each organ more finely. In addition, the deformation pattern of multiple abdominal organs is significantly different from that of multi-region registration problems such as lung parenchyma and interstitial lungs. The latter usually has relatively smooth and dynamic deformation characteristics, while the deformation of a single abdominal organ will cause complex non-smooth deformation in the surrounding area. In head and neck registration, due to the large elasticity of the soft tissue of the head and neck, there are common adhesions and occlusions between organs, which will lead to insufficient smoothness of the overall registration results. In addition, there are many organs in the head and neck and their structures are complex, and traditional registration methods often find it difficult to provide valuable deformation information.

[0005] Patent document with application number CN202311107895.4 discloses an abdominal multi-organ registration method based on deep probability map and vector fusion, which uses several parallel VoxelMorphs to generate deformation fields and global probability maps of different organs, and introduces the VFF vector field fusion module to fuse the deformation fields of each organ using the attention mechanism. Although this method can overcome the deformation field interference problem during the fusion of multi-organ deformation fields and reduce the fusion error, the deformation field of each organ in this method is generated independently, and there is a lack of information exchange between different organs and modalities when generating each single organ deformation field. Therefore, the ability to capture the spatial correspondence between different organs and modalities is limited, resulting in the registration results for the fusion of multi-modal and multi-organ information. Still insufficient.

[0006] The patent document with application number CN202310944659.1 discloses an abdominal multi-organ registration method based on an adaptive multi-gated hybrid expert model, which first uses the encoder of VoxelMorph to fuse the information of the two modal images, and then modifies the decoder part of the VoxelMorph network into three parallel multi-branch networks, using different branches to generate feature maps corresponding to different organs, and uses an adaptive multi-gated hybrid expert model to fuse the feature maps of all organs, thereby obtaining a fused feature map containing information of different modalities and different organs; then, through three parallel upsampling branch networks, a single organ deformation map of each organ is generated according to the fused feature map, and finally the single organ deformation maps of all organs are added to obtain an overall multi-organ deformation map, and the registration result is obtained accordingly. Although this method fuses the information of multi-modal and multi-organ, the fusion method is relatively rough and has the following two shortcomings:

[0007] First, when fusing multimodal information, only a simple CNN encoder is used to fuse images of different modalities in a downsampling manner. This fusion method cannot learn the spatial correspondence between images of different modalities well, thus affecting the final registration effect.

[0008] Second, when fusing multi-organ information, the multi-organ deformation field is obtained by simply adding together the deformation fields of different single organs. This method of fusing single-organ deformation fields is too simple and lacks the mining and utilization of the relationship between the deformation fields of each single organ, which limits the ability to learn the correspondence between different organs, resulting in insufficient utilization of the multi-organ fusion information in the registration results. Summary of the invention

[0009] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a multi-organ registration method based on cross-modal attention mechanism and vector fusion, so as to better learn the spatial correspondence between images of different modalities, reduce the mutual interference between the registration deformation fields of each organ, reduce the errors caused by the registration of each organ, and improve the overall registration effect.

[0010] To achieve the above objectives, the technical idea of ​​the present invention is: in terms of cross-modal image registration, the spatial correspondence between the features of CT and MRI images of two different modalities is learned by using a cross-modal attention mechanism, and the deep features extracted from the two modalities are mapped to the same feature to achieve cross-modal feature fusion; in terms of multi-organ registration, the deformation field of each organ is generated by fusing the features of different modal information, and the registration deformation field of each organ is optimized separately by returning the loss of each organ, and then the deformation field is fused using a vector gradient fusion module to reduce the direct conflict of multi-organ deformation field fusion, and finally achieve cross-modal multi-organ registration.

[0011] According to the above technical ideas, the implementation steps of the present invention include the following:

[0012] (1) Obtain abdominal and head and neck CT and MRI data, screen and preprocess them, and divide the abdominal and head and neck CT and MRI image data into training sets, validation sets, and test sets respectively;

[0013] (2) Constructing a multi-modal multi-organ registration network CAVF-MMR:

[0014] (2a) A multimodal information encoder U1 consisting of several cascaded downsampling convolution blocks and several cascaded cross-modal attention mechanism modules is established to extract the feature maps fea of ​​the two modal images of CT and MRI. cross_0 、 across_1 ;

[0015] (2b) establishing a single organ deformation map decoder U2 including multiple branches, for decoding the single organ deformation map of each organ;

[0016] (2c) Select the existing U-Net network as the global probability map generation network u3 to generate the global probability map g a ;

[0017] (2d) establishing a vector fusion module U8 including a multi-head attention mechanism module U4, a Poisson fusion module u5, a multi-organ deformation map decoding network u6, and a spatial transformation module U7, for fusing all single-organ deformation maps;

[0018] (2e) Connect the multimodal information encoder u1 to the single organ deformation map decoder u2, and then cascade the single organ deformation map decoder U2, the probability map generation network U3 and the vector fusion module U8 to form a multimodal multi-organ registration network CAVF-MMR based on cross-modal attention mechanism and vector fusion;

[0019] (3) Training the multi-modal multi-organ registration network CAVF-MMR:

[0020] (3a) Define the global loss function L of the multimodal multi-organ registration network CAVF-MMR:

[0021] L=∑L i +L a +L mse +L sm

[0022] Where L i is the Dice loss of the i-th organ in the reference image A and the registered image B, L a is the Dice loss of the union of all organs in the reference image A and the registered image B, L mse is the MSE loss of the union of all organs in the reference image A and the registered image B, L sm is the smoothing loss of the registered image B;

[0023] (3b) Input a batch of data in the training set into the multimodal multi-organ registration network, and t and its label At , registration image B t and its registration label l Bt , use the global loss function to calculate the global loss value L of the training set t ;

[0024] (3c) Global loss L for the training set t Perform back propagation iteration to update network parameters;

[0025] (3d) Based on the reference image A of a batch of data in the validation set v and its label At , registration image B v and its registration label l Bv Calculate the global loss L of the validation set v , used to evaluate the registration performance of the current registration network;

[0026] (3e) Repeat (3b) to (3c) until the global loss of the training set is L t Converge and obtain a trained registration network;

[0027] (4) The images to be registered in the test set are input into the trained registration network model to obtain the multi-organ registration results of the test set.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] 1. Since the present invention uses a cross-modal attention mechanism in the established multimodal information encoder U1, information of different modalities is input into the attention mechanism layer to obtain the position attention information of one modal information to another modal information, thereby learning the cross-modal spatial position correspondence and improving the accuracy of image registration.

[0030] 2. The global loss function L defined in the present invention not only takes into account the overlap of organ registration, but also takes into account the structural restoration of organ registration and the smoothness of the edges of the registered organs, so that the defined global loss function L contains rich information and can accurately and comprehensively guide the registration network to generate accurate and smooth registration results.

[0031] 3. In the single organ deformation map decoder U2, the present invention fuses the feature maps of different levels extracted by the multimodal information encoder U1 to obtain a fused feature map containing multimodal image structure information and cross-modal spatial relationship information, and uses this as the input of each branch network in the single organ deformation map decoder U2, so that the registration network can better understand and utilize the complementary information between multiple modalities, thereby enhancing the multimodal registration capability of the registration network.

[0032] 4. The present invention utilizes a multi-head attention mechanism and a customized Laplace operator in the vector fusion module U8 to improve the Poisson fusion strategy, effectively integrates multiple single-organ vector fields, reduces the mutual interference between organ correspondence problems and deformation maps in the multi-organ registration process, and enhances the multi-organ registration capability of the registration network. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flowchart of the implementation process of the present invention;

[0034] Figure 2 It is a schematic diagram of the overall structure of the multi-modal multi-organ registration neural network model constructed in the present invention;

[0035] Figure 3 It is a structural diagram of the cross-modal attention mechanism module established in the present invention;

[0036] Figure 4 It is a structural diagram of the vector fusion module established in the present invention;

[0037] Figure 5 This is a comparison diagram of the effects of the present invention and five existing registration methods on registration of abdominal bladder, cervix and rectum organs;

[0038] Figure 6 This is a comparison chart of the registration effects of the present invention and five existing registration methods on the head and neck oral cavity, left parotid gland, right parotid gland and brainstem organs. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the present invention, not all embodiments. Based on the embodiment of the present invention, other embodiments obtained by ordinary technicians in this field without making creative work should all belong to the protection scope of the present invention.

[0040] Reference Figure 1 The implementation steps of this example include the following:

[0041] Step 1: Obtain data, preprocess it, and divide it into training set, validation set, and test set.

[0042] (1.1.1) Obtain abdominal data, preprocess it and classify it:

[0043] Obtain abdominal data from the medical department, screen out 2D image slices that have both CT and MRI modal images and contain the three organs and labels of the abdominal bladder, cervix, and rectum, and perform grayscale value normalization and standardization preprocessing on them in turn to obtain the final 2D abdominal CT and MRI image data;

[0044] (1.1.2) The final abdominal data is divided into an abdominal training set, an abdominal validation set, and an abdominal test set in a ratio of 8:1:1;

[0045] (1.2) Obtain head and neck data, preprocess and classify it:

[0046] (1.2.1) Obtain head and neck data from public datasets, select 2D image slices that have both CT and MRI modal images and contain four organs and labels of the head and neck, namely, oral cavity, left parotid gland, right parotid gland, and brainstem, and then rigidly align the MRI and CT images in each pair of data, and perform normalization and standardization preprocessing in turn to obtain the final 2D head and neck CT and MRI image data;

[0047] (1.2.2) The final head and neck data is divided into head and neck training set, head and neck validation set and head and neck test set in a ratio of 8:1:1.

[0048] Step 2: Construct a multimodal multi-organ registration network CAVF-MMR based on cross-modal attention mechanism and vector fusion.

[0049] Reference Figure 2 , this step includes the following:

[0050] (2.1) A multimodal information encoder U1 consisting of several cascaded downsampling convolution blocks and several cascaded cross-modal attention mechanism modules is established to extract the feature maps fea of ​​the two modal images of CT and MRI. cross_0 、fea cross_1 ;

[0051] (2.1.2) Construct downsampling convolution block C d :

[0052] Create multiple downsampling convolutional layers consisting of convolution kernels, batch normalization, and ReLU activation connections;

[0053] Each two downsampling convolutional layers are connected with a maximum pooling layer to form a downsampling convolutional block C. d ;

[0054] (2.1.3) Constructing the cross-modal attention mechanism module M A :

[0055] Reference Figure 3 , the implementation of this step includes the following:

[0056] The three convolution kernels are cascaded with the three flattening layers to form three convolution layers Conv q 、Conv k and Conv v ;

[0057] The three Conv q 、Conv k 、Conv v They are connected to the query input, key input, and value input of the attention mechanism layer to form a cross-modal attention mechanism module M. A ;

[0058] (2.1.4) The three downsampling convolution blocks C d With 3 cross-modal attention mechanism modules M A Cascaded into a branch, the two branches are connected in parallel to form a multimodal information encoder U1.

[0059] (2.2) Establish a single organ deformation map decoder U2 including multiple branches to decode the single organ deformation map g of each organ i , where i represents the i-th organ;

[0060] (2.2.1) Establish a deconvolution block consisting of two convolutional layers and one deconvolutional layer in cascade, where each convolutional layer consists of a convolution kernel, batch normalization, and LeakyReLU activation, and the deconvolutional layer consists of a deconvolution kernel, batch normalization, and LeakyReLU function connection;

[0061] (2.2.2) Establish a convolutional block consisting of two convolutional layers in cascade, where each convolutional layer consists of a convolution kernel, batch normalization, and LeakyReLU activation connection;

[0062] (2.2.3) Cascade two deconvolution blocks and one convolution block to form a single organ deformation map decoding branch i , decode n single organ deformation maps into branches i The multi-branched single-organ deformation map decoder U2 is connected in parallel, where n is the total number of organs and i represents the i-th organ;

[0063] (2.3) Select the existing U-Net network as the global probability map generation network U3 to generate the global probability map g a ;

[0064] (2.4) Establishing a vector fusion module U8 including a multi-head attention mechanism module U4, a Poisson fusion module U5, a multi-organ deformation map decoding network U6, and a spatial transformation module U7, for fusing all single-organ deformation maps;

[0065] Reference Figure 4 , this step includes the following:

[0066] (2.4.1) Constructing the multi-head attention mechanism module U4

[0067] Cascade a linear layer and a normalization layer to form the first linear network Linear1;

[0068] The three linear layers are cascaded with the three normalization layers to form the second linear network Linear2, the third linear network Linear3 and the fourth linear network Linear4. Then the three linear networks Linear2, Linear3 and Linear4 are connected in parallel to form the merged linear network Linaer. a ;

[0069] Combine the first linear network Linear1 and the merged linear network Linear a All of them are input into a multi-head attention layer, and multi-head attention interaction operation is performed to obtain the attention weights w between images of different modalities. i, forming a multi-head attention mechanism module U4, and taking the output of the first linear network Linear1 as the key input and value input of the multi-head attention, and merging the linear network flow a The output of is used as the query input for multi-head attention, where i is the i-th organ.

[0070] (2.4.2) Construct Poisson fusion module U5

[0071] Create a convolution kernel of size 3×3 and initialize the convolution kernel parameters, that is, set the values ​​of the first row of the convolution kernel to 0, 1, 0; the values ​​of the second row to 1, -4, 1; the values ​​of the third row to 0, 1, 0, and obtain the Laplacian operator Δ of the 2D image;

[0072] The attention weight w output by the multi-head attention mechanism module U4 is used as the weighted weight, and the Δg output by the Laplace operator Δ is used as the weighted weight. i The weighted summation operation is performed as a weighted object to obtain the Laplacian operator result Δg of the multi-organ deformation map g:

[0073]

[0074] Among them, w i is the attention weight of the ith organ, Δg i is the single organ deformation graph g of the ith organ i Laplace operator result; i = 1, 2, ..., n, n is the total number of organs;

[0075] (2.4.3) Establish a multi-organ deformation map decoding network U6 including an MLP layer and a normalization layer, wherein the output of the MLP layer is connected to the input of the normalization layer, and the input residual is connected to the output of the normalization layer;

[0076] (2.4.4) Construct a spatial transformation module U7 including deformation coordinate calculation and bilinear interpolation. It takes the multi-organ deformation map g and the image to be registered as input. First, the deformation coordinates based on the multi-organ deformation map g are obtained by deformation coordinate calculation. Then, the image to be registered and the labels of each organ are resampled into the deformation coordinates using bilinear interpolation to obtain the final registered image B and the registration labels l of each organ. B , realize multi-organ multi-modality registration of the image to be registered to the reference image A;

[0077] (2.4.5) The multi-head attention mechanism module U4 is connected in parallel with the Poisson fusion module U5, and then cascaded with the multi-organ deformation decoding network diagram U6 and the spatial transformation module U7 to form a vector fusion module U8.

[0078] (2.5) Connect the multimodal information encoder U1 to the single organ deformation map decoder U2, and then cascade the single organ deformation map decoder U2, the probability map generation network U3 and the vector fusion module U8 to form a multimodal multi-organ registration network CAVF-MMR based on cross-modal attention mechanism and vector fusion.

[0079] Step 3: Train the multi-modal multi-organ registration network CAVF-MMR.

[0080] (3.1) Define the global loss function L of the multimodal multi-organ registration network CAVF-MMR:

[0081] (3.1.1) Define the Dice loss L for each single organ in the reference image A and the registered image B i :

[0082]

[0083] is the Dice similarity coefficient, which is a metric used to evaluate the overlap between two samples. i represents the number of pixels of the i-th organ in the image, represents the kth organ in the i-th organ in the reference image A i pixels, represents the kth organ in the registered image B i pixels, k i =1,2,…,Y i ; i = 1, 2, ..., n, n is the total number of organs;

[0084] (3.1.2) Define the Dice loss L of the union of all organs in the reference image A and the registered image B a :

[0085]

[0086] is the Dice similarity coefficient, which is a metric used to evaluate the similarity or overlap between two samples. a represents the union of all organs, and Y a represents the number of pixels of all organs in the image. represents the kth organ in the union of all organs in the reference image A a pixels, represents the kth union of all organs in the registered image B a pixels, k a =1,2,…,Y a ;

[0087] (3.1.3) Define the MSE loss L of all organs in the reference image A and the registered image Bmse :

[0088]

[0089] w and h represent the width and height of the reference image A, respectively; x represents the pixel index in the horizontal direction of the image, and y represents the pixel index in the vertical direction;

[0090] (3.1.4) Define the smoothing loss L of the registered image B sm :

[0091]

[0092] w and h represent the width and height of the reference image A, respectively; x represents the pixel index in the horizontal direction of the image, and y represents the pixel index in the vertical direction;

[0093] (3.1.5) According to the loss function defined from (3.1.1) to (3.1.4), the global loss function L of the multimodal multi-organ registration network CAVF-MMR is obtained:

[0094] L=∑L i +L a +L mse +L sm .

[0095] (3.2) Input a batch of data in the training set into the multimodal multi-organ registration network, and then calculate the reference image A of the batch of data in the training set. t and its label At , registration image B t and its registration label l Bt , use the global loss function to calculate the global loss value L of this batch of data in the training set t ;

[0096] (3.3) Global loss L for the training set t Perform back propagation iteration to update network parameters;

[0097] (3.3.1) Set the hyperparameters of backpropagation: set the initial learning rate to 0.0001 and the batch size to 1;

[0098] (3.3.2) Use SGD as the optimizer for training the multimodal multi-organ registration network CAVF-MMR and set the global loss L of the training set to t Input into the SGD optimizer for back propagation to update the parameters of the CAVF-MMR network;

[0099] (3.4) Based on the reference image A of a batch of data in the validation set v and its label At , registration image Bv and its registration label l Bv Calculate the global loss L of the validation set v , used to evaluate the registration performance of the current registration network;

[0100] (3.5) Repeat (3.2) to (3.3) until the global loss of the training set is L t Converge and obtain a trained registration network;

[0101] Step 4: Use the trained registration network to obtain the registration result of the image to be registered.

[0102] 4.1) The image to be registered and its organ labels, the reference image and its organ labels are input into the trained CAVF-MMR network, which calculates the feature maps fea of ​​the two modal images of CT and MRI through the multimodal information encoder U1 in the network. cross_0 、fea cross_1 This feature map passes through the single organ deformation map decoder U2 in the network to obtain the single organ deformation map g of each organ i ;

[0103] 4.2) Input the organ labels of the image to be registered and the organ labels of the reference image into the global probability map generation network U3 to obtain the global probability map g a ;

[0104] 4.3) Single organ deformation map g of all organs i and the global probability map g a The final registration result is obtained through the vector fusion module U8.

[0105] The effect of the present invention can be further illustrated by the following simulation.

[0106] 1. Simulation conditions:

[0107] The simulation platform of this experiment is a computer with an Intel Core i7-8700K CPU and 32GB of memory. The operating system is Windows 10. The Pytorch 3.8 deep learning framework is used to build and train CAVF-MMR. The NVIDIA GeForce GTX1080Ti GPU and CUDA 11.2 are used to accelerate the training of the CAVF-MMR network and the generation of registered images.

[0108] The registration performance evaluation indicators used in the simulation include the Dyess similarity coefficient DSC, mutual information MI and average surface distance ASD. The specific calculation formulas are as follows:

[0109]

[0110] MI(A,B)=H(A)+H(B)-H(A,B)

[0111]

[0112] A represents the organ label of the reference image, B represents the organ label of the registered image, H(A) and H(B) represent the information entropy of label A and label B respectively, H(A,B) is the joint entropy of label A and label B, S(A) represents the surface pixel of the organ label of the reference image, S(B) represents the surface pixel of the organ label of the registered image, d(s A , S(B)) represents the shortest distance from any pixel of the reference image organ label to the surface pixel of the registered image organ label, d(s B ,S(A)) represents the shortest distance from any pixel of the prediction result to the pixel on the surface of the true label.

[0113] 2. Simulation content and results

[0114] Simulation 1. Under the above simulation conditions, the present invention and the five existing registration methods SyN, demons, VoxelMorph, DFMIR and PoissonNet are used to perform cross-modal multi-organ registration on abdominal organs, and the evaluation indicators of each on different organs are calculated: Dyss similarity coefficient DSC, mutual information MI and average surface distance ASD. The results are shown in Table 1.

[0115] Table 1 Comparative experimental results on the abdomen

[0116]

[0117] As can be seen from Table 1, although the demons algorithm takes into account the smoothness of the deformation field and is also suitable for cross-modal registration, it cannot accurately register to each organ. SyN is unable to complete the registration task due to the interference of noise under different modalities. The VoxelMorph registration method based on deep learning cannot learn the characteristics of each modality. Although the DFMIR registration method is suitable for cross-modal registration, it cannot solve the problem of deformation field conflict between organs, and weakens the deformation ability. Although the PoissonNet registration method is suitable for multi-organ registration, it cannot handle cross-modal problems. These existing registration methods have basically not completed the cross-modal multi-organ registration task. Due to the use of the cross-modal attention mechanism splicing and deformation field conflict fusion strategy, the present invention has obtained the best results in the three indicators of DICE, ASD and MI compared with the prior art.

[0118] Simulation 2: Under the above simulation conditions, the present invention and the five existing registration methods SyN, demons, VoxelMorph, DFMIR and PoissonNet are used to perform multi-modal multi-organ registration of the three organs of the abdomen, bladder, cervix and rectum. The registration results are shown in Figure 2. Figure 5 shown.

[0119] from Figure 5 It can be observed intuitively that in the multimodal multi-organ registration with large deformation, the Syn algorithm is basically unable to complete the registration. Although the Demons algorithm can solve the problem of large deformation across modalities, it cannot accurately focus on each organ, resulting in completely wrong registration results. The DFMIR method can pay close attention to the features suitable for organ cross-modal registration, but cannot complete multi-organ registration. Although PoissonNet can solve the conflict problem of multi-organ registration, it cannot complete cross-modal registration, resulting in the registration results of each organ not being very good. The multimodal multi-organ registration results of the present invention maintain the morphology and structure of each organ, with clear organ edges and accurate organ morphology.

[0120] Simulation 3. Under the above simulation conditions, the present invention and the five existing registration methods SyN, demons, VoxelMorph, DFMIR and PoissonNet are used to perform cross-modal multi-organ registration on the head and neck organs, and the Dyss similarity coefficient DSC, mutual information MI and average surface distance ASD are used to evaluate them. The results are shown in Table 2.

[0121] Table 2 Comparative experimental results on the head and neck

[0122]

[0123] It can be observed from Table 2 that for the four tiny soft tissues of the head and neck, neither the traditional algorithm nor the Demons, SyN, voxelmorph and DFMIR deep learning registration methods can complete the cross-modal multi-organ registration task. This is because the organs are too small and these registration methods cannot achieve organ registration. Although PoissonNet can basically achieve multi-organ registration, it cannot handle cross-modal images well. The multi-modal multi-organ registration results of the present invention maintain the morphology and structure of each organ, with clear organ edges and accurate organ morphology, thanks to the use of cross-modal attention mechanism, deformation map fusion strategy and Poisson image editing method.

[0124] Simulation 4, under the above simulation conditions, the present invention and the five existing registration methods SyN, demons, VoxelMorph, DFMIR and PoissonNet are used to perform multi-modal multi-organ registration on the four organs of the head and neck, oral cavity, left parotid gland, right parotid gland and brainstem. The results are shown in the figure. Figure 6 shown.

[0125] from Figure 6 It can be clearly observed from the deformation field that the deformation field generated by the registration method CAVF-MMR proposed in the present invention is mainly concentrated in the relevant areas of the four organs. Other registration methods cannot accurately register the four key organs at all. Although PoissonNet is slightly better than other methods, it cannot handle cross-modal registration, so it cannot learn relevant features, so it still cannot accurately register the four organs.

[0126] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. It is obvious that for professionals in this field, after understanding the content and principles of the present invention, it is possible to make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0127] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the implementation scheme of the present invention to facilitate understanding, and the order of the step numbers is not limited.

Claims

1. A multi-organ registration method based on cross-modal attention mechanism and vector fusion, characterized in that: include: (1) Obtain abdominal and head and neck CT and MRI data, screen and preprocess them, and divide the abdominal and head and neck CT and MRI image data into training sets, validation sets, and test sets respectively; (2) Constructing a multimodal multi-organ registration network CAVF-MMR based on cross-modal attention mechanism and vector fusion: (2a) A multimodal information encoder U1 consisting of several cascaded downsampling convolution blocks and several cascaded cross-modal attention mechanism modules is established to extract the feature maps fea of ​​the two modal images of CT and MRI. cross_0 、fea cross_1 ; (2b) establishing a single organ deformation map decoder U2 including multiple branches, for decoding the single organ deformation map of each organ; (2c) Select the existing U-Net network as the global probability map generation network U3 to generate the global probability map g a ; (2d) establishing a vector fusion module U8 including a multi-head attention mechanism module U4, a Poisson fusion module U5, a multi-organ deformation map decoding network U6, and a spatial transformation module U7, for fusing all single-organ deformation maps; (2e) Connect the multimodal information encoder U1 to the single organ deformation map decoder U2, and then cascade the single organ deformation map decoder U2, the probability map generation network U3 and the vector fusion module U8 to form a multimodal multi-organ registration network CAVF-MMR based on cross-modal attention mechanism and vector fusion; (3) Training the multi-modal multi-organ registration network MAVF-MMR: (3a) Define the global loss function L of the multimodal multi-organ registration network MAVF-MMR: L=∑L i +L a +L mse +L sm L i is the Dice loss of the i-th organ in the reference image A and the registered image B, L a is the Dice loss of the union of all organs in the reference image A and the registered image B, L mse is the MSE loss of the union of all organs in the reference image A and the registered image B, L sm is the smoothing loss of the registered image B; (3b) Input a batch of data in the training set into the multimodal multi-organ registration network, and t and its label At , registration image B t and its registration label l Bt , use the global loss function to calculate the global loss value L of the training set t ; (3c) Global loss L for the training set t Perform back propagation iteration to update network parameters; (3d) Based on the reference image A of a batch of data in the validation set v and its label At , registration image B v and its registration label l Bv Calculate the global loss L of the validation set v , used to evaluate the registration performance of the current registration network; (3e) Repeat (3b) to (3c) until the global loss of the training set is L t Converge and obtain a trained registration network; (4) The images to be registered in the test set are input into the trained registration network model to obtain the multi-organ registration results of the test set.

2. The method according to claim 1, characterized in that In step (1), the abdominal and head and neck CT and MRI data are screened and preprocessed, and divided into training set, validation set and test set, which is implemented as follows: (1a) 2D image slices with both CT and MRI modal images and labels of the three organs of the abdomen, namely, bladder, cervix, and rectum, are selected from the abdominal data obtained from the medical department, and grayscale value normalization and standardization preprocessing is performed on the slices to obtain the final 2D abdominal CT and MRI image data; (1b) The final abdominal data is divided into an abdominal training set, an abdominal validation set, and an abdominal test set in a ratio of 8:1:1; (1c) 2D image slices with both CT and MRI modal images and labels of four organs, namely, the oral cavity, left parotid gland, right parotid gland, and brainstem, are selected from the head and neck data obtained from the public dataset. Then, the MRI and CT images in each pair of data are rigidly registered, and normalized and standardized preprocessing is performed in turn to obtain the final 2D head and neck CT and MRI image data; (1d) The final head and neck data are divided into head and neck training set, head and neck validation set and head and neck test set in a ratio of 8:1:

1.

3. The method according to claim 1, characterized in that In step (2a), a multimodal information encoder U1 consisting of several cascaded downsampling convolution blocks and several cascaded cross-modal attention mechanism modules is established, which is implemented as follows: (2a1) Construct a downsampling convolution block: (2a1.1) Create a downsampling convolutional layer consisting of a convolution kernel, batch normalization, and ReLU activation connections; (2a1.2) Connect every two downsampling convolutional layers with a maximum pooling layer to form a downsampling convolutional block; (2a2) Construct a cross-modal attention mechanism module: (2a2.1) The three convolution kernels are cascaded with the three flattening layers to form three convolution layers Conv q 、Conv k and Conv v ; (2a2.2) The three Conv q 、Conv k 、Conv v They are connected to the query input, key input, and value input of the attention mechanism layer respectively to form a cross-modal attention mechanism module; (2a3) Cascade the three downsampling convolution blocks and the three cross-modal attention mechanism modules into one branch, and connect the two branches in parallel to form the multimodal information encoder U1.

4. The method according to claim 1, characterized in that: Step (2b) establishes a single organ deformation map decoder U2 including multiple branches, which is implemented as follows: (2b1) Establish a deconvolution block consisting of two convolutional layers and one deconvolutional layer in cascade, where each convolutional layer consists of a convolution kernel, batch normalization and LeakyReLU activation, and the deconvolution layer consists of a deconvolution kernel, batch normalization and LeakyReLU function connection; (2b2) ​​Building a convolutional block consisting of two convolutional layers in cascade, where each convolutional layer consists of a convolution kernel, batch normalization, and a LeakyReLU activation connection; (2b3) Two deconvolution blocks and one convolution block are cascaded to form a branch of the single organ deformation map decoder U2 i , where i represents the i-th organ, and the multiple-branch single-organ deformation map decoder U2 has a total of n branches, where n is the total number of organs.

5. The method according to claim 1, characterized in that The global probability map generation network U3 in step (2c) is composed of a cascaded downsampling network, a bottleneck layer and an upsampling network, and the downsampling network is connected to the corresponding network layer of the upsampling network at its different network layers, realizing information exchange between features of different scales, thereby being able to generate a global probability map that takes into account features of various scales.

6. The method according to claim 1, characterized in that Step (2d) establishes a vector fusion module U8 including a multi-head attention mechanism module U4, a Poisson fusion module U5, a multi-organ deformation map decoding network U6, and a spatial transformation module U7, and is implemented as follows: (2d1) Constructing multi-head attention mechanism module U4 Cascade a linear layer and a normalization layer to form the first linear network Linear1; The three linear layers are cascaded with the three normalization layers to form the second linear network Linear2, the third linear network Linear3 and the fourth linear network Linear4. Then the three linear networks Linear2, Linear3 and Linear4 are connected in parallel to form the merged linear network Linaer. a ; Combine the first linear network Linear1 and the merged linear network Linear a All are input into a multi-head attention layer for multi-head attention interaction operation to form a multi-head attention mechanism module U4, in which the output of the first linear network Linear1 is used as the key and value of the multi-head attention, and the linear network network flow is merged. a The output of is used as the query for multi-head attention. (2d2) Construct Poisson fusion module U5 Create a convolution kernel of size 3×3 and initialize the convolution kernel parameters, that is, set the values ​​of the first row of the convolution kernel to 0, 1, 0; the values ​​of the second row to 1, -4, 1; the values ​​of the third row to 0, 1, 0, and obtain the Laplacian operator Δ of the 2D image; The attention weight w output by the multi-head attention mechanism module U4 i As a weighted weight, the Δg output by the Laplace operator Δ i The weighted summation operation is performed as a weighted object to obtain the Laplacian operator result Δg of the multi-organ deformation map g: w i is the attention weight of the ith organ, Δg i is the single organ deformation graph g of the ith organ i Laplace operator result; i = 1, 2, ..., n, n is the total number of organs; (2d3) establishing a multi-organ deformation map decoding network U6 including an MLP layer and a normalization layer, wherein the output of the MLP layer is connected to the input of the normalization layer, and the input residual of the MLP layer is connected to the output of the normalization layer; (2d4) Construct a spatial transformation module U7 including deformation coordinate calculation and bilinear interpolation. It takes the multi-organ deformation map g and the image to be registered as input. First, the deformation coordinates based on the multi-organ deformation map g are obtained by deformation coordinate calculation. Then, the image to be registered and the labels of each organ are resampled into the deformation coordinates using bilinear interpolation to obtain the final registered image B and the registration labels l of each organ. B , realize multi-organ multi-modality registration of the image to be registered to the reference image A; (2d5) The multi-head attention mechanism module U4 is connected in parallel with the Poisson fusion module U5, and then cascaded with the multi-organ deformation decoding network diagram U6 and the spatial transformation module U7 to form a vector fusion module U8.

7. The method according to claim 1, characterized in that Step (3a) defines the different losses L involved in the global loss function L i , L a , L mse , L sm The calculation formulas are as follows: (3a1) Dice loss L of each single organ in the reference image A and the registered image B i : is the Dice similarity coefficient, which is a metric used to evaluate the overlap between two samples. i represents the number of pixels of the i-th organ in the image, represents the kth organ in the i-th organ in the reference image A i pixels, represents the kth organ in the registered image B i pixels, k i =1,2,…,Y i ; i = 1, 2, ..., n, n is the total number of organs; (3a2) Dice loss L of the union of all organs in the reference image A and the registered image B a : is the Dice similarity coefficient, which is a metric used to evaluate the similarity or overlap between two samples. a represents the union of all organs, and Y a represents the number of pixels of all organs in the image. represents the kth organ in the union of all organs in the reference image A a pixels, represents the kth union of all organs in the registered image B a pixels, k a =1,2,…,Y a ; (3a3) The MSE loss L of all organs in the reference image A and the registered image B mse : w and h represent the width and height of the reference image A, respectively; x represents the pixel index in the horizontal direction of the image, and y represents the pixel index in the vertical direction; (3a4) Smoothness loss L of the registered image B sm : w and h represent the width and height of the reference image A, respectively; x represents the pixel index in the horizontal direction of the image; and y represents the pixel index in the vertical direction.

8. The method according to claim 1, characterized in that In step (3c), the global loss L for the training set is t Perform back propagation to iteratively update the network parameters, which can be achieved as follows (3c1) Set the hyperparameters of back propagation: set the initial learning rate to 0.0001 and the batch size to 1; (3c2) Use SGD as the optimizer for training the multimodal multi-organ registration network CAVF-MMR and set the global loss L of the training set to t The result is fed into the SGD optimizer for back-propagation and iteratively updates the parameters of the CAVF-MMR network.

Images