A multi-modality based digestive tract tumor image fusion method
By extracting the multimodal features of MR and PET images of digestive tract tumors and using the cross-attention mechanism and residual connection module, the problems of insufficient information and loss of single modal feature are solved, and the accuracy of image fusion is improved and diagnostic effects.
Patent Information
- Application Number
- CN202411273212.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-09-12
AI Technical Summary
The existing multimodal digestive tract tumor image fusion method has problems such as insufficient extraction of single modal feature information and loss of global texture information in convolutional neural networks.
By extracting the frequency and spatial domain features of the MR image and combining the brightness and grayscale features of the PET image, the cross-attention mechanism and residual connection module fusion features are used to ensure the effective fusion of global and local information.
It improves the comprehensive capture and diagnostic accuracy of image information, reduces information loss, and enhances the evaluation and treatment effect of digestive tract tumors.
Smart Images

Figure CN118799691B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image fusion, and in particular to a multi-modality-based digestive tract tumor image fusion method. Background Art
[0002] Gastrointestinal tumors are one of the most common and deadly types of cancer worldwide. Early diagnosis and accurate staging are crucial to improving patient survival rates. However, single-modality imaging technology often fails to fully reflect the complexity and heterogeneity of tumors. Multimodal image fusion methods combine the advantages of different imaging technologies to provide more comprehensive and accurate tumor information. By fusing these multimodal images, the size, location, infiltration depth of the tumor and its relationship with surrounding tissues can be more accurately assessed, thereby improving the accuracy of diagnosis and the effectiveness of treatment. However, existing multimodal gastrointestinal tumor image fusion methods have the problem of insufficient extraction of single-modality feature information and the problem of losing global texture information when using convolutional neural networks. Summary of the invention
[0003] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides a method for gastrointestinal tumor image fusion based on multimodality, specifically for fusing MR images and PET images of gastrointestinal tumors. In view of the problem of insufficient extraction of single modality feature information, the present invention extracts and fuses the frequency domain features and spatial domain features of MR images, and extracts and fuses the brightness features and grayscale features of PET images based on the respective characteristics of MR images and PET images, thereby ensuring comprehensive capture of image information; in view of the problem of losing global texture information when using convolutional neural networks, the present invention utilizes residual connections to fuse the original information of input features during output feature processing, and at the same time utilizes a cross self-attention mechanism in the feature fusion process to ensure effective fusion of global and local features, thereby alleviating the problem of information loss.
[0004] The technical solution adopted by the present invention is as follows: The present invention provides a digestive tract tumor image fusion method based on multimodality, and the specific steps are as follows:
[0005] Step S1: obtaining MR images and PET images of digestive tract tumors;
[0006] Step S2: extracting the frequency domain features and the spatial domain features of the MR image respectively, and fusing the frequency domain features and the spatial domain features to obtain the MR features;
[0007] Step S3: extracting brightness features and grayscale features of the PET image respectively, and fusing the brightness features and grayscale features to obtain PET features;
[0008] Step S4: Fuse the MR features and PET features through the cross self-attention mechanism to obtain fused features;
[0009] Step S5: Use a convolution module to process the fused features to obtain a fused image.
[0010] Furthermore, the step S2 specifically includes:
[0011] Step S21: Extracting frequency domain features of the MR image: Performing a two-dimensional discrete Fourier transform on the MR image, converting each pixel into a complex number consisting of a real part and an imaginary part, and obtaining the amplitude and phase of the image by calculating the complex number to form the frequency domain features. The specific formula is as follows:
[0012] The formula for the two-dimensional discrete Fourier transform is as follows:
[0013] ;
[0014] in, represents the two-dimensional discrete Fourier transform result, and is the frequency variable, and is the spatial domain variable, In MR images The pixel value of the element, and are the maximum number of rows and columns of the image, respectively. ;
[0015] The amplitude is calculated as follows:
[0016] | F(u,v) |= [ R(u,v) 2 + I(u,v) 2 ] 1 2 ;
[0017] in, express The real part of express The imaginary part of
[0018] The phase calculation formula is as follows:
[0019] ϕ(u,v) =arctan[ I(u,v) R(u,v) ] ;
[0020] in, Indicates phase;
[0021] Step S22: extracting the spatial domain features of the MR image of the digestive tract tumor. The specific steps are as follows:
[0022] Step S221: extracting edge features of the MR image using the Laplace operator and the Sobel operator respectively to obtain Laplace edge features and Sobel edge features;
[0023] Step S222: define Laplace weight and Sobel weight, multiply Laplace weight by Laplace edge feature, multiply Sobel weight by Sobel edge feature, add the product of the two, and obtain edge feature;
[0024] Step S223: passing the MR image through a convolution module and three consecutive residual connection modules in sequence to obtain convolution features;
[0025] Step S224: Perform maximum fusion on the convolution features and edge features to obtain spatial domain features. The specific formula for maximum fusion is as follows:
[0026] ;
[0027] in, represents the convolution feature, represents edge features, Indicates taking the maximum value;
[0028] Step S23: Calculate the L1 regularization terms of the frequency domain features and the spatial domain features respectively, and use the L1 regularization terms to calculate the weights of the feature vectors of the channel dimensions of the frequency domain features and the spatial domain features through the Softmax function, and then calculate the sum of the products of the weights and the corresponding feature vectors to obtain the MR features. The specific formula is as follows:
[0029] ;
[0030] ;
[0031] in, Represents categories, including frequency domain features and spatial domain features. Indicates The weight of each category, Indicates The channel-dimensional vector of the features of the categories, Indicates The channel-dimensional vector of the features of the categories, and Respectively and The L1 regularization term of Represents MR characteristics.
[0032] Furthermore, the step S3 specifically includes:
[0033] Step S31: extracting brightness features of the PET image of the digestive tract tumor. The specific steps are as follows:
[0034] Step S311: Convert the RGB image of the PET image to the HSV color space to obtain the V component. The calculation formula of the V component is as follows:
[0035]
[0036] in, Represents the HSV color space Quantity, Represents the PET images The value of the channel;
[0037] Step S312: setting the brightness range and creating a mask matrix. If the V component of each pixel in the PET image is within the brightness range, the corresponding value of the mask matrix is 1, otherwise it is 0;
[0038] Step S313: multiplying the PET image by the mask matrix to obtain a differential image;
[0039] Step S314: performing a convolution operation on the difference image through a 3×3 convolution kernel and using tanh as an activation function to obtain the brightness feature of the PET image;
[0040] Step S32: Convert the PET image into a grayscale image. The grayscale image is sequentially passed through a convolution module and three consecutive residual connection modules to obtain grayscale features. The grayscale image conversion formula is as follows:
[0041] ;
[0042] in, Represents grayscale value;
[0043] Step S33: Calculate the L1 regularization terms of the brightness feature and the grayscale feature respectively, and use the L1 regularization term to calculate the weights of the feature vectors of the channel dimension of the brightness feature and the grayscale feature through the Softmax function, and then calculate the sum of the product of the weight and the corresponding feature vector to obtain the PET feature.
[0044] Furthermore, the step S4 specifically includes:
[0045] Step S41: respectively obtaining a query vector, a key vector and a value vector of the MR feature, and respectively obtaining a query vector, a key vector and a value vector of the PET feature;
[0046] Step S42: Calculate the attention score of the query vector of the MR feature and the key vector of the PET feature, multiply the attention score by the value vector of the PET feature after passing the Softmax function, and obtain the first attention value. The formula of this process is as follows:
[0047] ;
[0048] in, is the first attention value, is the Softmax function, is the query vector of MR features, is the transposed matrix of the key vector of the PET feature, For the dimension, is the value vector of PET features;
[0049] Step S43: Calculate the attention score of the query vector of the PET feature and the key vector of the MR feature, multiply the attention score by the value vector of the MR feature after passing the Softmax function, and obtain the second attention value. The formula of this process is as follows:
[0050] ;
[0051] in, is the second attention value, is the query vector of PET features, is the transposed matrix of the key vector of the MR feature, is the value vector of MR features;
[0052] Step S44: perform a connection operation on the first attention value and the second attention value to obtain a fusion feature.
[0053] The residual connection module includes input features, four dense connection blocks, independent convolution kernels and output features: each dense connection block consists of The network consists of convolution kernels, Leaky ReLU and a connection layer. The four densely connected blocks are connected in sequence, and the number of convolution kernels is 6, 6, 6, and 4 respectively. The connection layer of each densely connected block performs a connection operation on the output result of Leaky ReLU, the input features, and the connection layer of the densely connected block before the densely connected block. The input features are added to the output results of the independent convolution kernels to obtain the output features.
[0054] The convolution module includes a 3×3 convolution kernel, Leaky ReLU and a 3×3 convolution kernel, and each part is connected in sequence.
[0055] The beneficial results achieved by the present invention using the above scheme are as follows:
[0056] (1) To address the problem of insufficient extraction of MR image feature information, the present invention extracts the frequency domain features of MR images through two-dimensional discrete Fourier transform to capture the global information and periodic features in the image. At the same time, the Laplace operator and Sobel operator are used to extract the edge features of the MR images. The convolution module and the residual connection module are used to further extract the convolution features to comprehensively obtain the spatial domain features.
[0057] (2) To address the problem of insufficient feature information extraction of PET images, the present invention performs HSV color space conversion on PET images to extract brightness features, thereby enhancing the contrast and details of the images. At the same time, the PET images are converted into grayscale images, and grayscale features are extracted through convolution modules and residual connection modules to capture the details and texture information in the images.
[0058] (3) To address the problem of losing global texture information when using convolutional neural networks, the present invention uses residual connections to fuse the original information of input features when processing output features. At the same time, the cross-self-attention mechanism is used in the feature fusion process to ensure the effective fusion of global and local features, thereby alleviating the problem of information loss. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Picture 1 This is a framework diagram of a multi-modality-based digestive tract tumor image fusion model proposed by the present invention;
[0060] Picture 2 The flowchart of extracting spatial domain features of MR images proposed by the present invention;
[0061] Picture 3 This is a framework diagram of the residual connection module proposed in the present invention.
[0062] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION
[0063] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0064] Example 1, see Picture 1 , Picture 1 This is a framework diagram of a multimodal digestive tract tumor image fusion model proposed by the present invention. In this embodiment, the method includes the following steps:
[0065] Step S1: obtaining MR images and PET images of digestive tract tumors;
[0066] Step S2: extracting the frequency domain features and the spatial domain features of the MR image respectively, and fusing the frequency domain features and the spatial domain features to obtain the MR features;
[0067] Step S3: extracting brightness features and grayscale features of the PET image respectively, and fusing the brightness features and grayscale features to obtain PET features;
[0068] Step S4: Fuse the MR features and PET features through the cross self-attention mechanism to obtain fused features;
[0069] Step S5: Use a convolution module to process the fused features to obtain a fused image.
[0070] In this embodiment, 490 MR-PET image pairs are selected as training sets. The size of each image is 256×256. During the training process, the initial learning rate is set to 0.001, the maximum number of iterations is set to 400, and the batch size is set to 4. Adam is used as the optimization function, and Python 3.9 is used for implementation. The PyTorch version is 1.8.1, the corresponding CUDA version is 11.3, and the GPU model is NVIDIA GeForce RTX 4080. During the training process, the SSIM structural similarity indicator is used as the loss function.
[0071] Example 2, see Picture 2 , Picture 2 This is a flowchart of extracting spatial domain features from MR images proposed by the present invention. Based on the above embodiments, this embodiment implements step S2, which specifically includes the following steps:
[0072] Step S21: Extracting frequency domain features of the MR image: Performing a two-dimensional discrete Fourier transform on the MR image, converting each pixel into a complex number consisting of a real part and an imaginary part, and obtaining the amplitude and phase of the image by calculating the complex number to form the frequency domain features. The specific formula is as follows:
[0073] The formula for the two-dimensional discrete Fourier transform is as follows:
[0074] ;
[0075] in, represents the two-dimensional discrete Fourier transform result, and is the frequency variable, and is the spatial domain variable, In MR images The pixel value of the element, and are the maximum number of rows and columns of the image, respectively. ;
[0076] The amplitude is calculated as follows:
[0077] | F(u,v) |= [ R(u,v) 2 + I(u,v) 2 ] 1 2 ;
[0078] in, express The real part of express The imaginary part of
[0079] The phase calculation formula is as follows:
[0080] ϕ(u,v)=arctan[ I(u,v) R(u,v) ] ;
[0081] in, Indicates phase;
[0082] Step S22: extracting the spatial domain features of the MR image of the digestive tract tumor. The specific steps are as follows:
[0083] Step S221: extracting edge features of the MR image using the Laplace operator and the Sobel operator respectively to obtain Laplace edge features and Sobel edge features;
[0084] Step S222: define Laplace weight and Sobel weight, multiply the Laplace weight by the Laplace edge feature, multiply the Sobel weight by the Sobel edge feature, add the product of the two, and obtain the edge feature. In this embodiment, the Laplace weight and the Sobel weight are both defined as 1;
[0085] Step S223: passing the MR image through a convolution module and three consecutive residual connection modules in sequence to obtain convolution features;
[0086] Step S224: Perform maximum fusion on the convolution features and edge features to obtain spatial domain features. The specific formula for maximum fusion is as follows:
[0087] ;
[0088] in, represents the convolution feature, represents edge features, Indicates taking the maximum value;
[0089] Step S23: Calculate the L1 regularization terms of the frequency domain features and the spatial domain features respectively, and use the L1 regularization terms to calculate the weights of the feature vectors of the channel dimensions of the frequency domain features and the spatial domain features through the Softmax function, and then calculate the sum of the products of the weights and the corresponding feature vectors to obtain the MR features. The specific formula is as follows:
[0090] ;
[0091] ;
[0092] in, Represents categories, including frequency domain features and spatial domain features. Indicates The weight of each category, Indicates The channel-dimensional vector of the features of the categories, Indicates The channel-dimensional vector of the features of the categories, and Respectively and The L1 regularization term of Represents MR characteristics.
[0093] Embodiment 3: Based on the above embodiment, this embodiment implements step S3, which specifically includes the following steps:
[0094] Step S31: extracting brightness features of the PET image of the digestive tract tumor. The specific steps are as follows:
[0095] Step S311: Convert the RGB image of the PET image to the HSV color space to obtain the V component. The calculation formula of the V component is as follows:
[0096]
[0097] in, Represents the HSV color space Quantity, Represents the PET images The value of the channel;
[0098] Step S312: setting the brightness range and creating a mask matrix. If the V component of each pixel in the PET image is within the brightness range, the value corresponding to the mask matrix is 1, otherwise it is 0. In this embodiment, the brightness range is set to 0.8-1, that is, if the V component is in the interval [0.8-1], the value corresponding to the mask matrix is 1, and if it is in the interval [0-0.8) the value corresponding to the mask matrix is 0;
[0099] Step S313: multiplying the PET image by the mask matrix to obtain a differential image;
[0100] Step S314: performing a convolution operation on the difference image through a 3×3 convolution kernel and using tanh as an activation function to obtain the brightness feature of the PET image;
[0101] Step S32: Convert the PET image into a grayscale image. The grayscale image is sequentially passed through a convolution module and three consecutive residual connection modules to obtain grayscale features. The grayscale image conversion formula is as follows:
[0102] ;
[0103] in, Represents grayscale value;
[0104] Step S33: Calculate the L1 regularization terms of the brightness feature and the grayscale feature respectively, and use the L1 regularization term to calculate the weights of the feature vectors of the channel dimension of the brightness feature and the grayscale feature through the Softmax function, and then calculate the sum of the product of the weight and the corresponding feature vector to obtain the PET feature.
[0105] Embodiment 4: Based on the above embodiment, this embodiment implements step S4, which specifically includes the following steps:
[0106] Step S41: respectively obtaining a query vector, a key vector and a value vector of the MR feature, and respectively obtaining a query vector, a key vector and a value vector of the PET feature, wherein the query vector, the key vector and the value vector are all obtained by linearly transforming the input MR feature and the input PET feature using a trainable weight matrix;
[0107] Step S42: Calculate the attention score of the query vector of the MR feature and the key vector of the PET feature, multiply the attention score by the value vector of the PET feature after passing the Softmax function, and obtain the first attention value. The formula of this process is as follows:
[0108] ;
[0109] in, is the first attention value, is the Softmax function, is the query vector of MR features, is the transposed matrix of the key vector of the PET feature, For the dimension, is the value vector of PET features;
[0110] Step S43: Calculate the attention score of the query vector of the PET feature and the key vector of the MR feature, multiply the attention score by the value vector of the MR feature after passing the Softmax function, and obtain the second attention value. The formula of this process is as follows:
[0111] ;
[0112] in, is the second attention value, is the query vector of PET features, is the transposed matrix of the key vector of the MR feature, is the value vector of MR features;
[0113] Step S44: perform a connection operation on the first attention value and the second attention value to obtain a fusion feature.
[0114] Example 5, see Picture 3 , Picture 3 This is a framework diagram of the residual connection module proposed in the present invention. Based on the above embodiments, this embodiment implements the residual connection module and the convolution module. The specific contents are as follows:
[0115] The residual connection module includes input features, four dense connection blocks, independent convolution kernels and output features: each dense connection block consists of The network consists of convolution kernels, Leaky ReLU and a connection layer. The four densely connected blocks are connected in sequence and the number of convolution kernels is 6, 6, 6, and 4 respectively. The connection layer of each densely connected block performs a connection operation on the output result of Leaky ReLU, the input features, and the connection layer of the densely connected block before the densely connected block. The input features are added to the output results of the independent convolution kernel to obtain the output features. The sizes of the independent convolution kernels and the convolution kernels in the densely connected blocks are both 3×3.
[0116] The convolution module contains a 3×3 convolution kernel, Leaky ReLU and a 3×3 convolution kernel, and each part is connected in sequence.
Claims
1. A multi-modality digestive tract tumor image fusion method. Features: The following steps are involved: Step S1: acquiring MR images and PET images of digestive tract tumors; Step S2: extracting the frequency domain features and the spatial domain features of the MR image respectively, and fusing the frequency domain features and the spatial domain features to obtain the MR features; Step S3: extracting brightness features and grayscale features of the PET image respectively, and fusing the brightness features and grayscale features to obtain PET features; Step S4: Fuse the MR features and PET features through the cross self-attention mechanism to obtain fused features; Step S5: using a convolution module to process the fused features to obtain a fused image; The step S4 is specifically as follows: Step S41: respectively obtaining a query vector, a key vector and a value vector of the MR feature, and respectively obtaining a query vector, a key vector and a value vector of the PET feature; Step S42: Calculate the attention score of the query vector of the MR feature and the key vector of the PET feature, and multiply the attention score by the value vector of the PET feature after passing the Softmax function to obtain a first attention value; Step S43: Calculate the attention score of the query vector of the PET feature and the key vector of the MR feature, and multiply the attention score by the value vector of the MR feature after passing the Softmax function to obtain a second attention value; Step S44: perform a connection operation on the first attention value and the second attention value to obtain a fusion feature.
2. According to claim 1, a multi-modality digestive tract tumor image fusion method, Features: The step S2 is specifically as follows: Step S21: extracting frequency domain features of the MR image: performing a two-dimensional discrete Fourier transform on the MR image, converting each pixel into a complex number consisting of a real part and an imaginary part, and obtaining the amplitude and phase of the image by calculating the complex number to form frequency domain features; Step S22: extracting the spatial domain features of the MR image. The specific steps are as follows: Step S221: extracting edge features of the MR image using the Laplace operator and the Sobel operator respectively to obtain Laplace edge features and Sobel edge features; Step S222: define Laplace weight and Sobel weight, multiply Laplace weight by Laplace edge feature, multiply Sobel weight by Sobel edge feature, add the product of the two, and obtain edge feature; Step S223: passing the MR image through a convolution module and three consecutive residual connection modules in sequence to obtain convolution features; Step S224: performing maximum fusion on the convolution feature and the edge feature to obtain the spatial domain feature; Step S23: Calculate the L1 regularization terms of the frequency domain features and the spatial domain features respectively, and use the L1 regularization terms to calculate the weights of the feature vectors of the channel dimensions of the frequency domain features and the spatial domain features through the Softmax function, and then calculate the sum of the products of the weights and the corresponding feature vectors to obtain the MR features.
3. According to claim 1, a multi-modality digestive tract tumor image fusion method, Features: The step S3 is specifically as follows: Step S31: extracting brightness features of the PET image. The specific steps are as follows: Step S311: converting the RGB image of the PET image into the HSV color space to obtain the V component; Step S312: setting the brightness range and creating a mask matrix. If the V component of each pixel in the PET image is within the brightness range, the corresponding value of the mask matrix is 1, otherwise it is 0; Step S313: multiplying the PET image by the mask matrix to obtain a differential image; Step S314: performing a convolution operation on the difference image through a 3×3 convolution kernel and using tanh as an activation function to obtain the brightness feature of the PET image; Step S32: converting the PET image into a grayscale image, passing the grayscale image through a convolution module and three consecutive residual connection modules in sequence to obtain grayscale features; Step S33: Calculate the L1 regularization terms of the brightness feature and the grayscale feature respectively, and use the L1 regularization term to calculate the weights of the feature vectors of the channel dimension of the brightness feature and the grayscale feature through the Softmax function, and then calculate the sum of the product of the weight and the corresponding feature vector to obtain the PET feature.
4. According to claim 1, a multi-modality digestive tract tumor image fusion method, Features: The convolution module contains a 3×3 convolution kernel, Leaky ReLU and a 3×3 convolution kernel, and each part is connected in sequence.
5. A multi-modality digestive tract tumor image fusion method according to claim 2 or 3, Features: The residual connection module includes input features, four dense connection blocks, independent convolution kernels and output features: each dense connection block consists of The network consists of convolution kernels, Leaky ReLU and a connection layer. The four densely connected blocks are connected in sequence, and the number of convolution kernels is 6, 6, 6, and 4 respectively. The connection layer of each densely connected block performs a connection operation on the output result of Leaky ReLU, the input features, and the connection layer of the densely connected block before the densely connected block. The input features are added to the output results of the independent convolution kernels to obtain the output features.
