Multi-level multi-view multi-mode segmentation method for medical image
By using the 3MU-Net network model and leveraging the cross-field context cross-sensing processor and CMformer module, the problem of insufficient semantic information in cross-modal medical image segmentation is solved, achieving accurate segmentation of PET/CT images and improving segmentation accuracy and the ability to express lesion regions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-22
- Publication Date
- 2026-04-07
AI Technical Summary
Existing cross-modal medical image segmentation networks fail to fully exploit the semantic information of cross-modal medical images, resulting in defects in the expression of lesion regions in PET/CT images and reducing image segmentation accuracy.
The 3MU-Net network model is adopted, combined with the U-Net encoder, cross-view context cross-sensing processor and CMformer module. Through cross-modal and context semantic information cross-sensing processing, multi-view multimodal features are extracted to achieve accurate segmentation.
It improves the segmentation accuracy of medical images, enhances the ability to express lesion areas, and improves the accuracy and consistency of image segmentation.
Smart Images
Figure CN121811035A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and more specifically to a multi-level, multi-view, and multi-modal segmentation method for medical images. Background Technology
[0002] Cross-modal medical images can reliably and accurately describe lesion category attributes, which is of great significance for clinical diagnosis. PET / CT images are one of the important multimodal images, combining the high sensitivity of PET images for lesion detection with the anatomical information of CT images. This allows for the comprehensive representation of multi-source information such as the function and anatomy of the same lesion organ on a single image, ensuring accurate segmentation of the lesion region. However, due to limitations in fusion techniques, most encoder-decoder segmentation networks only extract target slices and their continuous slice information from a single modality. Although this improves the network's segmentation performance for medical images, it cannot fully exploit the semantic information of cross-modal medical images and ignores the feature representation capabilities of other modalities of medical images for lesions. Therefore, PET / CT images have deficiencies in representing lesion regions, reducing the segmentation accuracy of the images.
[0003] Therefore, how to improve the segmentation accuracy of cross-modal images is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0004] In view of the above problems, this invention is proposed to provide a multi-level, multi-view, multi-modal segmentation method for medical images that overcomes or at least partially solves the above problems. It utilizes a cross-view contextual cross-sensing processor and CMformer to fully extract cross-modal and contextual semantic information, thereby achieving accurate segmentation of medical images.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a multi-level, multi-view, and multi-modal segmentation method for medical images, comprising the following steps: Step 1: Acquire PET / CT images, PET images, and CT images, and label them to build a training dataset; Step 2: Construct a 3MU-Net network model and train it using the training dataset to obtain the target segmentation model. The target segmentation model includes a U-Net encoder, a cross-view contextual cross-sensing processor, a CMformer module, and a decoder. The U-Net encoder extracts multi-modal feature maps from PET / CT images. The cross-view contextual cross-sensing processor extracts cross-modal and contextual semantic information from the multi-modal feature maps to obtain multi-view target features. Four sets of CMformer modules are used to learn the detailed information of the multi-modal feature maps to establish cross-modal pixel dependencies and obtain cross-modal fusion feature maps. The decoder decodes and segments the target based on the multi-view target features and the cross-modal fusion feature maps to obtain the segmentation result. Step 3: Acquire the PET / CT image to be segmented and input it into the target segmentation model to obtain the segmentation result.
[0006] Preferably, the U-Net encoder includes a PET / CT branch, a CT branch, and a PET branch; the PET / CT branch includes four sets of sequentially stacked downsampling blocks, which downsample the PET / CT image four times, and the PET / CT feature map output by each set of downsampling blocks is input to a set of CMformer modules; the CT branch and the PET branch each include three sets of sequentially stacked downsampling blocks; the CT branch downsamples the CT image three times, and the CT feature map output by each set of downsampling blocks and the final output CT feature map are input to a set of CMformer modules; the PET branch downsamples the PET image three times, and the PET feature map output by each set of downsampling blocks and the final output PET feature map are input to a set of CMformer modules.
[0007] Preferably, the CMformer module utilizes Transformer to calculate the correlation between corresponding pixels in the cross-modal feature map in the skip connection part, while simultaneously using CNN branches to learn cross-modal detail information, establishing cross-modal pixel dependencies in both local and global domains. This includes a Transformer branch, a CNN branch, and a cross-modal feature aggregation module. The Transformer branch performs cross-modal cross-attention calculation based on the feature maps of the three modalities to generate a cross-modal global feature map. The CNN branch performs local feature enhancement based on the feature maps of the three modalities to generate a cross-modal local feature map. The cross-modal feature aggregation module fuses the cross-modal global feature map and the cross-modal local feature map to obtain a cross-modal fused feature map.
[0008] Preferably, the Transformer branches include a first branch, a second branch, a third branch, and an adaptive fusion layer. The first, second, and third branches each include a Transformer input layer, a cross-modal attention layer, a self-attention layer, and a feature adjustment unit connected in sequence. The adaptive fusion layer includes three multiplication units and one addition unit. The Transformer input layer of the first branch uses the PET / CT feature map as query Q and the CT feature map as key values K and V. The Transformer input layer of the second branch uses the PET / CT feature map as query Q and the PET feature map concatenated with the CT feature map as key values K and V. The Transformer input layer of the third branch uses the PET / CT feature map as query Q and the PET feature map as key values K and V. The cross-modal attention layer calculates the correlation between query Q and each corresponding pixel in key values K and V to obtain cross-modal attention. The self-attention layer calculates self-attention based on the cross-modal attention. The feature adjustment unit converts the self-attention into branch feature maps. The adaptive fusion layer performs adaptive weight fusion on the feature maps of the three branches to obtain a cross-modal global feature map.
[0009] Preferably, the cross-modal attention layer and the self-attention layer each include a multi-head attention block, a first residual connection and layer normalization block Add&Norm, a multilayer perceptron (MLP), and a second residual connection and layer normalization block Add&Norm, which are connected in sequence; the feature adjustment unit includes a reshaping layer and an interpolation layer connected in sequence.
[0010] Preferably, the processing procedure for each branch of the Transformer is as follows: The Transformer input layer receives PET / CT feature maps, CT feature maps, and PET feature maps, and sets the query Q, key value K, and key value V; The cross-modal attention layer first utilizes three linear transformation matrices W Q W K and W v Project the query Q, key K, and key V into a cross-modal query matrix Q, respectively. C Cross-modal bond matrix K C and the cross-modal value matrix V C Secondly, the multi-head attention block (Multi-Head Attention) is used for cross-modal query matrix Q. C and the cross-modal bond matrix K C The similarity matrix is obtained by performing a dot product operation to calculate the similarity of elements in the matrix. The similarity matrix is then transformed into a probability distribution between 0 and 1 using the softmax function to obtain a probability matrix. Finally, the probability matrix is compared with the cross-modal value matrix V. CMultiplication yields the output of a cross-modal self-attention mechanism. Finally, the cross-modal self-attention mechanism outputs... The cross-modal attention is obtained by sequentially normalizing the first residual connection and layer normalization block Add&Norm, the multilayer perceptron (MLP), and the second residual connection and layer normalization block Add&Norm. ; The self-attention layer calculates self-attention based on cross-modal attention, first using three linear transformation matrices W. Q W K and W v The cross-modal attention is projected as a self-attention query matrix Q. S Self-attention key matrix K S and self-attention value matrix V S Then, through multi-head attention blocks, the self-attention mechanism output Att is obtained based on the matrix, and finally, the self-attention mechanism outputs... The self-attention mechanism is obtained by sequentially normalizing the first residual connection and layer normalization block Add&Norm, the multilayer perceptron (MLP), and the second residual connection and layer normalization block Add&Norm. ; The feature adjustment unit sequentially applies the self-attention function through the reshaping operation of the reshaping layer and the interpolation operation of the interpolation layer. Transform into a branch feature map .
[0011] Preferably, the adaptive fusion layer performs adaptive weight fusion on the branch feature maps of the three branches, as expressed by: W1, W2, and W3 represent adaptive weights, This is a cross-modal global feature map.
[0012] Preferably, the CNN branch uses a CBAM module, which includes a CNN input layer, a channel attention layer, a first multiplication unit, a spatial attention layer, and a second multiplication unit connected in sequence. The CBAM module performs serial-parallel spatial channel mechanism operations on the feature maps of the three modalities to obtain cross-modal local feature maps; the expression is:
[0013] Represents cross-modal local feature maps; This represents the CNN input feature map obtained by concatenating the three modality feature maps from the input layer output; CA represents the channel attention operation; SA represents the spatial attention operation. This indicates the multiplication operation.
[0014] Preferably, the cross-modal feature aggregation module includes three addition units, two multiplication units, a batch normalization layer (BN), a global branch, and a local branch. The global branch includes an average pooling layer (AvgPool), a first pointwise convolutional layer (PConv), a first batch normalization layer (BN), an activation function layer (ReLU), a second pointwise convolutional layer (PConv), and a second batch normalization layer (BN) connected in sequence. The local branch includes a first pointwise convolutional layer (PConv), a first batch normalization layer (BN), an activation function layer (ReLU), a second pointwise convolutional layer (PConv), and a second batch normalization layer (BN).
[0015] Preferably, the cross-modal feature aggregation module will aggregate cross-modal local feature maps. and cross-modal global feature maps After addition, attention is achieved in both local and global branches by changing the size of the space pool. and global attention , local attention and global attention After addition, normalization is performed to obtain the fusion weight W. The fusion weight is then combined with the cross-modal local features. Cross-modal global features The cross-modal fusion feature map is obtained by weighted multiplication and summation. The expression is:
[0016]
[0017]
[0018]
[0019] Where BN represents batch normalization; PConv represents pointwise convolution; ReLU represents activation function; and GlobalAvgPool represents average pooling.
[0020] Preferred, cross-view context cross-aware processor (C 2 P) includes four deep and shallow feature interaction enhancement modules (DS-FEM) and one cross-view attention module (CAM), which are used to extract contextual semantic information for information compensation and extract target features at different scales to obtain multi-view target features.
[0021] Preferably, the deep-shallow feature interaction enhancement module includes a convolutional layer, a channel-aware processor, a spatial-aware processor, four sets of multiplication units, a stitching unit, and an addition unit; the convolutional layer receives the feature map input from each downsampled block in the PET / CT branch as the shallow feature map F. SThe channel sensing processor receives the feature map output from each downsampled block in the PET / CT branch as a deep feature map. The channel-aware processor includes a max-pooling layer (MaxPool), an average-pooling layer (AvgPool), a multilayer perceptron (MLP), summing units, and non-linear activation function layers; the spatial-aware processor includes two sets of k×1 convolutional layers (Conv), two sets of 1×k convolutional layers (Conv), summing units, and non-linear activation function layers; the specific process is as follows: In the convolutional layer, the shallow feature map F S Convolution operation is performed to obtain coarse-grained feature maps. Then, a spatial awareness processor is used to generate spatial attention weights Ws; firstly, two sets of parallel asymmetric convolutional layers, each consisting of convolutional layers with kernel sizes of k×1 and 1×k respectively, are used to generate spatial attention weights Ws from coarse-grained feature maps. Global information is obtained from the first set of data, and then the two sets of coarse-grained feature maps are combined. The spatial attention weights Ws of the shallow features are obtained by summing the values and applying the non-linear activation function Sigmoid. For deep feature maps Spatial attention weights W are generated using a channel-aware processor. C For deep feature maps After performing parallel global max pooling and global average pooling operations, the input is fed into a shared multilayer perceptron (MLP). The outputs of the shared MLP are summed and merged element by element, and the channel attention weights W are generated using the non-linear activation function Sigmoid. C ; Integrate spatial attention weights Ws with deep feature maps Multiplication yields deep spatial feature maps , ; Adjust the channel attention weight W C With shallow feature map Multiplication yields a semantic shallow feature map , ; Integrate spatial attention weights Ws with shallow feature maps Multiplication yields shallow spatial feature maps Channel attention weight W C With deep feature maps Multiplication yields semantic deep feature maps And the shallow spatial feature map and semantic deep feature maps Fine-grained feature maps are obtained by splicing. ,
[0022] Deep spatial feature map semantic shallow feature map and fine-grained feature maps Perform weighted fusion to obtain semantic detail features ; ,in , , These are the learnable parameters.
[0023] Preferably, the cross-view attention module includes three receptive field extraction branches and a fusion layer. Each receptive field extraction branch extracts features from one receptive field to obtain multi-view output features. The fusion layer concatenates and convolves the multi-view output features to obtain multi-view target features (multi-scale lesion information is obtained through features from parallel multi-views). Each receptive field extraction branch includes an expanded convolutional kernel. The system consists of dilated convolutional layers, parallel max-pooling layers (MaxPool and AvgPool), concatenation layers (Concat), convolutional layers (Conv), a non-linear activation function (Sigmoid), and multiplication units; the fusion layer includes addition units and convolutional layers (Conv); the specific process is as follows: Each receptive field extraction branch uses a different dilated convolution kernel. semantic detail features Multi-view feature extraction was performed to obtain receptive field features of multiple different receptive fields. k represents the number of convolution kernels, and d represents the dilation rate; for example, by using an operation with a convolution kernel of 3 and a dilation rate of 1, we can obtain... ; Sensing Wild Features The receptive field pooling features were obtained using the max pooling layer MaxPool and the average pooling layer AvgPool, respectively. and average characteristics of receptive fields ; Will experience the characteristics of noike (orchid) and average characteristics of receptive fields Perform splicing to obtain spliced pooling features; After performing a 3×3 convolution operation and activation with the sigmoid non-linear activation function on the concatenated pooling features, they are then combined with the receptive field features. Multiplication yields the receptive field extension feature. This achieves uniform expansion of the receptive field among data, highlighting the lesion portion in the feature map where the receptive field is d; The receptive field extended features extracted from the receptive field branch are summed and then convolutional is performed to obtain multi-view target features.
[0024] Preferably, the decoder includes four upsampling blocks. Each upsampling block receives a cross-modal fusion feature map output by a CMformer module. The first upsampling block simultaneously receives multi-view target features output by a cross-view contextual cross-sensory processor, which is used to aggregate features from different layers, as well as cross-modal semantic information and contextual semantic information, to obtain the final segmentation result.
[0025] As can be seen from the above technical solution, compared with the prior art, this invention discloses a multi-level, multi-view, and multi-modal segmentation method for medical images. It utilizes a cross-view contextual cross-sensing processor and CMformer to fully extract cross-modal and contextual semantic information, achieving accurate segmentation of medical images. Specifically: (1) A multi-layer, multi-field, multi-modal 3MU-Net network model for medical PET / CT images is proposed to achieve segmentation. This model fully extracts cross-modal and contextual semantic information by utilizing cross-field context cross-sensing processor and CMformer module to achieve accurate segmentation of medical images.
[0026] (2) The CMformer module sets up a Transformer branch to calculate the correlation between each pixel in the PET / CT modality feature map and the corresponding pixel in the PET or CT modality, interactively learning cross-modal image information; sets up a CNN branch parallel to the Transformer branch to learn cross-modal detail information, establishes cross-modal pixel dependencies in the local and global domains, and extracts cross-modal semantic information in a more effective way.
[0027] (3) The cross-field context cross-perception processor sets up a deep and shallow layer feature interaction enhancement module DS-FEM, which uses feature maps of different resolutions of adjacent downsampling blocks in the U-Net network encoding stage to evaluate the feature differences between different levels, learn the semantic information between different resolutions, use low-level features rich in spatial information to refine high-level features, and use deep features rich in semantic information to guide low-level features, reduce the semantic information difference between adjacent level feature maps, integrate coarse-grained semantic features and fine-grained detail features, and enhance the backbone network's ability to express lesion features.
[0028] (4) Cross-field context cross-sensing processor sets up multi-field attention module CAM, obtains multi-scale lesion information through multi-field feature extraction, expands the diversity of spatial features of lesion area and models global context. Attached Figure Description
[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0030] Figure 1 This is a schematic diagram of the 3MU-Net network model structure provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the CMformer module structure provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the deep and shallow layer feature interaction enhancement module structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the cross-field attention module structure provided in an embodiment of the present invention. Detailed Implementation
[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0032] This invention discloses a multi-level, multi-view, and multi-modal segmentation method for medical images, comprising the following steps: S1: Acquire PET / CT images, PET images, and CT images, label them, and construct a training dataset; S2: Construct a 3MU-Net network model and train it using the training dataset to obtain an object segmentation model; the object segmentation model is as follows: Figure 1 The diagram includes a U-Net encoder, a cross-view contextual cross-sensing processor, a CMformer module, and a decoder. The U-Net encoder extracts multi-modal feature maps from PET / CT images. The cross-view contextual cross-sensing processor extracts cross-modal and contextual semantic information from the multi-modal feature maps to obtain multi-view target features. Four sets of CMformer modules learn the detailed information of the multi-modal feature maps to establish cross-modal pixel dependencies and obtain cross-modal fusion feature maps. The decoder decodes and segments the target based on the multi-view target features and the cross-modal fusion feature maps to obtain the segmentation results. S3: Acquire the PET / CT image to be segmented and input it into the target segmentation model to obtain the segmentation result.
[0033] Furthermore, the U-Net encoder includes a PET / CT branch, a CT branch, and a PET branch. The PET / CT branch consists of four sets of stacked downsampling blocks, which downsample the PET / CT image four times. The PET / CT feature map output from each set of downsampling blocks is input into a set of CMformer modules. The CT and PET branches each consist of three sets of stacked downsampling blocks. The CT branch downsamples the CT image three times, and the CT feature map output from each set of downsampling blocks and the final output CT feature map are input into a set of CMformer modules. The PET branch downsamples the PET image three times, and the PET feature map output from each set of downsampling blocks and the final output PET feature map are input into a set of CMformer modules. The backbone network of the U-Net encoder is the PET / CT branch, which extracts image information from the PET / CT image through four downsampling operations. The auxiliary network consists of two branches, the PET branch and the CT branch, which extract image information from the PET and CT images through three downsampling operations respectively, ultimately extracting multimodal medical image information. The input dimensions of the first downsampling block L1 are 256(H)×256(W)×64(C), the second downsampling block L2 is 128(H)×128(W)×128(C), the third downsampling block L3 is 64(H)×64(W)×256(C), and the fourth downsampling block L3 is 32(H)×32(W)×512(C). The output dimension of the fourth downsampling block L3 is 16(H)×16(W)×1024(C). The input and output feature maps of each downsampling block in the PET / CT branch are input to a corresponding deep and shallow feature interaction enhancement module in the cross-field contextual cross-sensing processor; the input feature map serves as the shallow feature map, and the output feature map serves as the deep feature map.
[0034] In one specific embodiment, the CMformer module fully considers features from each modality and leverages cross-modal information interaction to enhance the complementarity of multimodal medical images in describing lesions. In the skip connection part, it uses Transformer to calculate the correlation between corresponding pixels in the cross-modal feature maps, while simultaneously using CNN branches to learn cross-modal detail information, establishing cross-modal pixel dependencies in both local and global domains; for example... Figure 2 As shown in (a), the CMformer module includes a Transformer branch, a CNN branch, and a cross-modal feature aggregation module. The Transformer branch performs cross-modal cross-attention calculation based on the feature maps of the three modalities to generate a cross-modal global feature map. The CNN branch performs local feature enhancement based on the feature maps of the three modalities to generate a cross-modal local feature map. The cross-modal feature aggregation module fuses the cross-modal global feature map and the cross-modal local feature map to obtain a cross-modal fused feature map.
[0035] Furthermore, the main steps of the CMformer module are as follows: The Transformer branch calculates the correlation between each pixel in the PET / CT modal feature map and each corresponding pixel in other modalities through cross-modal cross-attention, establishes cross-modal global dependencies through self-attention enhancement feature representation learning, and adaptively fuses the feature maps after adjusting the feature maps through the feature adjustment unit. The CNN branch utilizes the CBAM module to enhance local feature representations. The CBAM module is an attention mechanism used to enhance the performance of convolutional neural networks. The Cross-Modal Feature Aggregation Module (CFFM) achieves attention at different scales, both global and local, by changing the size of the spatial pool, capturing cross-modal semantic information and low-level spatial details, and establishing cross-modal pixel dependencies in both local and global domains.
[0036] Furthermore, the Transformer branch excels at capturing long-range feature dependencies and utilizes the Transformer mechanism to explore cross-modal sample relationships, such as... Figure 2 As shown in (a), the Transformer branch includes a first branch, a second branch, a third branch, and an adaptive fusion layer. The inputs to the three branches are respectively a sub-branch consisting of a PET / CT feature map and a CT feature map, a PET / CT feature map and a PET feature map, and a PET / CT feature map and a CT feature map stitched together with a PET feature map. The first, second, and third branches each include a Transformer input layer, a cross-modal attention layer, a self-attention layer, and a feature adjustment unit connected in sequence. The adaptive fusion layer includes three multiplication units and one addition unit. The Transformer input layer of the first branch uses the PET / CT feature map as the query Q, and the CT feature map... The second branch's Transformer input layer uses the PET / CT feature map as the query Q, and the PET feature map is stitched together with the CT feature map as the key values K and V. The third branch's Transformer input layer uses the PET / CT feature map as the query Q, and the PET feature map as the key values K and V. The cross-modal attention layer calculates the correlation between the query Q and each corresponding pixel in key values K and V to obtain cross-modal attention. The self-attention layer calculates self-attention based on the cross-modal attention. The feature adjustment unit transforms the self-attention into branch feature maps. The adaptive fusion layer performs adaptive weight fusion on the feature maps of the three branches to obtain the cross-modal global feature map.
[0037] Furthermore, the processing procedure for each branch of the Transformer is as follows: The Transformer input layer receives PET / CT feature maps, CT feature maps, and PET feature maps, and sets the query Q, key value K, and key value V; The cross-modal attention layer first utilizes three linear transformation matrices W Q W K and W v Project the query Q, key K, and key V into a cross-modal query matrix Q, respectively. C Cross-modal bond matrix K C and the cross-modal value matrix V C Secondly, the multi-head attention block (Multi-Head Attention) is used for cross-modal query matrix Q. C and the cross-modal bond matrix K C The similarity matrix is obtained by performing a dot product operation to calculate the similarity of elements in the matrix. The similarity matrix is then transformed into a probability distribution between 0 and 1 using the softmax function to obtain a probability matrix. Finally, the probability matrix is compared with the cross-modal value matrix V. C Multiplication yields the output of a cross-modal self-attention mechanism. Finally, the cross-modal self-attention mechanism outputs... The cross-modal attention is obtained by sequentially normalizing the first residual connection and layer normalization block Add&Norm, the multilayer perceptron (MLP), and the second residual connection and layer normalization block Add&Norm. ; The self-attention layer calculates self-attention based on cross-modal attention, first using three linear transformation matrices W. Q W K and W v The cross-modal attention is projected as a self-attention query matrix Q. S Self-attention key matrix K S and self-attention value matrix V S Then, through multi-head attention blocks, the self-attention mechanism output Att is obtained based on the matrix, and finally, the self-attention mechanism outputs... The self-attention mechanism is obtained by sequentially normalizing the first residual connection and layer normalization block Add&Norm, the multilayer perceptron (MLP), and the second residual connection and layer normalization block Add&Norm. ; The feature adjustment unit sequentially applies the self-attention function through the reshaping operation of the reshaping layer and the interpolation operation of the interpolation layer. Transform into a branch feature map .
[0038] For example, the input to the first branch P1 is... and First, cross-modal cross-attention is used to calculate the correlation between each pixel in the PET / CT feature map and each corresponding pixel in the CT feature map. Three linear transformation matrices W... Q W K and W v It is used to project the input vector into three distinct vectors: the query matrix Q. C1 Key matrix K C1 Sum matrix V C1 , for Q C1 Matrix and K C1 The dot product operation is performed to calculate the similarity of elements in the matrix; the softmax function is used to transform the similarity matrix into a probability distribution between 0 and 1, and the result of the softmax operation is compared with V. C1 Matrix multiplication to obtain self-attention mechanism The output is obtained by passing through an MLP layer and normalization. Secondly, regarding Perform multi-head self-attention operation. The self-attention layer is similar to cross-modal cross-attention, consisting of three linear transformation matrices W. Q W K and W v Used to input vector Projection is Q S1 K S1 and V S1 Output through a multi-head self-attention structure The output is obtained after passing through an MLP layer and normalization. Finally, the feature adjustment unit performs reshape and interpolation operations to... Transform into The form of feature maps.
[0039] Furthermore, the adaptive fusion layer uses adaptive fusion rules to train and adaptively adjust the weights of the three branches of the Transformer to obtain the final output of the Transformer branches. The expression is: ; Where W1, W2, and W3 are fusion coefficients representing the weights of the three branches in the Transformer branch, and the sum of W1, W2, and W3 is 1, which can be obtained through model training; The softmax function is used to define W1, W2, and W3 as control parameters, and its expression is: .
[0040] Furthermore, CNN branches are used to enhance local feature representations, specifically as follows: Figure 2 As shown in branch P4 (a), the input to the CNN branch is... , It is obtained by splicing the feature maps of three modalities. CBAM (Concurrent-Parallel Spatial Channel Mechanism) is used to focus on the lesion region to obtain cross-modal local feature maps. The expression is: .
[0041] The CNN branch uses the CBAM module, which consists of a CNN input layer, a channel attention layer, a first multiplication unit, a spatial attention layer, and a second multiplication unit connected in sequence.
[0042] Furthermore, the Cross-modal feature aggregation module (CFAM) is used to fuse local and global features from cross-modal feature maps, such as... Figure 2 As shown in (e). The input to CFAM is the cross-modal local feature map of the CNN branch. Cross-modal global feature map of Transformer branch CFAM will and After addition, attention is achieved globally and locally by changing the size of the space pool. and and will and After addition and normalization, the fusion weight W is obtained, which enables the network to... and The formula for soft selection between them is as follows:
[0043]
[0044]
[0045] The cross-modal local features of the fusion weight W and the CNN branch are respectively combined. and cross-modal global features of the Transformer branch The weighted multiplication and summation yields the output. This allows for better integration of cross-modal semantics and scale-inconsistent features; .
[0046] The cross-modal feature aggregation module includes three addition units, two multiplication units, a batch normalization layer (BN), a global branch, and a local branch. The global branch consists of an average pooling layer (AvgPool), a first pointwise convolutional layer (PConv), a first batch normalization layer (BN), an activation function layer (ReLU), a second pointwise convolutional layer (PConv), and a second batch normalization layer (BN), connected in sequence. The local branch consists of a first pointwise convolutional layer (PConv), a first batch normalization layer (BN), an activation function layer (ReLU), a second pointwise convolutional layer (PConv), and a second batch normalization layer (BN).
[0047] In one specific embodiment, a cross-view context-aware processor (C... 2 P), such as Figure 1 C 2 As shown in section P, the cross-field contextual perception processor consists of two parts: a deep-shallow feature interaction enhancement module (DS-FEM) and a cross-field attention module (CAM). The cross-field contextual perception processor uses shallow semantic information to compensate for the lack of deep semantic information while using deep features to guide shallow features. The deep-shallow feature interaction enhancement module enables the network to integrate coarse-grained semantic information and fine-grained detail information. By evaluating the feature differences between different levels, it learns semantic information between different resolutions and reduces the semantic information difference between adjacent level feature maps. The cross-field attention module obtains multi-scale lesion information through multi-field feature extraction, expanding the diversity of spatial features and modeling contextual information.
[0048] C 2 P consists of four DS-FEMs and one CAM, used to extract contextual semantic information for information compensation and extract target features at different scales to obtain multi-view target features. DS-FEM uses feature maps of different resolutions output by adjacent downsampling blocks in the U-Net network encoder to define the importance of spatial and channel features of the target, evaluate the feature differences between different levels, and learn semantic information between different resolutions, thereby reducing the semantic information difference between adjacent level feature maps. This allows the network to integrate coarse-grained semantic information and fine-grained detail information. In addition, a bidirectional learning method is used to gradually enhance the backbone network's ability to express lesion features, reduce the semantic information difference between adjacent level feature maps, and integrate coarse-grained semantic features and fine-grained detail feature representations. CAM is located in the bottleneck layer and models contextual semantic information based on the deep and shallow layer feature interaction enhancement module.
[0049] Furthermore, due to the semantic information difference between feature maps of different layers in the U-Net network encoder, shallow feature maps contain rich fine-grained details such as contours, textures, and shapes, while deep feature maps contain richer coarse-grained semantic information. This invention uses DS-FEM to adjust the semantic information difference between adjacent layers of the U-Net network encoder, supplementing and integrating fine-grained detail features while extracting high-level semantic information, thereby enhancing the backbone network's ability to express lesion regions. The constructed deep-shallow feature interaction enhancement module is as follows: Figure 3 As shown, firstly, the importance of spatial and channel features is determined by feature maps of different resolutions in adjacent downsampling blocks during the U-Net network encoding stage. This is achieved by evaluating feature differences between different levels and learning semantic information between different resolutions. Secondly, the deep-shallow layer feature interaction enhancement module uses a bidirectional learning method to gradually enhance the backbone network's ability to express lesion features, reduce the semantic information difference between adjacent level feature maps, and integrate coarse-grained semantic features and fine-grained detail feature representations. The bidirectional learning method refers to using spatially rich low-level features as guidance to refine high-level features, enhancing the representation ability of detail information, and using semantically rich deep features to guide low-level features.
[0050] Furthermore, DS-FEM includes convolutional layers, channel-aware processors, spatial-aware processors, four sets of multiplication units, stitching units, and addition units; the convolutional layers receive feature maps from each downsampled block in the PET / CT branch as shallow feature maps F. S The channel sensing processor receives the feature map output from each downsampled block in the PET / CT branch as a deep feature map. The channel-aware processor includes a max-pooling layer (MaxPool), an average-pooling layer (AvgPool), a multilayer perceptron (MLP), summing units, and non-linear activation function layers; the spatial-aware processor includes two sets of k×1 convolutional layers (Conv), two sets of 1×k convolutional layers (Conv), summing units, and non-linear activation function layers. The specific processing procedure of DS-FEM is as follows: In the convolutional layer, the shallow feature map F S Perform convolution operation to obtain The spatial attention weights Ws are generated by a spatial awareness processor. The spatial awareness processor calculates the weights for lesion regions in the shallow feature map, increasing the distinction between the background and lesion regions. Global information is obtained through two parallel asymmetric convolutional layers with kernel sizes of k×1 and 1×k. Next, the spatial awareness processor sums the outputs of the parallel asymmetric convolutional layers and uses a sigmoid function to obtain the spatial attention weights Ws for the shallow features. For deep feature maps Spatial attention weights W are generated using a channel-aware processor. C The channel-aware processor is used to calculate the importance of each channel in the deep features and assign corresponding weights. After performing parallel global max pooling and global average pooling operations on the deep features, the channel-aware processor inputs the data into a shared MLP. The outputs of the shared MLP are summed element-wise and merged, and the channel attention weights W are generated using the sigmoid function. C ; Ws and deep feature map Multiply to get Guided by low-level features rich in spatial information, the boundaries between lesion areas and the background in deep feature maps are gradually refined, enhancing the ability to represent details; W C With shallow feature map Multiply to get High-level features rich in semantic information are used to guide low-level features; the expression is: ; ; Shallow feature maps contain rich detail information; Ws and shallow features Multiply to get Deep feature maps contain rich semantic information, W C with deep features Multiplying together yields the result. ; and obtained by splicing The expression is: ; Learnable parameters , , Used for fusion and get The expression is: .
[0051] Furthermore, since lesion regions in medical images typically vary in size, convolutional kernels of different scales and different dilation rates can enable the network to obtain different receptive fields, thereby extracting lesion information at different scales. Therefore, a cross-view attention module is set up to obtain multi-scale lesion information through features from parallel multi-view processing. For example... Figure 4As shown, the cross-view attention module includes three receptive field extraction branches and a fusion layer. Each receptive field extraction branch extracts features from one receptive field to obtain multi-view output features. The fusion layer concatenates and convolves the multi-view output features to obtain multi-view target features (multi-scale lesion information is obtained through features from parallel multi-views). Each receptive field extraction branch includes an expanded convolutional kernel. The system consists of dilated convolutional layers, parallel max-pooling layers (MaxPool and AvgPool), concatenation layers (Concat), convolutional layers (Conv), a non-linear activation function (Sigmoid), and multiplication units; the fusion layer includes addition units and convolutional layers (Conv); the specific process is as follows: Each receptive field extraction branch uses a different dilated convolution kernel. semantic detail features Multi-view feature extraction was performed to obtain receptive field features of multiple different receptive fields. k represents the number of convolution kernels, and d represents the dilation rate; for example, by using an operation with a convolution kernel of 3 and a dilation rate of 1, we can obtain... ; Sensing Wild Features The receptive field pooling features were obtained using the max pooling layer MaxPool and the average pooling layer AvgPool, respectively. and average characteristics of receptive fields ; Will experience the characteristics of noike (orchid) and average characteristics of receptive fields Perform splicing to obtain spliced pooling features; After performing a 3×3 convolution operation and activation with the sigmoid non-linear activation function on the concatenated pooling features, they are then combined with the receptive field features. Multiplication yields the receptive field extension feature. This achieves uniform expansion of the receptive field among data, highlighting the lesion portion in the feature map where the receptive field is d; The receptive field extended features extracted from the receptive field branch are summed and then convolutional is performed to obtain multi-view target features.
[0052] Furthermore, the decoder includes four upsampling blocks. Each upsampling block receives a cross-modal fusion feature map output by a CMformer module. The first upsampling block simultaneously receives multi-view target features output by a cross-view contextual cross-sensing processor, which is used to aggregate features from different layers, as well as cross-modal semantic information and contextual semantic information, to obtain the final segmentation result.
[0053] On the other hand, in a specific embodiment, the segmentation effect of the present invention was verified through experiments. Six evaluation metrics were selected to evaluate the performance of the model, including accuracy (Accuracy, Acc), recall, Dice similarity coefficient (Dice), volumetric overlap error (Voe), relative volume difference (Rvd), and mean intersection-over-union (Miou). The formulas are defined as shown in Table 1 below. Wherein, a correctly segmented lesion region is defined as a true positive (TP), a correctly segmented normal tissue region is defined as a true negative (TN), a normal tissue region segmented into a lesion region is defined as a false positive (FP), and a lesion region segmented into a normal region is defined as a false negative (FN). P represents the target pixel predicted by the model, and G represents the target pixel in the ground truth label.
[0054] Table 1 Evaluation Indicators
[0055] Eight other models were selected for comparison with the model of this invention. Through comparative experiments, the superiority of the 3MU-Net network model of this invention compared with different advanced segmentation networks was verified. The experimental results are shown in Table 2 below.
[0056] Table 2. Segmentation results (%) of different segmentation networks in the comparative experiment.
[0057] Experiments on clinical datasets demonstrate that this invention fully considers the different imaging principles of CT, PET, and PET / CT modalities, as well as the characteristics of medical images. The method of this invention significantly improves common problems in medical image segmentation, such as adhesions in normal tissue and irregular lesions, proving the rationality of each module. The above comparative results show that the model of this invention achieves Miou, Dice, Voe, Rvd, Acc, and Recall of 90.97%, 95.33%, 93.87%, 94.13%, 97.97%, and 94.45% for lesion segmentation, respectively. For segmentation of lesions with complex shapes, it exhibits high accuracy and relatively low redundancy.
[0058] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0059] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A multi-level, multi-view, and multi-modal segmentation method for medical images, characterized in that, Includes the following steps: Step 1: Acquire PET / CT images, PET images, and CT images, and label them to build a training dataset; Step 2: Construct a 3MU-Net network model and train it using the training dataset to obtain the target segmentation model. The target segmentation model includes a U-Net encoder, a cross-view contextual cross-sensing processor, a CMformer module, and a decoder. The U-Net encoder extracts multi-modal feature maps from PET / CT images. The cross-view contextual cross-sensing processor extracts cross-modal and contextual semantic information from the multi-modal feature maps to obtain multi-view target features. Four sets of CMformer modules are used to learn the detailed information of the multi-modal feature maps to obtain cross-modal fusion feature maps. The decoder decodes and segments the target based on the multi-view target features and the cross-modal fusion feature maps to obtain the segmentation result. Step 3: Acquire the PET / CT image to be segmented and input it into the target segmentation model to obtain the segmentation result.
2. The multi-level, multi-view, multi-modal segmentation method for medical images as described in claim 1, characterized in that, The U-Net encoder includes a PET / CT branch, a CT branch, and a PET branch. The PET / CT branch consists of four sets of stacked downsampling blocks, which downsample the PET / CT image four times. The PET / CT feature map output from each set of downsampling blocks is input to a set of CMformer modules. The CT branch and the PET branch each consist of three sets of stacked downsampling blocks. The CT branch downsamples the CT image three times, and the CT feature map output from each set of downsampling blocks and the final output CT feature map are input to a set of CMformer modules. The PET branch downsamples the PET image three times, and the PET feature map output from each set of downsampling blocks and the final output PET feature map are input to a set of CMformer modules.
3. The multi-level, multi-view, multi-modal segmentation method for medical images as described in claim 2, characterized in that, The CMformer module includes a Transformer branch, a CNN branch, and a cross-modal feature aggregation module; The Transformer branch performs cross-modal cross-attention computation based on the feature maps of the three modalities to generate a cross-modal global feature map; the CNN branch performs local feature enhancement based on the feature maps of the three modalities to generate a cross-modal local feature map; the cross-modal feature aggregation module fuses the cross-modal global feature map and the cross-modal local feature map to obtain a cross-modal fused feature map.
4. The multi-level, multi-view, multi-modal segmentation method for medical images as described in claim 3, characterized in that, The Transformer branch includes a first branch, a second branch, a third branch, and an adaptive fusion layer. The first, second, and third branches each include a Transformer input layer, a cross-modal attention layer, a self-attention layer, and a feature adjustment unit, connected sequentially. The adaptive fusion layer includes three multiplication units and one addition unit. The Transformer input layer of the first branch uses the PET / CT feature map as the query Q and the CT feature map as the key K and key V. The Transformer input layer of the second branch uses the PET / CT feature map as the query Q and the PET feature map concatenated with the CT feature map as the key K and key V. The Transformer input layer of the third branch uses the PET / CT feature map as the query Q and the PET feature map as the key K and key V. The cross-modal attention layer calculates the correlation between the query Q and each corresponding pixel in the key K and key V to obtain cross-modal attention. The self-attention layer calculates self-attention based on cross-modal attention; features The regulation unit transforms self-attention into branch feature maps; The adaptive fusion layer performs adaptive weight fusion on the feature maps of the three branches to obtain a cross-modal global feature map.
5. The multi-level, multi-view, multi-modal segmentation method for medical images as described in claim 4, characterized in that, The cross-modal attention layer and the self-attention layer each consist of a multi-head attention block, a first residual connection and layer normalization block, a multilayer perceptron, and a second residual connection and layer normalization block connected in sequence; the feature adjustment unit consists of a reshaping layer and an interpolation layer connected in sequence.
6. The multi-level, multi-view, multi-modal segmentation method for medical images as described in claim 4, characterized in that, The adaptive fusion layer performs adaptive weight fusion on the feature maps of the three branches, as expressed by: W1, W2, and W3 represent adaptive weights, respectively. , , These represent the branch feature maps output by the first branch, the second branch, and the third branch, respectively. This is a cross-modal global feature map.
7. The multi-level, multi-view, multi-modal segmentation method for medical images as described in claim 3, characterized in that, The CNN branch uses a CBAM module, which consists of a CNN input layer, a channel attention layer, a first multiplication unit, a spatial attention layer, and a second multiplication unit connected in sequence. The CBAM module performs serial-parallel spatial channel mechanism operations on the feature maps of the three modalities to obtain cross-modal local feature maps; the expression is: Represents cross-modal local feature maps; This represents the CNN input feature map obtained by concatenating the three modality feature maps from the input layer output; CA represents the channel attention operation; SA represents the spatial attention operation. This indicates the multiplication operation.
8. The multi-level, multi-view, multi-modal segmentation method for medical images as described in claim 3, characterized in that, The cross-modal feature aggregation module includes three addition units, two multiplication units, a batch normalization layer, a global branch, and a local branch. The global branch includes an average pooling layer, a first pointwise convolutional layer, a first batch normalization layer, an activation function layer, a second pointwise convolutional layer, and a second batch normalization layer connected in sequence. The local branch includes a first pointwise convolutional layer, a first batch normalization layer, an activation function layer, a second pointwise convolutional layer, and a second batch normalization layer.
9. The multi-level, multi-view, multi-modal segmentation method for medical images as described in claim 1, characterized in that, The cross-view context cross-awareness processor includes four deep and shallow feature interaction enhancement modules and one cross-view attention module, which are used to extract contextual semantic information for information compensation and extract target features at different scales to obtain multi-view target features. The deep and shallow layer feature interaction enhancement module includes convolutional layers, channel-aware processors, spatial-aware processors, four sets of multiplication units, concatenation units, and addition units. The channel-aware processor includes max pooling layers, average pooling layers, multilayer perceptrons, addition units, and non-linear activation function layers. The spatial-aware processor includes two sets of k×1 convolutional layers, two sets of 1×k convolutional layers, addition units, and non-linear activation function layers. The cross-view attention module includes three receptive field extraction branches and a fusion layer. Each receptive field extraction branch extracts features from one receptive field to obtain multi-view output features. The fusion layer concatenates and convolves the multi-view output features to obtain multi-view target features. Each receptive field extraction branch includes an expanded convolutional kernel. It consists of dilated convolutional layers, parallel max pooling and average pooling layers, splicing layers, convolutional layers, nonlinear activation functions, and multiplication units; the fusion layer includes addition units and convolutional layers.
10. The multi-level, multi-view, and multi-modal segmentation method for medical images as described in claim 1, characterized in that, The decoder includes four upsampling blocks. Each upsampling block receives a cross-modal fusion feature map output by a CMformer module. The first upsampling block simultaneously receives multi-view target features output by a cross-view context cross-awareness processor.