Pulmonary nodule MRI segmentation method based on attention-guided cross-modal fusion

Abstract

The invention discloses a pulmonary nodule MRI segmentation method based on attention-guided cross-modal fusion, belongs to the field of medical image processing, and aims to solve the problems of high missing detection rate of small nodules, insufficient multi-modal fusion and low boundary segmentation precision in existing pulmonary nodule MRI segmentation. The method comprises the following steps: preprocessing a lung multi-modal MRI image and generating a pulmonary nodule multi-modal attention feature map; constructing a segmentation network with attention jump connection and a cross-modal fusion network based on a conditional diffusion model, and forming a dual-branch joint optimization framework of segmentation and fusion decoupling; and end-to-end training is completed through a joint loss function including segmentation loss, fusion loss and consistency loss. According to the method, the pulmonary nodule segmentation precision and the model robustness are remarkably improved, and the method is suitable for accurate diagnosis scenes of clinical pulmonary nodules.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and more particularly to an MRI segmentation method for lung nodules based on attention-guided cross-modal fusion. Background Technology

[0002] With the rapid development of medical imaging technology, MRI (Magnetic Resonance Imaging) has become a core imaging tool for early screening and diagnosis of pulmonary nodules in clinical practice, thanks to its core advantages of multimodal imaging, high soft tissue contrast, and no ionizing radiation. Accurate segmentation of pulmonary nodules is a crucial prerequisite for quantitative assessment of lesions, differentiation between benign and malignant nodules, treatment planning, and follow-up on treatment efficacy, directly determining the accuracy of clinical diagnosis and the effectiveness of treatment.

[0003] Current MRI image segmentation of pulmonary nodules still faces several technical bottlenecks that urgently need to be addressed: First, pulmonary nodules, especially ground-glass nodules and micronodules, have very small signal differences from the surrounding normal lung tissue, with blurred nodule boundaries and are easily affected by respiratory motion artifacts and magnetic field inhomogeneity noise. Traditional segmentation methods based on manual features are prone to false positives and false negatives. Second, existing segmentation methods based on convolutional neural networks (such as the classic U-Net series) are limited by local receptive fields, have insufficient ability to model long-range dependencies, and are weak in feature extraction for small pulmonary nodules, making it difficult to accurately capture nodule boundary details. Third, existing multimodal MRI fusion methods mostly adopt simple channel stitching and weighted summation, which are shallow fusion strategies and cannot fully explore the complementary information of different modalities (such as anatomical information of T1-weighted imaging, signal differences of lesion tissue in T2-weighted imaging, and cell density information in diffusion-weighted imaging), resulting in poor fusion effects and incompatibility with different scanning equipment and different scanning protocols. Fourth, the application of existing attention mechanisms is mostly limited to the self-attention module within the network, without introducing prior attention guidance for the lung nodule region, which cannot effectively enhance the feature representation of the lesion region, and the interference of background noise is difficult to suppress; Fifth, existing methods mostly adopt a single-task segmentation network architecture, which does not achieve decoupling and collaborative optimization between the segmentation task and the multimodal fusion task, and cannot improve the segmentation accuracy in reverse through the denoising optimization of multimodal features, making it difficult to balance the robustness of the model with the segmentation accuracy.

[0004] To address the shortcomings of the existing technologies, this invention proposes a lung nodule MRI segmentation method based on attention-guided cross-modal fusion. By employing attention-prior-guided cross-modal feature fusion and a dual-branch joint optimization framework that decouples segmentation and fusion, this method solves the core problems of missed small nodules, insufficient multimodal fusion, ambiguous boundary segmentation, and poor generalization in the existing technologies, thereby achieving end-to-end accurate segmentation of lung nodule MRI images. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned defects of the prior art and provide a lung nodule MRI segmentation method based on attention-guided cross-modal fusion. Through the core design of cross-layer injection of lung nodule prior attention features, multimodal deep fusion based on conditional diffusion model, and end-to-end joint optimization of dual networks, this method effectively suppresses background noise, enhances the characterization of nodule regions, and fully explores the complementary information of multimodal MRI data, thereby significantly improving the accuracy, robustness and clinical applicability of lung nodule segmentation.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The MRI segmentation method for lung nodules based on attention-guided cross-modal fusion includes the following steps:

[0008] S1: Data preprocessing and multimodal attention feature map generation: The original lung multimodal MRI images are preprocessed to generate multimodal attention feature maps that match the resolution of the original images, and a standardized training dataset is constructed.

[0009] S2: Constructing an end-to-end joint optimization dual-branch network architecture: Constructing a joint optimization framework including a segmentation network and a fusion network; the segmentation network adopts an encoder-decoder U-shaped architecture with attention skip connections, taking the preprocessed MRI image and multimodal attention feature map as input, and outputting an initial segmentation probability map; the fusion network adopts an encoder-decoder U-shaped architecture based on a conditional diffusion probability model, taking the preprocessed MRI image and diffusion time step embedding as input, and outputting a denoised multimodal fused image;

[0010] S3: Construct an attention-guided cross-modal fusion mechanism: The multimodal attention feature map is used as cross-modal prior information and injected into the corresponding decoder layer of the segmentation network through skip connections to enhance the feature representation of the lung nodule target region; at the same time, the multimodal fused image output by the fusion network is input into the segmentation network to obtain the optimized secondary segmentation probability map;

[0011] S4: Constructing the joint loss function and training the network: Construct a total loss function that includes segmentation loss, fusion loss and consistency loss, and use the standardized training dataset to perform end-to-end joint training on the segmentation network and the fusion network, and iteratively update the network parameters through the backpropagation algorithm;

[0012] S5: Model Inference and Post-processing: Input the multimodal MRI images of the lungs to be tested into the trained segmentation network and fusion network, perform weighted fusion of the segmentation results output by the two networks, and then obtain the final lung nodule segmentation results through morphological operations and connected component analysis.

[0013] Furthermore, step S1 specifically includes the following steps:

[0014] 2.1 Image Normalization: The pixel values ​​of the original multimodal MRI images are linearly mapped to the [0,1] interval. The normalization formula is as follows:

[0015] ,

[0016] in, The original MRI image. This represents the minimum number of pixels in the original image. The maximum value of the original image pixels. These are normalized MRI images;

[0017] 2.2 Image cropping: The normalized MRI image is cropped using the sliding window method to generate image blocks of a fixed size. The window size and sliding step are preset according to the image resolution.

[0018] 2.3 Multimodal attention feature map generation: The preprocessed MRI image is input into a pre-trained attention detection network to generate an attention probability map with the same resolution as the original image. The value of each pixel in the attention probability map is the probability that the pixel belongs to the lung nodule region.

[0019] 2.4 Attention Feature Map Optimization: The generated attention probability map is binarized, and features with probability values ​​greater than a preset threshold are selected. The pixels in the binarized image are assigned a value of 1, and the rest of the pixels are assigned a value of 0. Then, the binarized image is subjected to Gaussian blurring to enhance the connectivity and smoothness of the nodule region, and the final multimodal attention feature map is obtained.

[0020] 2.5 Dataset partitioning: The preprocessed MRI images and their corresponding multimodal attention feature maps are divided into training and validation sets according to a preset ratio to construct a standardized training dataset.

[0021] Furthermore, the segmentation network in step S2 is a 4-layer U-shaped network with attention skip connections, and its construction process specifically includes the following steps:

[0022] 3.1 Encoder Path Construction: It consists of four cascaded downsampling modules stacked together. Each downsampling module contains two convolutional layers with a kernel size of 3×3, a stride of 1, and the same padding method, a ReLU activation function layer, and a max pooling layer with a stride of 2 and a kernel size of 2×2. The downsampling modules progressively extract multi-scale high-dimensional semantic features from the input image while reducing the spatial resolution of the feature maps. The unpooled feature map output by each downsampling module is used as the encoder side feature of the skip connections.

[0023] 3.2 Decoder Path Construction: It consists of four cascaded upsampling modules stacked together. Each upsampling module contains a transposed convolutional layer with a kernel size of 2×2 and a stride of 2, two convolutional layers with a kernel size of 3×3, a stride of 1, and the same padding method, and a ReLU activation function layer. The spatial resolution of the feature map is gradually restored through the upsampling modules, and finally restored to the same size as the input image.

[0024] 3.3 Attention Skip Connection Construction: In each upsampling module, the features of the current layer decoder, the features output by the corresponding layer of the encoder, and the multimodal attention feature map downsampled to the current layer resolution are weighted and fused. The fusion formula is as follows:

[0025] ,

[0026] in, For the enhanced features after fusion, These are the original features of the current layer decoder. A multimodal attention feature map that matches the current layer resolution; These are the weight coefficients for the attention features; the fused enhanced features are then input into subsequent convolutional layers for feature extraction.

[0027] 3.4 Output Layer Construction: The output of the last upsampling module is sequentially connected to a convolutional layer with a kernel size of 1×1 and a sigmoid activation function layer, outputting an initial segmentation probability map with the same resolution as the input image.

[0028] Furthermore, the fusion network in step S2 is a U-shaped network based on the conditional diffusion probability model, and its construction process specifically includes the following steps:

[0029] 4.1 Define the forward diffusion process: Construct a fixed-length Markov chain and progressively diffuse it towards the original MRI image. Add Gaussian noise to generate a series of latent variables until It conforms to a standard normal distribution; each step of the noise addition process is represented as:

[0030] ,

[0031] in, For diffusion time step, The total number of diffusion steps, These are the preset noise scheduling parameters. Add intensity to the noise at each step. It is the identity matrix. It follows a normal distribution;

[0032] 4.2 Define the inverse denoising process: Construct a learnable noise prediction network It is used to predict the Gaussian noise added at each step in the forward diffusion process, from latent variables that conform to a standard normal distribution. Initially, a clear multimodal fused image is obtained by gradually restoring it; each step of the reverse denoising process is represented as:

[0033] ,

[0034] in, The predicted mean of the normal distribution. The variance is a preset normal distribution; the formula for calculating the mean is:

[0035] ,

[0036] in, The predicted noise output by the noise prediction network. For cumulative noise scheduling parameters, From step 1 to step 2 The result of the cumulative product of the steps;

[0037] 4.3 Construction of the noise prediction network architecture: The noise prediction network... An encoder-decoder U-shaped architecture is adopted, and a diffusion time step is introduced in each residual block. The embedded vectors enable the network to perceive the current denoising stage; the encoder extracts multi-scale features through layer-by-layer downsampling, and the decoder restores the feature map resolution through layer-by-layer upsampling. Multi-scale feature fusion is achieved between corresponding layers of the encoder and decoder through skip connections. The network's final output is the same as its input. A consistent resolution prediction noise map is generated, and a denoised multimodal fused image is reconstructed based on the prediction noise map. .

[0038] Furthermore, step S4 specifically includes the following steps:

[0039] 5.1 Calculate the segmentation loss Using the binary classification cross-entropy loss function With Dice loss function The weighted fusion method is used to modify the initial segmentation probability map output by the segmentation network. For monitoring, the segmentation loss formula is:

[0040] ,

[0041] Where G is the true binary label for lung nodule segmentation. Used to constrain the classification error between the segmentation probability map and the true label. This is used to alleviate the sample imbalance problem between the foreground and background regions of lung nodules. This is the loss balance coefficient;

[0042] 5.2 Calculate the fusion loss The mean square error function is used to constrain the predicted noise output of the noise prediction network. Compared to the Gaussian noise actually added during the forward diffusion process Consistent, the fusion loss formula is:

[0043] ,

[0044] in, For mathematical expectation, To follow a standard normal distribution True Gaussian noise, It is an L2 norm;

[0045] 5.3 Calculate the consistency loss : The multimodal fused image output by the fusion network Inputting the segmentation network yields a quadratic segmentation probability map. ,calculate Compared with the initial segmentation probability map The mean squared error between the two networks constrains the semantic consistency of the outputs. The consistency loss formula is as follows:

[0046] ;

[0047] 5.4 Calculating the Total Loss Function and Network Parameter Updates: Combining the above losses, the total loss function is constructed as follows:

[0048] ,

[0049] in, , The loss balance coefficient is used; the Adam optimizer is employed to perform backpropagation to calculate the gradient based on the total loss function, and to jointly update all learnable parameters of the segmentation network and the fusion network.

[0050] 5.5 Set training termination conditions: After each training round, evaluate the model performance on the validation set and record the Dice coefficient of the validation set; if the Dice coefficient of the validation set does not improve for a preset number of consecutive rounds, terminate the training early; if the preset maximum number of training rounds has not been reached, continue iterative training.

[0051] Furthermore, step S5 specifically includes the following steps:

[0052] 6.1 Segmentation Network Inference: After preprocessing the lung multimodal MRI images to be tested, they are input into the trained segmentation network to obtain the initial test segmentation probability map. ;

[0053] 6.2 Fusion Network Inference: The preprocessed multimodal MRI images of the lungs to be tested are input into the trained fusion network, and a complete inverse denoising process is performed to generate denoised test fused images. Then Input the trained segmentation network to obtain the optimized test segmentation probability map. ;

[0054] 6.3 Weighted Fusion of Segmentation Results: The initial test segmentation probability map and the optimized test segmentation probability map are weighted and fused to obtain the final segmentation probability map. The fusion formula is as follows:

[0055] ,

[0056] in, For the final segmentation probability map, The preset fusion weights;

[0057] 6.4 Post-processing: The final segmentation probability map is binarized, and then isolated noise points are removed, small holes are filled, and segmentation boundaries are smoothed through morphological operations such as dilation and erosion, connected component analysis, and Gaussian smoothing to obtain the final accurate lung nodule segmentation result.

[0058] 6.5 Output Results: The final lung nodule segmentation results are overlaid with the original MRI image to generate a visualization result, and the segmentation result file is saved according to the preset format.

[0059] Furthermore, the lung multimodal MRI images include T1-weighted imaging, T2-weighted imaging, and diffusion-weighted imaging.

[0060] Beneficial effects: Compared with the prior art, the present invention has the following significant beneficial effects:

[0061] 1. This invention constructs an attention-guided cross-modal fusion mechanism, which injects the prior attention feature map of lung nodules into the segmentation network decoder layer by layer through skip connections. During the feature decoding process, the feature weight of the nodule region is continuously strengthened, and the interference of background noise is suppressed. This effectively solves the problem of high false negative rate of small lung nodules and nodules with blurred boundaries in the prior art, and significantly improves the feature representation ability of lesion region.

[0062] 2. This invention designs a dual-branch end-to-end joint optimization framework that decouples segmentation and fusion. Based on the conditional diffusion probability model, a fusion network is constructed to fully explore the complementary information of multimodal MRI data and generate a high-quality multimodal fusion image after denoising. At the same time, the semantic consistency of the two networks is constrained by the consistency loss, realizing the collaborative optimization of the fusion task and the segmentation task. This not only improves the fusion depth of multimodal data, but also enhances the feature extraction capability of the segmentation network, thus significantly improving the segmentation accuracy and model robustness.

[0063] 3. This invention employs a segmentation loss function that combines binary cross-entropy loss and Dice loss, effectively solving the problem of extreme imbalance between foreground and background samples in lung nodule segmentation tasks. It also balances pixel-level classification accuracy and region overlap, significantly improving the segmentation accuracy of nodule boundaries. The design of the joint loss function enables end-to-end joint training of the two networks, avoiding error accumulation caused by step-by-step training and simplifying the model training process.

[0064] 4. The entire process of this invention adopts an end-to-end processing architecture, which eliminates the need for manual design of features and human intervention. It has a strong generalization ability for multimodal MRI data with different scanning devices and different scanning protocols. The segmentation results are stable and reliable, and can provide accurate quantitative basis for the early diagnosis, benign and malignant differentiation and treatment planning of clinical pulmonary nodules. It has extremely high clinical application value. Attached Figure Description

[0065] Figure 1 is an overall flowchart of the method for precise segmentation of pulmonary nodule MRI images based on attention-guided cross-modal fusion as described in this invention;

[0066] Figure 2 is a schematic diagram of the end-to-end joint optimization dual-branch network architecture described in this invention;

[0067] Figure 3 is a schematic diagram of the segmentation network structure with attention skipping connections described in this invention;

[0068] Figure 4 is a schematic diagram of the fusion network structure based on the conditional diffusion probability model described in this invention.

[0069] In the accompanying drawings, the same reference numerals refer to the same or similar components or steps. The flowcharts and block diagrams are only used to illustrate the technical logic of the present invention and are not used to limit the scope of protection of the present invention. Detailed Implementation

[0070] The present invention will be further described in detail below with reference to specific embodiments and accompanying drawings. This embodiment is implemented based on the technical solution of the present invention, and provides detailed implementation methods and specific operating procedures; however, the scope of protection of the present invention is not limited to the following embodiments.

[0071] The lung multimodal MRI images used in this embodiment are clinically desensitized data, including three modalities: T1-weighted imaging, T2-weighted imaging, and diffusion-weighted imaging.

[0072] The specific implementation steps of the attention-guided cross-modal fusion-based precise segmentation method for lung nodule MRI images described in this embodiment are as follows:

[0073] Step 1: Data Preprocessing and Multimodal Attention Feature Map Generation

[0074] The raw multimodal MRI images of the lungs were standardized and preprocessed to generate prior attention feature maps of lung nodules. A standardized dataset that can be directly used for network training was then constructed. The specific steps are as follows:

[0075] 1.1 Image normalization: The minimum-maximum normalization method is used to linearly map the pixel values ​​of the original multimodal MRI images to the [0,1] interval, eliminating the differences in pixel value distribution caused by different scanning devices and different scanning sequences. The normalization formula is consistent with the formula in the aforementioned invention.

[0076] 1.2 Image Cropping: The normalized 3D MRI image is cropped without overlap using a sliding window method. The preferred sliding window size is 256×256×64, and the preferred sliding step size is 128. The selectable window size range is 128×128×32~512×512×128, and the step size range is 64~256. This parameter setting ensures that the cropped image block completely covers the entire lung region, while matching the network input size requirements and avoiding memory overflow.

[0077] 1.3 Multimodal attention feature map generation: The preprocessed multimodal MRI images are input into a pre-trained ResNet-50 attention detection network to generate an attention probability map with the same resolution as the original image. The value range of each pixel is [0,1], representing the probability that the pixel belongs to the lung nodule region. The ResNet-50 attention detection network is pre-trained on a large-scale lung nodule MRI dataset and can accurately identify candidate regions of lung nodules.

[0078] 1.4 Attention Feature Map Optimization: The generated attention probability map is binarized, and the binarization threshold is set. The preferred value is 0.5, and the selectable range is 0.3~0.7. Pixels with a probability value greater than the threshold are assigned a value of 1 (foreground of nodules), and the remaining pixels are assigned a value of 0 (background). Then, the binarized image is subjected to Gaussian blur processing. The preferred Gaussian blur kernel size is 5×5, the standard deviation is 1, and the selectable kernel size range is 3×3~7×7. Gaussian blur enhances the connectivity of the nodule region and eliminates isolated noise points to obtain the final multimodal attention feature map.

[0079] 1.5 Dataset Splitting: The preprocessed multimodal MRI images, along with the corresponding multimodal attention feature maps and the true binary labels of lung nodule segmentation, were randomly divided into training and validation sets in a 7:3 ratio. The selectable split ratio range is 6:4 to 8:2. All data were saved in .nii format to construct a standardized training dataset.

[0080] Step 2: Construct an end-to-end jointly optimized dual-branch network architecture

[0081] An end-to-end joint optimization framework comprising a segmentation network and a fusion network is constructed. The two networks are designed and trained in parallel. The specific construction process is as follows:

[0082] 2.1 Segmentation Network Construction

[0083] The segmentation network employs a 4-layer U-shaped network with attention skip connections. The input consists of preprocessed multimodal MRI images and multimodal attention feature maps, and the output is an initial segmentation probability map with the same resolution as the input. The specific structure is as follows:

[0084] 2.1.1 Encoder Path Construction: It consists of four cascaded downsampling modules stacked together. Each downsampling module contains two convolutional layers with a kernel size of 3×3, a stride of 1, and the same padding method, a ReLU activation function layer, and a max pooling layer with a stride of 2 and a kernel size of 2×2. The number of output channels of the four downsampling modules are 64, 128, 256, and 512, respectively. The multi-scale high-dimensional semantic features of the input image are extracted step by step through the downsampling modules, while the spatial resolution of the feature map is reduced by 1 / 2 step by step. The unpooled feature map output by each downsampling module is used as the encoder side feature of the skip connection.

[0085] 2.1.2 Decoder Path Construction: It consists of four cascaded upsampling modules stacked together. Each upsampling module contains a transposed convolutional layer with a kernel size of 2×2 and a stride of 2, two convolutional layers with a kernel size of 3×3, a stride of 1, and the same padding method, and a ReLU activation function layer. The number of output channels of the four upsampling modules are 256, 128, 64, and 32, respectively. The spatial resolution of the feature map is gradually increased by a factor of 2 through the transposed convolutional layers, and finally restored to the same size as the input image.

[0086] 2.1.3 Attention Skip Connection Construction: In each upsampling module, the current layer decoder features, the encoder corresponding layer output features, and the multimodal attention feature map downsampled to the current layer resolution are weighted and fused. The fusion formula is consistent with the formula in the aforementioned invention. The weight coefficient λ of the attention features is preferably 0.5, and can be selected from 0.3 to 0.7. This value can effectively balance the weights of the original image features and the nodule prior attention features, avoid the attention features from overly covering the details of the original image, and significantly enhance the feature representation of the nodule region. The fused enhanced features are input into subsequent convolutional layers for further feature extraction.

[0087] 2.1.4 Output Layer Construction: The output of the last upsampling module is sequentially connected to a convolutional layer with a kernel size of 1×1 and an output channel of 1, and a sigmoid activation function layer to map the output values ​​to the [0,1] interval, outputting an initial segmentation probability map with the same resolution as the input image. Each pixel value represents the probability that the pixel belongs to the lung nodule region.

[0088] 2.2 Construction of Converged Network

[0089] The fusion network employs a U-shaped network based on a conditional diffusion probability model, with a learnable noise prediction network at its core. The input consists of preprocessed multimodal MRI images and diffusion time-step embeddings, while the output is a denoised multimodal fused image. The specific construction process is as follows:

[0090] 2.2.1 Defining the forward diffusion process: Constructing a fixed-length Markov chain, progressively diffusing the original MRI image... Add Gaussian noise to generate a series of latent variables until It conforms to a standard normal distribution; total number of diffusion steps The preferred value is 1000, with an optional range of 500~2000; the formula for each step of the noise addition process is consistent with the formula in the aforementioned invention; wherein, the noise addition intensity... A linearly increasing strategy is adopted, with values ​​ranging from [0.0001, 0.02], corresponding to the noise scheduling parameters. Cumulative noise scheduling parameters for From step 1 to step 2 The result of the cumulative product of the steps;

[0091] 2.2.2 Define the inverse denoising process: Construct a learnable noise prediction network It is used to predict the Gaussian noise added at each step in the forward diffusion process, from latent variables that conform to a standard normal distribution. Begin, through A clear multimodal fused image is obtained by iterative restoration step by step; the formula for the reverse denoising process and the formula for calculating the mean are consistent with the formulas in the aforementioned invention.

[0092] 2.2.3 Noise Prediction Network Architecture Construction: The noise prediction network ϵθ adopts a 4-layer encoder-decoder U-shaped architecture symmetrical to the segmentation network. The core component is a residual block with time-step embedding. The encoder extracts multi-scale features by downsampling stepwise through 4 downsampling modules. Each downsampling module contains two 3×3 convolutional layers, a Batch Normalization layer, a ReLU activation function layer, and a 2×2 max pooling layer, with output channels of 64, 128, 256, and 512 respectively. The decoder restores the feature map resolution by upsampling stepwise through 4 upsampling modules. Each upsampling module contains a 2×2 transposed convolutional layer, two 3×3 convolutional layers, a Batch Normalization layer, and a ReLU activation function layer, with output channels of 256, 128, 64, and 32 respectively. Multi-scale feature fusion is performed between corresponding layers of the encoder and decoder through skip connections.

[0093] An embedding vector for each diffusion time step t is introduced into each residual block. Sine positional encoding maps the discrete time step t into a high-dimensional feature vector. After feature transformation through two linear layers, this vector is injected into the convolutional feature map of the residual block. This allows the network to perceive the current denoising stage and adapt to noise prediction requirements with different diffusion steps. The network ultimately outputs a predicted noise map with the same resolution as the input xt. The denoised multimodal fusion image is then reconstructed using the mean calculation formula for inverse denoising. .

[0094] Step 3: Construct an attention-guided cross-modal fusion mechanism

[0095] A cross-modal fusion mechanism with dual paths is constructed to achieve deep fusion of multimodal prior information and dual networks, as detailed below:

[0096] The first approach is to use the multimodal attention feature map as prior information for the lung nodule region. Through downsampling operations synchronized with the segmentation network encoder, the resolution is gradually reduced. Then, it is injected into the decoder of the corresponding layer of the segmentation network through skip connections. It is then weighted and fused with the decoder features and encoder features. Throughout the feature decoding process, the feature weights of the nodule region are continuously strengthened, and background noise is suppressed.

[0097] The second approach: merging the denoised multimodal image output by the fusion network. The data is then fed back into the segmentation network during training to obtain the optimized secondary segmentation probability map. By integrating the multimodal denoising optimization of the network, a higher quality input image is provided for the segmentation network, which in turn improves the feature extraction capability of the segmentation network.

[0098] Step 4: Construct the joint loss function and train the network.

[0099] A multi-task joint loss function is constructed, and the segmentation network and fusion network are jointly trained end-to-end. The specific steps are as follows:

[0100] 4.1 Calculate the segmentation loss Using the binary classification cross-entropy loss function With Dice loss function The weighted fusion method is used to modify the initial segmentation probability map output by the segmentation network. The supervision is carried out using the same formula as described in the aforementioned invention. The loss balance coefficient λ1 is preferably 0.5, and can be selected from 0.2 to 0.8. Among them, the binary cross-entropy loss constrains the classification accuracy of each pixel, and the Dice loss focuses on the segmentation overlap of the nodule foreground region. The combination of the two can effectively solve the problem of imbalance between foreground and background samples in lung nodule segmentation, while taking into account the boundary segmentation accuracy.

[0101] 4.2 Calculate the fusion loss The mean square error function is used to constrain the consistency between the predicted noise output by the noise prediction network and the Gaussian noise actually added during the forward diffusion process. The formula is consistent with the formula in the aforementioned invention. This loss is the core training objective of the diffusion model, ensuring that the network can accurately predict noise and achieve high-quality multimodal image denoising and fusion.

[0102] 4.3 Calculate the consistency loss Calculate the probability graph of quadratic segmentation. Compared with the initial segmentation probability map The mean square error between the two networks constrains the semantic consistency of the outputs of the two networks, ensuring that the image generated by the fusion network can retain complete lung nodule structural information. The formula is consistent with the formula in the aforementioned invention.

[0103] 4.4 Calculation of Total Loss Function and Network Parameter Update: Combining the above three types of losses, a total loss function is constructed, with the formula consistent with the formula in the aforementioned invention content; the loss balance coefficient is then integrated. The preferred value is 0.3, with an optional range of 0.1 to 0.5; consistency loss balance coefficient The preferred value is 0.1, and the selectable range is 0.05~0.2;

[0104] The Adam optimizer is used for parameter optimization. The initial learning rate is preferably 0.001, with a selectable range of 0.0001~0.01. The weight decay factor is preferably 1, and the training batch size is set to 8. Backpropagation is performed based on the total loss function to calculate the gradient and jointly update all learnable parameters of the segmentation network and the fusion network.

[0105] 4.5 Set training termination conditions: After each training round, evaluate the model's segmentation performance on the validation set. The core evaluation metric is the Dice coefficient. If the Dice coefficient on the validation set does not improve for 10 consecutive rounds, trigger the early stopping mechanism to terminate training early. The range of 5 to 20 consecutive rounds without improvement can be selected to avoid model overfitting. The maximum number of training rounds is set to 500. If the early stopping mechanism is not triggered, training will terminate after reaching the maximum number of training rounds, and the model weights will be saved.

[0106] Step 5: Model Inference and Post-processing

[0107] The trained model is used to perform inference segmentation on the multimodal MRI images of the lungs under test. The specific steps are as follows:

[0108] 5.1 Image preprocessing: Perform normalization and cropping preprocessing operations on the lung multimodal MRI images to be tested, consistent with the training set;

[0109] 5.2 Segmentation Network Inference: The preprocessed test image is input into the trained segmentation network, and forward propagation is performed to obtain the initial test segmentation probability map. ;

[0110] 5.3 Fusion Network Inference: The preprocessed test image is input into the trained fusion network, and a complete 1000-step reverse denoising process is performed to generate a denoised test fused image. Then Input the trained segmentation network to obtain the optimized test segmentation probability map. ;

[0111] 5.4 Weighted Fusion of Segmentation Results: The initial test segmentation probability map and the optimized test segmentation probability map are weighted and fused to obtain the final segmentation probability map. The formula is consistent with the formula in the aforementioned invention content; fusion weight The preferred value is 0.7, and the optional range is 0.5~0.9;

[0112] 5.5 Post-processing: The final segmentation probability map is binarized. The binarization threshold is preferably 0.5, with an optional range of 0.3~0.7. Pixels larger than the threshold are assigned a value of 1, and the rest are assigned a value of 0, resulting in a binary segmentation result. Morphological operations are then performed on the binary segmentation result. A 5×5 convolution kernel is used for dilation, and a 3×3 convolution kernel is used for erosion. The kernel size can be selected from 3×3 to 7×7 to fill in small holes in the segmented region and remove isolated noise points. Subsequently, connected component analysis is performed, retaining connected components with an area greater than 500 pixels. The optional area threshold range is 200~1000 pixels, and areas smaller than the area threshold are set as background. Finally, a 3×3 Gaussian blur kernel is used to smooth the segmentation boundary, reduce the jagged effect, optimize the segmentation contour, and obtain the final accurate lung nodule segmentation result.

[0113] 5.6 Output Results: The final lung nodule segmentation results are overlaid with the original MRI image to generate a visualized segmentation result image. The segmentation results are saved as a PNG format visualization file and a .nii format three-dimensional data file for clinical diagnostic analysis.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lung nodule MRI segmentation method based on attention-guided cross-modal fusion, characterized in that, Includes the following steps: S1: Data preprocessing and multimodal attention feature map generation: The original lung multimodal MRI images are preprocessed to generate multimodal attention feature maps that match the resolution of the original images, and a standardized training dataset is constructed. S2: Constructing an end-to-end joint optimization dual-branch network architecture: Constructing a joint optimization framework including a segmentation network and a fusion network; the segmentation network adopts an encoder-decoder U-shaped architecture with attention skip connections, taking the preprocessed MRI image and multimodal attention feature map as input, and outputting an initial segmentation probability map; the fusion network adopts an encoder-decoder U-shaped architecture based on a conditional diffusion probability model, taking the preprocessed MRI image and diffusion time step embedding as input, and outputting a denoised multimodal fused image; S3: Construct an attention-guided cross-modal fusion mechanism: The multimodal attention feature map is used as cross-modal prior information and injected into the corresponding decoder layer of the segmentation network through skip connections to enhance the feature representation of the lung nodule target region; at the same time, the multimodal fused image output by the fusion network is input into the segmentation network to obtain the optimized secondary segmentation probability map; S4: Constructing the joint loss function and training the network: Construct a total loss function that includes segmentation loss, fusion loss and consistency loss, and use the standardized training dataset to perform end-to-end joint training on the segmentation network and the fusion network, and iteratively update the network parameters through the backpropagation algorithm; S5: Model Inference and Post-processing: Input the multimodal MRI images of the lungs to be tested into the trained segmentation network and fusion network, perform weighted fusion of the segmentation results output by the two networks, and then obtain the final lung nodule segmentation results through morphological operations and connected component analysis.

2. The method according to claim 1, characterized in that, S1 specifically includes the following steps: 2.1 Image Normalization: The pixel values ​​of the original multimodal MRI images are linearly mapped to the [0,1] interval. The normalization formula is as follows: ， in, The original MRI image. This represents the minimum number of pixels in the original image. The maximum value of the original image pixels. These are normalized MRI images; 2.2 Image cropping: The normalized MRI image is cropped using the sliding window method to generate image blocks of a fixed size. The window size and sliding step are preset according to the image resolution. 2.3 Multimodal attention feature map generation: The preprocessed MRI image is input into a pre-trained attention detection network to generate an attention probability map with the same resolution as the original image. The value of each pixel in the attention probability map is the probability that the pixel belongs to the lung nodule region. 2.4 Attention Feature Map Optimization: The generated attention probability map is binarized, and probability values ​​greater than a preset threshold are selected. The pixels in the binarized image are assigned a value of 1, and the rest of the pixels are assigned a value of 0. Then, the binarized image is subjected to Gaussian blurring to enhance the connectivity and smoothness of the nodule region, and the final multimodal attention feature map is obtained. 2.5 Dataset partitioning: The preprocessed MRI images and their corresponding multimodal attention feature maps are divided into training and validation sets according to a preset ratio to construct a standardized training dataset.

3. The method according to claim 1, characterized in that, The segmentation network in S2 is a 4-layer U-shaped network with attention skip connections, and its construction process specifically includes the following steps: 3.1 Encoder Path Construction: It consists of four cascaded downsampling modules stacked together. Each downsampling module contains two convolutional layers with a kernel size of 3×3, a stride of 1, and the same padding method, a ReLU activation function layer, and a max pooling layer with a stride of 2 and a kernel size of 2×2. The downsampling modules progressively extract multi-scale high-dimensional semantic features from the input image while reducing the spatial resolution of the feature maps. The unpooled feature map output by each downsampling module is used as the encoder side feature of the skip connections. 3.2 Decoder Path Construction: It consists of four cascaded upsampling modules stacked together. Each upsampling module contains a transposed convolutional layer with a kernel size of 2×2 and a stride of 2, two convolutional layers with a kernel size of 3×3, a stride of 1, and the same padding method, and a ReLU activation function layer. The spatial resolution of the feature map is gradually restored through the upsampling modules, and finally restored to the same size as the input image. 3.3 Attention Skip Connection Construction: In each upsampling module, the features of the current layer decoder, the features output by the corresponding layer of the encoder, and the multimodal attention feature map downsampled to the current layer resolution are weighted and fused. The fusion formula is as follows: ， in, For the enhanced features after fusion, These are the original features of the current layer decoder. A multimodal attention feature map that matches the current layer resolution; These are the weight coefficients for the attention features; the fused enhanced features are then input into subsequent convolutional layers for feature extraction. 3.4 Output Layer Construction: The output of the last upsampling module is sequentially connected to a convolutional layer with a kernel size of 1×1 and a sigmoid activation function layer, outputting an initial segmentation probability map with the same resolution as the input image. .

4. The method according to claim 1, characterized in that, The fusion network in S2 is a U-shaped network based on the conditional diffusion probability model, and its construction process specifically includes the following steps: 4.1 Define the forward diffusion process: Construct a fixed-length Markov chain and progressively diffuse it towards the original MRI image. Add Gaussian noise to generate a series of latent variables until It conforms to a standard normal distribution; each step of the noise addition process is represented as: , in, For diffusion time step, The total number of diffusion steps, These are the preset noise scheduling parameters. Add intensity to the noise at each step. It is the identity matrix. It follows a normal distribution; 4.2 Define the inverse denoising process: Construct a learnable noise prediction network It is used to predict the Gaussian noise added at each step in the forward diffusion process, from latent variables that conform to a standard normal distribution. Initially, a clear multimodal fused image is obtained by gradually restoring it; each step of the reverse denoising process is represented as: , in, The predicted mean of the normal distribution. The variance is a preset normal distribution; the formula for calculating the mean is: , in, The predicted noise output by the noise prediction network. For cumulative noise scheduling parameters, From step 1 to step 2 The result of the cumulative product of the steps; 4.3 Construction of the noise prediction network architecture: The noise prediction network... An encoder-decoder U-shaped architecture is adopted, and a diffusion time step is introduced in each residual block. The embedded vectors enable the network to perceive the current denoising stage; the encoder extracts multi-scale features through layer-by-layer downsampling, and the decoder restores the feature map resolution through layer-by-layer upsampling. Multi-scale feature fusion is achieved between corresponding layers of the encoder and decoder through skip connections. The network's final output is the same as its input. A consistent resolution prediction noise map is generated, and a denoised multimodal fused image is reconstructed based on the prediction noise map. .

5. The method according to claim 1, characterized in that, S4 specifically includes the following steps: 5.1 Calculate the segmentation loss Using the binary classification cross-entropy loss function With Dice loss function The weighted fusion method is used to modify the initial segmentation probability map output by the segmentation network. For monitoring, the segmentation loss formula is: , Where G is the true binary label for lung nodule segmentation. Used to constrain the classification error between the segmentation probability map and the true label. This is used to alleviate the sample imbalance problem between the foreground and background regions of lung nodules. This is the loss balance coefficient; 5.2 Calculate the fusion loss The mean square error function is used to constrain the predicted noise output of the noise prediction network. Compared to the Gaussian noise actually added during the forward diffusion process Consistent, the fusion loss formula is: ， in, For mathematical expectation, To follow a standard normal distribution True Gaussian noise, It is an L2 norm; 5.3 Calculate the consistency loss : The multimodal fused image output by the fusion network Inputting the segmentation network yields a quadratic segmentation probability map. ,calculate Compared with the initial segmentation probability map The mean squared error between the two networks constrains the semantic consistency of the outputs. The consistency loss formula is as follows: ； 5.4 Calculating the Total Loss Function and Network Parameter Updates: Combining the above losses, the total loss function is constructed as follows: ， in, , The loss balance coefficient is used; the Adam optimizer is employed to perform backpropagation to calculate the gradient based on the total loss function, and to jointly update all learnable parameters of the segmentation network and the fusion network. 5.5 Set training termination conditions: After each training round, evaluate the model performance on the validation set and record the Dice coefficient of the validation set; if the Dice coefficient of the validation set does not improve for a preset number of rounds, terminate the training early; if the preset maximum number of training rounds has not been reached, continue iterative training.

6. The method according to claim 1, characterized in that, The S5 specifically Includes the following steps: 6.1 Segmentation Network Inference: After preprocessing the multimodal MRI images of the lungs to be tested, the images are input into the trained segmentation network to obtain the initial test segmentation probability map. ; 6.2 Fusion Network Inference: The preprocessed multimodal MRI images of the lungs to be tested are input into the trained fusion network, and a complete inverse denoising process is performed to generate denoised test fused images. Then Input the trained segmentation network to obtain the optimized test segmentation probability map. ; 6.3 Weighted Fusion of Segmentation Results: The initial test segmentation probability map and the optimized test segmentation probability map are weighted and fused to obtain the final segmentation probability map. The fusion formula is as follows: ， in, For the final segmentation probability map, The preset fusion weights; 6.4 Post-processing: The final segmentation probability map is binarized, and then isolated noise points are removed, small holes are filled, and segmentation boundaries are smoothed through morphological operations such as dilation and erosion, connected component analysis, and Gaussian smoothing to obtain the final accurate lung nodule segmentation result. 6.5 Output Results: The final lung nodule segmentation results are overlaid with the original MRI image to generate a visualization result, and the segmentation result file is saved according to the preset format.

7. The method according to any one of claims 1 to 6, characterized in that, The lung multimodal MRI images include T1-weighted imaging, T2-weighted imaging, and diffusion-weighted imaging.

Images