Cross-modal ultrasound image segmentation method based on conditional diffusion model and transfer learning

Abstract

The present invention discloses a cross-modal ultrasound image segmentation method based on a conditional diffusion model and transfer learning, comprising: first obtaining a three-dimensional original modality dataset and a three-dimensional target modality dataset and performing preprocessing; then constructing a conditional diffusion model, setting training parameters and training the model, obtaining the trained conditional diffusion model and weight parameters of a conditional denoising network; then using the trained conditional diffusion model to infer the preprocessed three-dimensional original modality dataset, and batch translate it into labeled three-dimensional simulated ultrasound images; then migrating the trained weight parameters to a hybrid attention segmentation network and pretraining the segmentation network; finally, using transfer learning, fine-tuning the parameters of the pre-trained hybrid attention segmentation network through a three-dimensional target modality training set to adapt to three-dimensional real ultrasound data, and using the fine-tuned model that performs best in a three-dimensional target modality verification set as the hybrid attention segmentation model for image segmentation.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a cross-modal ultrasound image segmentation method based on a conditional diffusion model and transfer learning. Background Art

[0002] As a non-invasive imaging technology, ultrasound imaging is widely used for the diagnosis of various diseases, including brain tumors, pneumonia, and pleural effusions, in multiple locations, including the brain, chest, and abdomen, due to its real-time, high safety, and low cost. However, ultrasound image quality is limited by the imaging equipment, patient positioning, and the scanner's operating skills. Problems such as high noise, low contrast, and unclear anatomical structures often exist, making ultrasound image segmentation a difficult task in the diagnosis of brain and chest and abdominal diseases. Ultrasound image segmentation can not only accurately locate the lesion area but also provide important information about organ morphology, size, and function, assisting in early diagnosis and treatment decisions.

[0003] In recent years, deep learning technology, especially semantic segmentation algorithms based on convolutional neural networks (CNNs), has been widely used in medical image segmentation tasks in modalities such as MRI (Magnetic Resonance Imaging), CT (Computed Tomography), and US (Ultrasound), and has made significant progress. However, there are still several major problems in the automatic segmentation of brain and chest and abdominal ultrasound images: (1) Unclear boundary segmentation: Ultrasound images are noisy and tissue boundaries are not obvious, especially in imaging of chest and abdomen or deep brain structures. This makes it difficult for traditional image processing and segmentation algorithms to accurately segment the boundaries of the target area. (2) Poor model generalization ability: The anatomical structure of the brain and chest and abdomen is complex, and there are large individual differences among patients. Especially under different pathological conditions, the morphology and size of organs may change significantly. Such diversity requires ultrasound image segmentation models to have strong adaptability and generalization ability. (3) Insufficient standard data: Since ultrasound image annotation requires manual annotation by experts, and the ultrasound image sections and angles of each patient are different, the training data is scarce and the annotation is incomplete, which further increases the difficulty of the segmentation task. (4) Cross-modal data adaptation problem: Ultrasound images of the brain and chest and abdomen are often significantly different from other modal medical imaging data such as CT and MRI. Existing single-modality deep learning models have difficulty directly extracting consistent features from images of different modalities, which limits the joint analysis and utilization of multimodal images.

[0004] To address these challenges, diffusion models and transfer learning techniques have demonstrated promising results in ultrasound image segmentation in recent years. Diffusion models effectively suppress noise interference and improve image boundary segmentation accuracy by gradually reconstructing or generating high-quality feature distributions. Furthermore, transfer learning can significantly alleviate the problem of scarce labeled data by leveraging large-scale pre-trained models to fine-tune the target task.

[0005] Therefore, developing an ultrasound image segmentation algorithm based on diffusion model and transfer learning will be able to effectively expand the labeled data set, improve the model's adaptability to complex anatomical structures and multimodal features, and achieve accurate and clear-boundary segmentation of ultrasound images, which is of great significance for the diagnosis and treatment of diseases such as the brain and chest and abdomen. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the technical problem to be solved by the present invention is to provide a cross-modal ultrasound image segmentation method based on conditional diffusion model and transfer learning.

[0007] The technical solution of the present invention to solve the technical problem is to provide a cross-modal ultrasound image segmentation method based on a conditional diffusion model and transfer learning, characterized in that the method comprises the following steps:

[0008] Step 1: Acquire a 3D original modality dataset and a 3D target modality dataset of the same organ or lesion, perform data preprocessing to adapt the input of the conditional diffusion model and the hybrid attention segmentation model, obtain a preprocessed dataset, and divide the preprocessed dataset into a 3D original modality training set, a 3D original modality verification set, a 3D original modality test set, a 3D target modality training set, a 3D target modality verification set, and a 3D target modality test set; the 3D target modality dataset is a 3D real ultrasound dataset, consisting of 3D real ultrasound images;

[0009] Step 2: Construct a conditional diffusion model. Set training parameters based on the three-dimensional original modality training set and the three-dimensional target modality training set obtained in step 1 and fully train the conditional diffusion model by adopting dynamic learning rate adjustment and noise adjustment. After the training is completed, obtain the trained conditional diffusion model and the weight parameters of the conditional denoising network TU-Net in the trained conditional diffusion model.

[0010] Step 3: Use the trained conditional diffusion model from step 2 to perform inference on each 3D original modality image in the preprocessed 3D original modality dataset obtained from step 1, batch translate it into labeled 3D simulated ultrasound images, and then form a labeled 3D simulated ultrasound dataset. The labeled 3D simulated ultrasound dataset is then divided into a training set and a test set.

[0011] Step 4: Migrate the weight parameters of the conditional denoising network TU-Net in the trained conditional diffusion model obtained in step 2 to the hybrid attention segmentation network, and pre-train the hybrid attention segmentation network using the training set in step 3 to obtain the pre-trained hybrid attention segmentation network;

[0012] Step 5. Using the transfer learning method, the parameters of the pre-trained hybrid attention segmentation network obtained in step 4 are fine-tuned through the three-dimensional target modality training set obtained in step 1 to adapt to the three-dimensional real ultrasound data. The fine-tuned model that performs best in the three-dimensional target modality verification set in step 1 after the fine-tuning training is completed is used as the hybrid attention segmentation model; the three-dimensional target modality test set in step 1 is then processed using the hybrid attention segmentation model to obtain the segmentation result.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] (1) The present invention utilizes the conditional diffusion model for cross-modal translation to generate a high-quality labeled three-dimensional simulated ultrasound dataset. At the same time, the generated three-dimensional simulated ultrasound dataset is combined with the conditional diffusion model and the hybrid attention segmentation model using migration technology to achieve accurate, fast, and efficient training of the hybrid attention segmentation model, achieving high accuracy, fast speed, and clear boundary effects for lesion segmentation in the three-dimensional ultrasound dataset.

[0015] (2) The present invention can directly realize cross-modal image translation of three-dimensional medical images. By combining multiple modal data sets and using the conditional diffusion model, multiple three-dimensional original modal images can be directly translated into three-dimensional simulated ultrasound images.

[0016] (3) The present invention utilizes a conditional diffusion model to generate a 3D simulated ultrasound dataset, providing a large amount of pre-training data for downstream segmentation tasks. This solves the problem of overfitting and poor performance of model training caused by insufficient 3D ultrasound images. It also effectively mitigates the differences between multimodal data, enabling the model to better utilize multimodal data migration.

[0017] (4) The present invention establishes a connection between the conditional diffusion model and the semantic segmentation model by migrating the weights of the conditional diffusion model to the hybrid attention segmentation model. By sharing encoder feature extraction, the present invention further realizes the transfer learning from the conditional diffusion model to the hybrid attention segmentation model, thereby improving the training speed and segmentation accuracy of the hybrid attention segmentation model.

[0018] (5) The present invention uses a fine-tuning strategy to locally fine-tune the hybrid attention segmentation model on a three-dimensional real ultrasound dataset, which fully improves the generalization of the model and makes the model applicable to various three-dimensional ultrasound image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is the overall flow chart of the present invention;

[0020] Figure 2 Schematic diagram of the conditional diffusion model of the present invention;

[0021] Figure 3 Schematic diagram of the structure of the conditional denoising network TU-Net of the present invention;

[0022] Figure 4 Schematic diagram of the principle of the segmentation method of the present invention;

[0023] Figure 5 A raw brain ultrasound image according to an embodiment of the present invention;

[0024] Figure 6 For the present invention Figure 5 Brain tumor segmentation result map;

[0025] Figure 7 A raw brain ultrasound image according to another embodiment of the present invention;

[0026] Figure 8 For the present invention Figure 7 Brain tumor segmentation results. DETAILED DESCRIPTION

[0027] The specific embodiments of the present invention are given below. The specific embodiments are only used to further illustrate the present invention and do not limit the scope of protection of the present invention.

[0028] The present invention provides a cross-modal ultrasound image segmentation method based on a conditional diffusion model and transfer learning (hereinafter referred to as the method), which comprises the following steps:

[0029] Step 1: Obtain a 3D original modality dataset and a 3D target modality dataset of the same organ or lesion from a public medical imaging dataset (this embodiment uses the Resect public dataset), perform data preprocessing to adapt the input of the conditional diffusion model and the hybrid attention segmentation model, obtain the preprocessed dataset, and divide the preprocessed dataset into a 3D original modality training set, a 3D original modality verification set, a 3D original modality test set, a 3D target modality training set, a 3D target modality verification set, and a 3D target modality test set;

[0030] Preferably, in step 1, the three-dimensional original modality dataset is composed of three-dimensional original modality images, which are three-dimensional MRI images or three-dimensional CT images; and the three-dimensional target modality dataset is a three-dimensional real ultrasound dataset, which is composed of three-dimensional real ultrasound images.

[0031] Preferably, in step 1, the training set, validation set, and test set are divided in a ratio of 7:2:1.

[0032] Preferably, in step 1, the specific steps of data preprocessing are as follows:

[0033] S11. Using a Gaussian filtering algorithm to remove artifacts and noise from the 3D original modality dataset and the 3D target modality dataset;

[0034] S12, using the 3D target modality dataset as a standard, adjusting the image origin and coordinate system direction of the 3D original modality dataset to be the same as those of the 3D target modality dataset, and adjusting the voxel spacing and number of slices in the 3D original modality dataset to be the same as those of the 3D target modality dataset;

[0035] In this embodiment, the voxel spacing is uniformly 0.5 mm×0.5 mm×0.5 mm, and the number of slices is uniformly 256×256×256.

[0036] S13, using the organ or lesion label corresponding to the 3D original modality dataset to determine the field of view of the 3D original modality dataset, and then cropping the 3D original modality dataset based on the organ or lesion label information according to the field of view;

[0037] In this embodiment, the specific operation of determining the field of view of the three-dimensional original modality dataset is: taking the center of the lesion as the center point, and expanding the window size of the three-dimensional target modality dataset outward by 5mm for cropping, and the cropped area is used as the field of view of the three-dimensional original modality dataset.

[0038] S14, normalizing the three-dimensional target modal dataset and the cropped three-dimensional original modal dataset so that their intensity values ​​are adapted to the input of the conditional diffusion model;

[0039] In this embodiment, the initial intensity value of the 3D target modal dataset is 0-150, and the initial intensity value of the 3D original modal dataset is -200-1500, which are uniformly normalized to -1-1.

[0040] S15. Perform data enhancement on the data in the normalized three-dimensional original modal dataset and the three-dimensional target modal dataset used as training sets.

[0041] Preferably, in step S15, data enhancement uses flipping with a probability of 50% and Gaussian blurring.

[0042] Step 2: Construct a conditional diffusion model. Based on the three-dimensional original modality training set and the three-dimensional target modality training set obtained in step 1, set corresponding training parameters and fully train the conditional diffusion model by dynamic learning rate adjustment and noise adjustment. After the training is completed, obtain the trained conditional diffusion model and the weight parameters of the conditional denoising network TU-Net in the trained conditional diffusion model.

[0043] Preferably, in step 2, the overall structure of the conditional diffusion model is as follows: Figure 2 As shown in the figure, it consists of a forward diffusion process and a backward diffusion process performed sequentially. The forward diffusion process inputs the 3D original modality image in the 3D original modality training set, and generates a completely Gaussian noise image and process-noised images with different noise levels by adding Gaussian noise step by time. The backward diffusion process takes the completely Gaussian noise image as the starting point, and guides the conditional denoising network TU-Net to denoise (remove noise) step by time through conditional information, thereby improving the texture and semantic consistency of the generated image until a completely denoised image is generated.

[0044] Preferably, in step 2, during the forward diffusion process, the process of adding Gaussian noise time-step by time-step is as follows:

[0045]

[0046] In formula (1), x0 represents the input image; x t is the noisy image at time step t; I is the identity matrix; where α t =1-β t , β t represents the noise variance adjustment coefficient at time t; ∈ represents Gaussian noise with zero mean and unit variance.

[0047] Preferably, in step 2, during the back diffusion process, the process of denoising and restoring the image step by step is as follows:

[0048]

[0049] In formula (2), L represents conditional information; μ θ ,∑ θ Respectively represent the mean and variance of the neural network prediction noise; ∈ θ (x t , t) represents the noise of the neural network prediction time step t; x t-1 represents the denoised image at time step t-1.

[0050] Preferably, in step 2, the training parameters include the diffusion time step and the number of training rounds. In this embodiment, the time step of the forward diffusion process and the backward diffusion process are both set to 1000, and the number of training rounds is 500.

[0051] Preferably, in step 2, the dynamic learning rate adjustment adopts cosine annealing, and the noise adjustment adopts linear scheduling.

[0052] Preferably, in step 2, the structure of the conditional denoising network TU-Net is as follows: Figure 3As shown, it includes five layers of parallel encoders and five layers of shared decoders. The five layers of parallel encoders are connected in sequence, and the five layers of shared decoders are connected in sequence. The output of the fifth layer of parallel encoder is transmitted to the first layer of shared decoder.

[0053] The output of the parallel encoder on the first layer is simultaneously transmitted to the shared decoder on the fourth layer via a skip connection. The output of the parallel encoder on the second layer is simultaneously transmitted to the shared decoder on the third layer via a skip connection. The output of the parallel encoder on the third layer is simultaneously transmitted to the shared decoder on the second layer via a skip connection. The output of the parallel encoder on the fourth layer is simultaneously transmitted to the shared decoder on the first layer via a skip connection.

[0054] The structures of the 1st to 4th layer parallel encoders are the same, consisting of a CNN encoding block, a conditional information embedding module, a Transformer encoding block, and a feature fusion module (CFIM). The CNN encoding block, conditional information embedding module, and Transformer encoding block are parallel. The input enters the CNN encoding block, conditional information embedding module, and Transformer encoding block. The conditional information embedding module processes the conditional information and outputs the CNN local feature weight w. CNN and Transformer local feature weight w Tr , which are used to modulate the output of the CNN encoding block and the Transformer encoding block respectively; the feature fusion module combines the output features F from the CNN encoding block CNN and the output feature F of the Transformer encoding block Tr Perform fusion and obtain the fused feature map as the output of each layer of parallel encoder;

[0055] The fifth parallel encoder layer is the bottleneck layer, consisting of two convolution blocks. After each convolution, BatchNormalization is used for normalization, followed by Sigmoid activation function for activation, and then downsampling is performed. Preferably, each convolution block uses a 3×3×3 convolution kernel with padding and stride set to 1. Downsampling uses 2×2×2 maximum pooling.

[0056] The structures of the shared decoders in layers 1 to 5 are the same. They all first perform upsampling, then use two convolution blocks. Batch Normalization is used after each convolution, and then the Sigmoid activation function is used for activation. Preferably, cubic linear interpolation is used for upsampling. Each convolution block uses a 3×3×3 convolution kernel, and the padding and stride are both set to 1.

[0057] Preferably, in step 2, in the CNN encoding block, two convolution blocks are first used, and BatchNormalization is used for normalization after each convolution, and then a Sigmoid activation function is used for activation, and then downsampling is performed. More preferably, each convolution block uses a 3×3×3 convolution kernel, with padding and stride both set to 1; downsampling uses a 2×2×2 maximum pooling;

[0058] In the Transformer encoding block, a self-attention mechanism modulated by conditional information is first adopted, and then normalized using LayerNorm;

[0059] The conditional information embedding module uses a convolution block; more preferably, the convolution block uses a 3×3×3 convolution kernel;

[0060] The feature fusion module uses 1×1×1 convolution for feature alignment, followed by feature splicing, and finally uses 3×3×3 convolution for feature extraction.

[0061] Preferably, in step 2, in the Transformer encoding block, the specific implementation of the self-attention mechanism modulated by the conditional information is:

[0062]

[0063] In formula (3), Q = XW Q , K=XW K , V=XW V ;W Q , W K , W V is a learnable weight matrix; X is the input feature; d k is the dimension of the key; F MI is the weight matrix for dynamically modulating multi-head attention.

[0064] Preferably, in step 2, the feature fusion module is represented as:

[0065] F CFIM =Conv3×3×3([w CNN Conv1×1×1(F CNN )+w Tr Conv1×1×1(F Tr )]) (4)

[0067] In formula (4), Conv1×1×1 represents the convolution operation with a convolution kernel size of 1×1×1, which is used for feature alignment; [·] represents the feature splicing operation, and Conv3×3×3 is used to integrate the spliced ​​features.

[0068] Preferably, in step 2, the specific steps of denoising each time step of the reverse diffusion process are as follows:

[0069] (1) Calculate the local mutual information between the process-noised image of the current time step and the three-dimensional real ultrasound image with a dimension of n×a×b×c (1×256×256×256 in this embodiment) as the conditional information; then input the process-noised image of the current time step and the conditional information of the same dimension into the first layer parallel encoder of the conditional denoising network TU-Net; the conditional information enters the conditional information embedding module of the first layer parallel encoder to generate the CNN local feature weight w CNN and Transformer local feature weight w Tr The process-noised image of the current time step enters the CNN encoding block and the Transformer encoding block of the first-layer parallel encoder at the same time. After two convolutions in the CNN encoding block, the output feature map of the dimension is c×a / 2×b / 2×c / 2 (64×128×128×128 in this embodiment). After the self-attention mechanism in the Transformer encoding block, the output feature map of the dimension is c×a / 2×b / 2×c / 2. Then the output feature map F of the CNN encoding block is CNN and the output feature F of the Transformer encoding block Tr After splicing in the feature fusion module, the output feature map with a dimension of c×a / 2×b / 2×c / 2 is the output of the first layer parallel encoder;

[0070] (2) Calculating the local mutual information between the feature map output by the first-layer parallel encoder and the three-dimensional real ultrasound image as conditional information; feeding the conditional information and the feature map output by the first-layer parallel encoder into the second-layer parallel encoder, and the second-layer parallel encoder outputs a feature map with a dimension of 2c×a / 4×b / 4×c / 4 (128×64×64×64 in this embodiment);

[0071] (3) Calculating the local mutual information between the feature map output by the second-layer parallel encoder and the three-dimensional real ultrasound image as conditional information; feeding the conditional information and the feature map output by the second-layer parallel encoder into the third-layer parallel encoder, and the third-layer parallel encoder outputs a feature map with a dimension of 4c×a / 8×b / 8×c / 8 (256×32×32×32 in this embodiment);

[0072] (4) Calculating the local mutual information between the feature map output by the third-layer parallel encoder and the three-dimensional real ultrasound image as conditional information; feeding the conditional information and the feature map output by the third-layer parallel encoder into the fourth-layer parallel encoder, and the fourth-layer parallel encoder outputs a feature map with a dimension of 8c×a / 16×b / 16×c / 16 (512×16×16×16 in this embodiment);

[0073] (5) Calculate the local mutual information between the feature map output by the 4th parallel encoder and the 3D real ultrasound image as conditional information; send the conditional information and the feature map output by the 4th parallel encoder to the bottleneck layer. After two 3×3×3 convolutions at the bottleneck layer, the output feature map has a dimension of 16c×a / 32×b / 32×c / 32 (1024×8×8×8 in this embodiment);

[0074] (6) The feature map output by the bottleneck layer is fed into the first-layer shared decoder. It is first upsampled once and then passes through two convolution blocks. Batch Normalization is used after each convolution to perform normalization. The Sigmoid activation function is then used for activation. The feature map is then concatenated with the feature map output by the fourth-layer parallel encoder through a skip connection. The output dimension is a feature map of 8c×a / 16×b / 16×c / 16 (512×16×16×16 in this embodiment).

[0075] (7) The feature map output by the first-layer shared decoder is input to the second-layer shared decoder, and is sequentially upsampled, convolved twice, normalized once, and activated once. It is then concatenated with the feature map output by the third-layer parallel encoder through a skip connection, and the output dimension is a feature map of 4c×a / 8×b / 8×c / 8 (256×32×32×32 in this embodiment).

[0076] (8) The feature map output by the second-layer shared decoder is input to the third-layer shared decoder, and is sequentially up-sampled, convolved twice, normalized once, and activated once. It is then concatenated with the feature map output by the second-layer parallel encoder through a skip connection, and the output dimension is a feature map of 2c×a / 4×b / 4×c / 4 (128×64×64×64 in this embodiment).

[0077] (9) The feature map output by the third-layer shared decoder is input to the fourth-layer shared decoder, and is sequentially upsampled, convolved twice, normalized once, and activated once. It is then concatenated with the feature map output by the first-layer parallel encoder through a skip connection, and the output dimension is a feature map of c×a / 2×b / 2×c / 2 (64×128×128×128 in this embodiment).

[0078] (10) The feature map output by the 4th layer shared decoder is input to the 5th layer shared decoder, and is sequentially subjected to one upsampling, two convolutions, one normalization, and one activation, outputting a feature map with a dimension of c×a×b×c (64×256×256×256 in this embodiment);

[0079] (11) The feature map output by the fifth-layer shared decoder is subjected to 1×1×1 convolution for dimensionality reduction, and the denoised image of the current time step of n×a×b×c (1×256×256×256 in this embodiment) is output.

[0080] Preferably, in step 2, the conditional information used in the conditional denoising network TU-Net is local mutual information (LMI); in the training phase, by calculating the local mutual information, the conditional diffusion model is guided to learn the texture feature information of the target modality, adjust the noise distribution in the process-noised image, and finally train the conditional diffusion model to generate an image with the same style as the three-dimensional target modality image; the local mutual information is realized as follows:

[0081]

[0082] In formula (5), p δ (x, y) is the joint probability density function on the domain δ; In the field δ xi and δ yj The marginal probability density function of .

[0083] Preferably, in step 2, the denoising loss function of the conditional diffusion model combined with local mutual information is expressed as:

[0084]

[0085] In formula (6), LMI norm (F; F, F t ) is normalized to measure the target mode F and the current time step F t The local mutual information between the noisy images; S θ (F t , LMI, t) is the conditional denoising network combined with mutual information and the prediction noise of time step t; ∈ is the Gaussian noise of the current time step; λ is the loss weight of the local mutual information term; the denoising ability of the model is continuously optimized by minimizing the loss function, and all parameters in the conditional diffusion model are updated in reverse.

[0086] Step 3: Use the trained conditional diffusion model from step 2 to perform inference on each 3D original modality image in the preprocessed 3D original modality dataset obtained from step 1, batch translate it into labeled 3D simulated ultrasound images, and then form a labeled 3D simulated ultrasound dataset. The labeled 3D simulated ultrasound dataset is then divided into a training set and a test set.

[0087] Preferably, in step 3, the inference process uses the local mutual information of the 3D original modality image to guide the conditional diffusion model. The overall semantic features of the 3D original modality image are learned to further guide the prediction process of the conditional denoising network, so that the final generated 3D simulated ultrasound image is close to the texture features of the target modality while retaining the semantic features of the original modality.

[0088] Step 4: Migrate the weight parameters of the conditional denoising network TU-Net in the trained conditional diffusion model obtained in step 2 to the hybrid attention segmentation network, pre-train the hybrid attention segmentation network using the training set in step 3, and test the pre-training effect using the test set in step 3 to obtain the pre-trained hybrid attention segmentation network;

[0089] Preferably, in step 4, the structure of the hybrid attention segmentation network is the same as that of the conditional denoising network TU-Net, with the only difference being that no conditional information embedding module is set in the 1st to 4th layer parallel encoders.

[0090] Preferably, in step 4, the parallel encoder of the hybrid attention segmentation network is initialized using the weight parameters of the conditional denoising network in the trained conditional diffusion model obtained in step 2, and the trained weight parameters are assigned to the hybrid attention segmentation network, so that the encoders of the conditional denoising network and the hybrid attention segmentation network can directly share the ability to extract low-level and intermediate features, thereby improving the adaptability of the hybrid attention segmentation network to cross-modality, and enhancing the feature extraction capability of the encoder of the hybrid attention segmentation network for three-dimensional ultrasound images, while accelerating convergence and reducing the probability of overfitting, thereby ultimately improving the segmentation effect of the hybrid attention segmentation network.

[0091] Preferably, in step 4, the Transformer encoding block of the parallel encoder of the hybrid attention segmentation network captures global features through a self-attention mechanism; the self-attention mechanism is implemented as follows:

[0092]

[0093] In formula (7), Q = XW Q , K=XW K , V=XW V ;W Q , W K， W V is a learnable weight matrix; X is the input feature; d k is the dimension of the key.

[0094] Preferably, in step 4, the structure of the feature fusion module in the hybrid attention segmentation network is the same as the feature fusion module in the conditional denoising network TU-Net, with the only difference being that the feature weights of the CNN and the feature weights of the Transformer are fixed and their sum is equal to 1. In this embodiment, the feature weights of the CNN and the feature weights of the Transformer are both set to 0.5.

[0095] Preferably, in step 4, the loss function of the hybrid attention segmentation network is expressed as:

[0096] L seg =αL Dice +βL Edge

[0097]

[0098] In formula (8), L Dice Dice loss can improve the segmentation accuracy of small target areas; L Edge It is an edge-aware loss that focuses on matching edge areas and improves boundary segmentation accuracy. i is the predicted segmentation probability; g i is the true label; N is the total number of pixels; represents the Sobel gradient operator; N e Indicates the number of edge pixels; ∥·∥ 2 represents the square of the two norm; L seg is the total loss function; α and β are weighting coefficients, and in this embodiment, α and β are both set to 1; the segmentation effect of the overall area and edge of the hybrid attention segmentation network is optimized by using the joint loss function.

[0099] Step 5. Using the transfer learning method, the parameters of the pre-trained hybrid attention segmentation network obtained in step 4 are fine-tuned through the three-dimensional target modality training set obtained in step 1 to adapt to the three-dimensional real ultrasound data. The fine-tuned model that performs best in the three-dimensional target modality verification set in step 1 after the fine-tuning training is completed is used as the hybrid attention segmentation model; the three-dimensional target modality test set in step 1 is then processed using the hybrid attention segmentation model to obtain the segmentation result.

[0100] Preferably, in step 5, the fine-tuning training specifically includes: training and updating the parameters of the CNN encoding block of the pre-trained hybrid attention segmentation network obtained in step 4, freezing the parameters of the Transformer encoding block, and adjusting the network's ability to extract local features to make the network more adaptable to three-dimensional real ultrasound data.

[0101] Depend on Figure 5 and Figure 6 contrast, Figure 7and Figure 8 By comparison, it can be seen that the present invention can accurately segment the ultrasound image to obtain a segmentation map with clear boundaries.

[0102] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A cross-modal ultrasound image segmentation method based on conditional diffusion model and transfer learning, characterized in that: The method comprises the following steps: Step 1: Acquire a 3D original modality dataset and a 3D target modality dataset of the same organ or lesion, perform data preprocessing to adapt the input of the conditional diffusion model and the hybrid attention segmentation model, obtain a preprocessed dataset, and divide the preprocessed dataset into a 3D original modality training set, a 3D original modality verification set, a 3D original modality test set, a 3D target modality training set, a 3D target modality verification set, and a 3D target modality test set; the 3D target modality dataset is a 3D real ultrasound dataset, consisting of 3D real ultrasound images; Step 2: Construct a conditional diffusion model. Set training parameters based on the three-dimensional original modality training set and the three-dimensional target modality training set obtained in step 1 and fully train the conditional diffusion model by adopting dynamic learning rate adjustment and noise adjustment. After the training is completed, obtain the trained conditional diffusion model and the weight parameters of the conditional denoising network TU-Net in the trained conditional diffusion model. The conditional denoising network TU-Net includes five layers of parallel encoders and five layers of shared decoders. The five layers of parallel encoders are connected sequentially, and the five layers of shared decoders are connected sequentially. The output of the fifth layer of parallel encoder is transmitted to the first layer of shared decoder. The output of the parallel encoder on the first layer is simultaneously transmitted to the shared decoder on the fourth layer via a skip connection. The output of the parallel encoder on the second layer is simultaneously transmitted to the shared decoder on the third layer via a skip connection. The output of the parallel encoder on the third layer is simultaneously transmitted to the shared decoder on the second layer via a skip connection. The output of the parallel encoder on the fourth layer is simultaneously transmitted to the shared decoder on the first layer via a skip connection. The structures of the 1st to 4th layer parallel encoders are the same, all consisting of a CNN encoding block, a conditional information embedding module, a Transformer encoding block and a feature fusion module. The CNN encoding block, conditional information embedding module and Transformer encoding block are parallel. The input enters the CNN encoding block, conditional information embedding module and Transformer encoding block. The conditional information embedding module processes the conditional information and outputs the CNN local feature weight w. CNN and Transformer local feature weight w Tr , which are used to modulate the output of the CNN encoding block and the Transformer encoding block respectively; the feature fusion module combines the output features F from the CNN encoding block CNN and the output feature F of the Transformer encoding block Tr Perform fusion and obtain the fused feature map as the output of each layer of parallel encoder; The fifth parallel encoder layer is the bottleneck layer, which includes two convolution blocks. After each convolution, Batch Normalization is used for normalization, and then the Sigmoid activation function is used for activation, and then downsampling is performed. The structures of the shared decoders in layers 1 to 5 are the same. They all first perform upsampling, then use two convolution blocks. After each convolution, they are normalized using Batch Normalization and then activated using the Sigmoid activation function. The conditional information used in the conditional denoising network TU-Net is local mutual information. During the training phase, the conditional diffusion model is trained to generate images with the same style as the 3D target modality image by calculating the local mutual information. The implementation of local mutual information is as follows: In formula (5), p δ (x,y) is the joint probability density function on the domain δ; In the field δ xi and δ yj The marginal probability density function of Step 3: Use the trained conditional diffusion model from step 2 to perform inference on each 3D original modality image in the preprocessed 3D original modality dataset obtained from step 1, batch translate it into labeled 3D simulated ultrasound images, and then form a labeled 3D simulated ultrasound dataset. The labeled 3D simulated ultrasound dataset is then divided into a training set and a test set. Step 4: Migrate the weight parameters of the conditional denoising network TU-Net in the trained conditional diffusion model obtained in step 2 to the hybrid attention segmentation network, and pre-train the hybrid attention segmentation network using the training set in step 3 to obtain the pre-trained hybrid attention segmentation network; The hybrid attention segmentation network has the same structure as the conditional denoising network TU-Net, with the only difference being that the conditional information embedding module is not set in the 1st to 4th layer parallel encoders. The structure of the feature fusion module in the hybrid attention segmentation network is the same as that in the conditional denoising network TU-Net. The only difference is that the feature weights of the CNN and the Transformer are fixed and their sum is equal to 1. Step 5. Using the transfer learning method, the parameters of the pre-trained hybrid attention segmentation network obtained in step 4 are fine-tuned through the three-dimensional target modality training set obtained in step 1 to adapt to the three-dimensional real ultrasound data. The fine-tuned model that performs best in the three-dimensional target modality verification set in step 1 after the fine-tuning training is completed is used as the hybrid attention segmentation model; the three-dimensional target modality test set in step 1 is then processed using the hybrid attention segmentation model to obtain the segmentation result.

2. The cross-modal ultrasound image segmentation method based on conditional diffusion model and transfer learning according to claim 1, characterized in that: In step 1, the three-dimensional original modality dataset consists of three-dimensional original modality images, and the three-dimensional original modality images are three-dimensional MRI images or three-dimensional CT images; In step 1, the specific steps of data preprocessing are as follows: S11. Using a Gaussian filtering algorithm to remove artifacts and noise from the 3D original modality dataset and the 3D target modality dataset; S12, using the 3D target modality dataset as a standard, adjusting the image origin and coordinate system direction of the 3D original modality dataset to be the same as those of the 3D target modality dataset, and adjusting the voxel spacing and number of slices in the 3D original modality dataset to be the same as those of the 3D target modality dataset; S13, using the organ or lesion label corresponding to the 3D original modality dataset to determine the field of view of the 3D original modality dataset, and then cropping the 3D original modality dataset based on the organ or lesion label information according to the field of view; S14, normalizing the three-dimensional target modal dataset and the cropped three-dimensional original modal dataset so that their intensity values ​​are adapted to the input of the conditional diffusion model; S15. Perform data enhancement on the data in the normalized three-dimensional original modal dataset and the three-dimensional target modal dataset used as training sets.

3. The cross-modal ultrasound image segmentation method based on conditional diffusion model and transfer learning according to claim 1, characterized in that: In step 2, the conditional diffusion model consists of a forward diffusion process and a backward diffusion process. The forward diffusion process inputs the 3D original modality image in the 3D original modality training set and generates a completely Gaussian noisy image and process-noised images with different noise levels by adding Gaussian noise step by time. The backward diffusion process uses the completely Gaussian noisy image as the starting point and guides the conditional denoising network TU-Net to perform denoising step by time using conditional information to improve the texture and semantic consistency of the generated image until a completely denoised image is generated. In step 2, during the forward diffusion process, the process of adding Gaussian noise step by time is as follows: In formula (1), x0 represents the input image; x t is the noisy image at time step t; I is the identity matrix; where α t =1-β t , β t represents the noise variance adjustment coefficient at time t; ∈ represents Gaussian noise with zero mean and unit variance; In step 2, during the back diffusion process, the denoising process is as follows: In formula (2), L represents conditional information; μ θ ,∑ θ Respectively represent the mean and variance of the neural network prediction noise; ∈ θ (x t ,t) represents the noise of the neural network prediction time step t; x t-1 represents the denoised image at time step t-1.

4. The cross-modal ultrasound image segmentation method based on conditional diffusion model and transfer learning according to claim 1, characterized in that: In step 2, the training parameters include the diffusion time step and the number of training rounds; In step 2, cosine annealing is used for dynamic learning rate adjustment, and linear scheduling is used for noise regulation.

5. The cross-modal ultrasound image segmentation method based on conditional diffusion model and transfer learning according to claim 1, characterized in that: In step 2, in the CNN encoding block, two convolution blocks are first used. BatchNormalization is used for normalization after each convolution, and then the Sigmoid activation function is used for activation, and then downsampling is performed; In the Transformer encoding block, a self-attention mechanism modulated by conditional information is first adopted, and then normalized using LayerNorm; The conditional information embedding module adopts a convolutional block; The feature fusion module uses 1×1×1 convolution for feature alignment, followed by feature splicing, and finally uses 3×3×3 convolution for feature extraction; In step 2, the specific implementation of the self-attention mechanism modulated by conditional information in the Transformer encoding block is: In formula (3), Q = XW Q ,K=XW K ,V=XW V ;W Q ,W K ,W V is a learnable weight matrix; X is the input feature; d k is the dimension of the key; F MI is the weight matrix for dynamically modulating multi-head attention; In step 2, the feature fusion module is expressed as: F CFIM =Conv3×3×3([w CNN ·Conv1×1×1(F CNN )+w Tr ·Conv1×1×1(F Tr )]) (4) In formula (4), Conv1×1×1 represents the convolution operation with a convolution kernel size of 1×1×1, which is used for feature alignment; [·] represents the feature concatenation operation, and Conv3×3×3 is used to integrate the concatenated features.

6. The cross-modal ultrasound image segmentation method based on conditional diffusion model and transfer learning according to claim 3, characterized in that: In step 2, the specific steps of denoising each time step of the back diffusion process are as follows: (1) Calculate the local mutual information between the process-noised image of the current time step and the three-dimensional real ultrasound image with the dimension of n×a×b×c as the conditional information; Then the process noise image and conditional information of the current time step are input into the first layer parallel encoder of the conditional denoising network TU-Net; the conditional information enters the conditional information embedding module of the first layer parallel encoder to generate the CNN local feature weight w CNN and Transformer local feature weight w Tr The process-noised image of the current time step enters the CNN encoding block and the Transformer encoding block of the first-layer parallel encoder at the same time. After two convolutions in the CNN encoding block, the output feature map of the dimension is c×a / 2×b / 2×c / 2. After the self-attention mechanism in the Transformer encoding block, the output feature map of the dimension is c×a / 2×b / 2×c / 2. Then the output feature map of the CNN encoding block is F CNN and the output feature F of the Transformer encoding block Tr After splicing in the feature fusion module, the output feature map with a dimension of c×a / 2×b / 2×c / 2 is the output of the first layer parallel encoder; (2) Calculate the local mutual information between the feature map output by the first-layer parallel encoder and the three-dimensional real ultrasound image as conditional information; send the conditional information and the feature map output by the first-layer parallel encoder to the second-layer parallel encoder, and the second-layer parallel encoder outputs a feature map with a dimension of 2c×a / 4×b / 4×c / 4; (3) Calculate the local mutual information between the feature map output by the second-layer parallel encoder and the three-dimensional real ultrasound image as conditional information; send the conditional information and the feature map output by the second-layer parallel encoder to the third-layer parallel encoder, and the third-layer parallel encoder outputs a feature map with a dimension of 4c×a / 8×b / 8×c / 8; (4) Calculate the local mutual information between the feature map output by the third-layer parallel encoder and the three-dimensional real ultrasound image as conditional information; send the conditional information and the feature map output by the third-layer parallel encoder to the fourth-layer parallel encoder, and the fourth-layer parallel encoder outputs a feature map with a dimension of 8c×a / 16×b / 16×c / 16; (5) Calculate the local mutual information between the feature map output by the 4th parallel encoder and the 3D real ultrasound image as the conditional information; send the conditional information and the feature map output by the 4th parallel encoder to the bottleneck layer. After two convolutions in the bottleneck layer, the output feature map has a dimension of 16c×a / 32×b / 32×c / 32; (6) The feature map output by the bottleneck layer is fed into the first-layer shared decoder, which is first upsampled once and then passes through two convolution blocks. Batch Normalization is used after each convolution, and then the Sigmoid activation function is used for activation. The feature map is then concatenated with the feature map output by the fourth-layer parallel encoder through a skip connection, and the output dimension is a feature map of 8c×a / 16×b / 16×c / 16. (7) The feature map output by the first-layer shared decoder is input to the second-layer shared decoder, and is sequentially up-sampled, convolved twice, normalized once, and activated once. It is then concatenated with the feature map output by the third-layer parallel encoder through a skip connection, and the output dimension is a feature map of 4c×a / 8×b / 8×c / 8. (8) The feature map output by the second-layer shared decoder is input to the third-layer shared decoder, and is sequentially up-sampled, convolved twice, normalized once, and activated once. It is then concatenated with the feature map output by the second-layer parallel encoder through a skip connection, and the output dimension is a feature map of 2c×a / 4×b / 4×c / 4. (9) The feature map output by the third-layer shared decoder is input to the fourth-layer shared decoder, and is sequentially up-sampled, convolved twice, normalized once, and activated once. It is then concatenated with the feature map output by the first-layer parallel encoder through a skip connection, and the output dimension is a feature map of c×a / 2×b / 2×c / 2. (10) The feature map output by the 4th layer shared decoder is input into the 5th layer shared decoder, and is sequentially subjected to one upsampling, two convolutions, one normalization, and one activation, and the output feature map has a dimension of c×a×b×c; (11) The feature map output by the 5th layer shared decoder is subjected to 1×1×1 convolution for dimensionality reduction, and the denoised image of the current time step of n×a×b×c is output.

7. The cross-modal ultrasound image segmentation method based on conditional diffusion model and transfer learning according to claim 3, characterized in that: In step 2, the denoising loss function of the conditional diffusion model combined with local mutual information is expressed as: In formula (6), LMI norm (F; F, F t ) is normalized to measure the target mode F and the current time step F t The local mutual information between the noisy images; S θ (F t ,LMI,t) is the conditional denoising network combined with mutual information and the prediction noise of time step t; ∈ is the Gaussian noise of the current time step; λ is the loss weight of the local mutual information term; the denoising ability of the model is continuously optimized by minimizing the loss function, and all parameters in the conditional diffusion model are updated in reverse.

8. The cross-modal ultrasound image segmentation method based on conditional diffusion model and transfer learning according to claim 1, characterized in that: In step 4, the parallel encoder of the hybrid attention segmentation network is initialized using the weight parameters of the conditional denoising network in the trained conditional diffusion model obtained in step 2, and the trained weight parameters are assigned to the hybrid attention segmentation network, so that the encoders of the conditional denoising network and the hybrid attention segmentation network can directly share the ability to extract low-level and mid-level features, thereby improving the adaptability of the hybrid attention segmentation network to cross-modality and enhancing the feature extraction capability of the encoder of the hybrid attention segmentation network for three-dimensional ultrasound images. At the same time, it accelerates convergence, reduces the probability of overfitting, and ultimately improves the segmentation effect of the hybrid attention segmentation network. In step 4, the Transformer encoding block of the parallel encoder of the hybrid attention segmentation network captures global features through the self-attention mechanism; the self-attention mechanism is implemented as: In formula (7), Q = XW Q ,K=XW K ,V=XW V ;W Q ,W K ,W V is a learnable weight matrix; X is the input feature; d k is the dimension of the key; In step 4, the loss function of the hybrid attention segmentation network is expressed as: L seg =αL Dice +βL Edge In formula (8), L Dice is the Dice loss; L Edge is the edge perception loss; p i is the predicted segmentation probability; g i is the true label; N is the total number of pixels; represents the Sobel gradient operator; N e Indicates the number of edge pixels; ||·|| 2 represents the square of the two norm; L seg is the total loss function; α and β are weighting coefficients; the overall area and edge segmentation effect of the hybrid attention segmentation network are optimized by using the joint loss function.

9. The cross-modal ultrasound image segmentation method based on conditional diffusion model and transfer learning according to claim 1, characterized in that: In step 5, the fine-tuning training is specifically: training and updating the parameters of the CNN encoding block of the pre-trained hybrid attention segmentation network obtained in step 4, and freezing the parameters of the Transformer encoding block.

Images