Lymphoma ultrasound image semantic segmentation method based on self-attention mechanism and stable learning
By introducing a segmentation method of self-attention mechanism and stable learning in lymphoma ultrasound image processing, the problem of poor segmentation of lymphoma ultrasound image is solved, automated segmentation is achieved, and accuracy and efficiency are improved.
Patent Information
- Application Number
- CN202210604099.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-31
- Publication Date
- 2025-04-04
- Estimated Expiration
- 2042-05-31
AI Technical Summary
The existing lymphoma image segmentation methods mostly focus on three-dimensional PET and CT images, which have poor effect on lymphoma ultrasound image segmentation. Moreover, due to the high image noise and low contrast, the segmentation process depends on manual intervention and lacks objectivity and efficiency.
The semantic segmentation method of lymphoma ultrasound image based on self-attention mechanism and stable learning is adopted. Through data preprocessing, feature extraction, self-attention mechanism, stable learning and counterfactual interpretation modules, a segmentation network is built to realize automatic semantic segmentation of lymphoma ultrasound images.
Effective segmentation of lymphoma ultrasound images under a small amount of data improves segmentation accuracy and model credibility, reduces manual intervention, saves time and labor costs, and improves diagnostic efficiency and accuracy.
Smart Images
Figure CN114972266B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical image processing, and in particular to a semantic segmentation method of lymphoma ultrasound images based on self-attention mechanism and stable learning. Background Art
[0002] Lymphoma is a deadly cancer caused by abnormal mutations in immune system cells. There are many different histological subtypes, and their diagnosis is usually based on sampling (biopsy). Lymphoma originates in lymph nodes and lymphatic tissue, can occur anywhere in the body, and has a variety of clinical manifestations. The disease is usually characterized by painless and progressive lymphadenopathy, and can also invade extranodal organs and cause damage to the corresponding organs. It is mainly divided into two types: Hodgkin's lymphoma and non-Hodgkin's lymphoma.
[0003] The diagnosis of lymphoma requires a combination of the patient's clinical manifestations, physical examination, laboratory tests, imaging examinations, and pathological examination results. These different methods play different roles in the diagnosis of lymphoma staging and typing. Imaging technology plays an important role in the staging and typing of lymphoma. Commonly used imaging examination methods include CT, nuclear magnetic resonance imaging (MRI), positron emission tomography-computed tomography (PET-CT), ultrasound, and endoscopy.
[0004] Due to different imaging technologies, images obtained by different types of imaging methods have different effects on the diagnosis of lymphoma. Compared with other imaging technologies, ultrasound examination is a safe examination without radioactive damage. It is a completely non-destructive, non-invasive, and radiation-free examination technology. Since the ultrasound probe can be placed at will, multiple orientations and angles can be used for human body examinations. It is very flexible and can locate and measure the lesions, clarify the location of the lesions and the relationship between the surrounding tissues, and display them in real time. Especially for the position of the heart at different times, the changes in the Doppler blood flow of the heart, and the hemodynamics can be observed. Ultrasound examination can get the examination results immediately, and can also be repeated many times. Ultrasonic methods have wide availability (including bedside) and relatively low cost. Therefore, ultrasound examination plays an important role in the early diagnosis of lymphoma.
[0005] However, due to the inherent acoustic characteristics of ultrasound imaging, the resulting images are noisy, low in contrast, and poor in imaging quality. Image segmentation is necessary, and segmentation is the key to ultrasound image analysis. Generally speaking, this segmentation is performed manually by clinicians, which reduces the objectivity of diagnosis and consumes a lot of manpower. Even for experts, the descriptions made based on their experience and skills are slightly different. Therefore, for many medical image applications, the correct segmentation of the lesion area by the model is the key to successful application. An automatic segmentation model that can accurately obtain the region of interest (ROI) from the image can provide a basis for clinicians to diagnose or conduct pathological research. However, existing lymphoma image segmentation is mostly focused on three-dimensional positron emission tomography (PET) and computed tomography (CT) images, and no segmentation of lymphoma images is performed. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a semantic segmentation method for lymphoma ultrasound images based on self-attention mechanism and stable learning, which can effectively segment lymphoma ultrasound images with a small amount of data.
[0007] A semantic segmentation method for lymphoma ultrasound images based on self-attention mechanism and stable learning, comprising:
[0008] Obtaining lymphoma ultrasound images and processing the images as training samples;
[0009] Constructing a lymphoma ultrasound image segmentation network: the lymphoma ultrasound image segmentation network includes a data preprocessing module, a feature extraction network module, a self-attention mechanism module, a stable learning module, and a counterfactual explanation module;
[0010] The data preprocessing module is used to crop and annotate all lymphoma ultrasound image samples and use the preprocessed samples as input data for the segmentation network;
[0011] Feature extraction network module, used to extract spatial and global information of the image, and fuse the extracted features to capture clearer object boundaries;
[0012] The self-attention mechanism module is used to realize non-local interaction between features and alleviate the information decay problem caused by multiple sampling;
[0013] The stable learning module is used to eliminate the dependency between environmental features and essential features by random Fourier features and weighting samples to achieve generalization;
[0014] The counterfactual explanation module is used to provide instance-level explanations of model results to improve the credibility of the model.
[0015] Preferably, the operation steps of the data preprocessing module of the present invention include:
[0016] Step S11: cropping the ultrasound image scanned by the ultrasound instrument to remove the extra content added by the ultrasound scanning instrument;
[0017] Step S12 uploads the cropped image to the PLabel labeling system for free labeling. After the labeling is completed, the image labeled in the system is submitted to experienced medical experts for review and correction, and the final labeling result is exported;
[0018] Step S13 reads the exported json file to generate a real mask image;
[0019] Step S14 adjusts the pixel spacing between the annotated image and the original image to obtain an image with a resolution of 256×256, crops the image of the region of interest, and adjusts the resolution to 512×512.
[0020] Preferably, the feature extraction network module operation steps of the present invention include:
[0021] Step S21 uses the encoder structure to perform downsampling to extract low-level features, thereby using the extracted spatial information and global information for accurate segmentation;
[0022] In step S22, the decoder gradually restores the spatial information by upsampling and fuses the features extracted by the encoder during the encoding process to capture clearer object boundaries.
[0023] Preferably, the operation steps of the self-attention mechanism module of the present invention include:
[0024] Step S32 inputs the last layer of features encoded by the encoder into the TSA module. The TSA module performs different linear changes on the input feature map and the generated position embedding vector to generate three vectors: query (Q), key (K), and value (V) for attention calculation.
[0025] Step S32: Multiply the transpose of Q and K to obtain the similarity between the elements of Q and K, and divide it by the dimension d of vector Q. k The square root of ensures the gradient of softmax and normalizes it through softmax to get a context attention map A. Multiply A and V to get the weighted value of attention. The formula is as follows:
[0026]
[0027] Step S33 adds the return value obtained by the attention mechanism module to the last layer feature element by element to obtain a fused feature map F for the decoder input.
[0028] Preferably, the operation steps of the stable learning module of the present invention include:
[0029] Step S41 inputs the feature map into the stable learning module, maps the input features from low-dimensional space to high-dimensional space through random Fourier features, and eliminates the correlation between features. The random Fourier feature formula is as follows:
[0030]
[0031] h is the high-dimensional spatial feature obtained by performing a random Fourier transform on the input low-dimensional feature x, ω is a random variable sampled from a standard normal distribution, and ωx represents the transformation of the random variable ω by multiplying it with x. is a random variable sampled from a uniform distribution;
[0032] Step S42 obtains the optimal value w of the sample weight by calculating the minimum value of the covariance between the two random variables. * , the formula is as follows:
[0033] w * =argmin w∈Δn ∑ 1≤i<j≤n cov(w i X i ,w j X j ),
[0034] In the formula n is the number of input batches, that is, the number of sample feature maps passed in, X i and X j are the feature maps of different samples in the sample space, w i and w j They are samples X i and X j The corresponding weighted weights;
[0035] Step S43 Deep learning uses all samples to globally learn sample weights and features, which requires huge overhead, so it is necessary to store and reload sample weights through a learnable parameter α i To update the global weights and features, the weight update formula is as follows:
[0036] X′ Gi =α i X Gi +(1-α i )X L ,
[0037] W′ Gi =α i W Gi +(1-α i )W L ,
[0038] X Gi and X L are the global sample features and the current sample features respectively, W Gi and W L are the global sample weight and the current sample weight respectively;
[0039] Step S44 multiplies the calculated optimal weight by the loss value of the sample to obtain a new loss for model training. The loss update formula is as follows:
[0040] loss=SoftDiceLoss(SR,GT).view(1,-1).mm(w * ).view(1),
[0041]
[0042] SR and GT are the predicted value and true value given by the model, respectively.
[0043] Preferably, the operation steps of the counterfactual explanation module of the present invention include:
[0044] Step S51 generates segmented segments of the image using a fast displacement image segmentation algorithm
[0045] Step S52 masks the segmented segments to find a set of irreducible segments that can reduce the IoU score of the model the most. The formula is as follows:
[0046]
[0047] T(I\S) <T(I)(IoU reduce),
[0048] Where I represents the segmented segment of the generated image, S represents the over-segmented segment, and T represents the segmentation model;
[0049] Step S53 maps this set of segments to the original image, generating an instance-level explanation of the model results.
[0050] The present invention preprocesses lymphoma ultrasound scanning image data to obtain two data sets for model training, so that the model can obtain higher performance; the present invention automatically performs semantic segmentation on lymphoma ultrasound images based on a deep learning framework, solves the problems of ultrasound image boundary blur and generalization through a self-attention module and a stable learning module, and interprets the model at the instance level through a counterfactual interpretation module, thereby improving the segmentation accuracy and credibility of the model; automatic segmentation does not require the intervention of human factors, eliminates the interference of subjective factors, saves a lot of time and labor costs, and improves the efficiency and accuracy of diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 It is a schematic diagram of the process framework of the present invention;
[0052] Figure 2 It is a flow chart of the data preprocessing module in the present invention; data preprocessing module, self-attention mechanism module, stable learning module, counterfactual explanation module;
[0053] Figure 3 is a schematic diagram of a sample structure of a standard data set of the present invention;
[0054] Figure 4 This is a schematic diagram of the neural network architecture of the present invention;
[0055] Figure 5 This is a schematic diagram of the feature extraction network module architecture in the present invention;
[0056] Figure 6 Schematic diagram of the self-attention mechanism module architecture in the present invention;
[0057] Figure 7 This is a schematic diagram of the stable learning module architecture in the present invention;
[0058] Figure 8 Schematic diagram of the architecture of the counterfactual explanation module in the present invention;
[0059] Fig. 9 Schematic diagram of the process of mapping IOU from 0.89105 to 0.84559;
[0060] Fig.10 Schematic diagram of the process of mapping IOU 0.95403 to 0.91483. DETAILED DESCRIPTION
[0061] A semantic segmentation method for lymphoma ultrasound images based on self-attention mechanism and stable learning, comprising:
[0062] Obtaining lymphoma ultrasound images and processing the images as training samples;
[0063] Constructing a lymphoma ultrasound image segmentation network: the lymphoma ultrasound image segmentation network includes a data preprocessing module, a feature extraction network module, a self-attention mechanism module, a stable learning module, and a counterfactual explanation module;
[0064] The data preprocessing module is used to crop and annotate all lymphoma ultrasound image samples and use the preprocessed samples as input data for the segmentation network;
[0065] Feature extraction network module, used to extract spatial and global information of the image, and fuse the extracted features to capture clearer object boundaries;
[0066] The self-attention mechanism module is used to realize non-local interaction between features and alleviate the information decay problem caused by multiple sampling;
[0067] The stable learning module is used to eliminate the dependency between environmental features and essential features by random Fourier features and weighting samples to achieve generalization;
[0068] The counterfactual explanation module is used to provide instance-level explanations of model results to improve the credibility of the model.
[0069] like Figure 1 As shown, the process framework of the present invention is mainly composed of data preprocessing, deep neural network and counterfactuals. First, the patient's lymphoma ultrasound image is used as the original data set. After preprocessing the original data set, a data set for neural network training is obtained; the trained neural network can give segmentation results for the input image; the counterfactual interpretation method combines the deep neural network to give instance-level interpretation.
[0070] like Figure 2 As shown, a preprocessing module based on the patient's lymphoma ultrasound image is implemented, and the operation steps include:
[0071] Step S11: cropping the ultrasound image scanned by the ultrasound instrument to remove the extra content added by the ultrasound scanning instrument;
[0072] Step S12 uploads the cropped image to the PLabel labeling system for free labeling. After the labeling is completed, the image labeled in the system is submitted to experienced medical experts for review and correction, and the final labeling result is exported;
[0073] Step S13 reads the exported json file to generate a real mask image;
[0074] Step S14 adjusts the pixel spacing between the annotated image and the original image to obtain an image with a resolution of 256×256, crops the image of the region of interest, and adjusts the resolution to 512×512.
[0075] For the preprocessed data set, the training set and the test set are divided. The test set is used for training the neural network, and the test set is used to evaluate the accuracy of the model. Figure 3 , you can see the preprocessed images. The stage1 images are used for the first stage training of the deep neural network to roughly locate the lesion area; the stage2 images are the region of interest images, which are used for the second stage training of the deep neural network to optimize the target boundary of the segmented object.
[0076] like Figure 4 , Figure 5As shown in the figure, a feature extraction network module based on the patient's lymphoma ultrasound image is implemented. The module extracts information through the encoder hole convolution and restores the spatial information through the decoder sampling to obtain the image segmentation result. The operation steps include:
[0077] Step S21 uses the encoder structure to perform downsampling to extract low-level features, thereby using the extracted spatial information and global information for accurate segmentation;
[0078] In step S22, the decoder gradually restores the spatial information by upsampling and fuses the features extracted by the encoder during the encoding process to capture clearer object boundaries.
[0079] like Figure 6 As shown in the figure, a self-attention mechanism module based on the patient's lymphoma ultrasound image is implemented. The module adds position embedding to the patient's lymphoma image features and obtains the attention-weighted feature map through the self-attention mechanism. The operation steps include:
[0080] Step S32 inputs the last layer of features encoded by the encoder into the TSA module. The TSA module performs different linear changes on the input feature map and the generated position embedding vector to generate three vectors: query (Q), key (K), and value (V) for attention calculation.
[0081] Step S32: Multiply the transpose of Q and K to obtain the similarity between the elements of Q and K, and divide it by the dimension d of vector Q. k The square root of ensures the gradient of softmax and normalizes it through softmax to get a context attention map A. Multiply A and V to get the weighted value of attention. The formula is as follows:
[0082]
[0083] Step S33 adds the return value obtained by the attention mechanism module to the last layer feature element by element to obtain a fused feature map F for the decoder input.
[0084] like Figure 7 As shown, a stable learning module based on the patient's lymphoma ultrasound image is implemented. The module obtains the sample weights for updating the loss function by performing random Fourier transform RFF and weighted decorrelation LSWD on the patient's lymphoma ultrasound image sample features. The operation steps include:
[0085] Step S41 inputs the feature map into the stable learning module, maps the input features from low-dimensional space to high-dimensional space through random Fourier features, and eliminates the correlation between features. The random Fourier feature formula is as follows:
[0086]
[0087] h is the high-dimensional spatial feature obtained by performing a random Fourier transform on the input low-dimensional feature x, ω is a random variable sampled from a standard normal distribution, and ωx represents the transformation of the random variable ω by multiplying it with x. is a random variable sampled from a uniform distribution;
[0088] Step S42 obtains the optimal value w of the sample weight by calculating the minimum value of the covariance between the two random variables. * , the formula is as follows:
[0089] w * =argmin w∈Δn ∑ 1≤i<j≤n cov(w i X i ,w j X j ),
[0090] In the formula n is the number of input batches, that is, the number of sample feature maps passed in, X i and X j are the feature maps of different samples in the sample space, w i and w j They are samples X i and X j The corresponding weighted weights;
[0091] Step S43 Deep learning uses all samples to globally learn sample weights and features, which requires huge overhead, so it is necessary to store and reload sample weights through a learnable parameter α i To update the global weights and features, the weight update formula is as follows:
[0092] X′ Gi =α i X Gi +(1-α i )X L ,
[0093] W′ Gi =α i W Gi +(1-α i )W L ,
[0094] X Gi and X L are the global sample features and the current sample features respectively, W Gi and W L are the global sample weight and the current sample weight respectively;
[0095] Step S44 multiplies the calculated optimal weight by the loss value of the sample to obtain a new loss for model training. The loss update formula is as follows:
[0096] loss=SoftDiceLoss(SR,GT).view(1,-1).mm(w * ).view(1),
[0097]
[0098] SR and GT are the predicted value and true value given by the model, respectively.
[0099] like Figure 8 As shown in the figure, a counterfactual explanation module based on the patient's lymphoma ultrasound image is implemented. The module processes the patient's lymphoma ultrasound image through the fast displacement image segmentation algorithm SETC and the Counterfactual module to generate instance-level counterfactual explanations. The running steps include:
[0100] Step S51 generates segmented segments of the image using a fast displacement image segmentation algorithm
[0101] Step S52 masks the segmented segments to find a set of irreducible segments that can reduce the IoU score of the model the most. The formula is as follows:
[0102]
[0103] T(I\S) <T(I)(IoU reduce),
[0104] Where I represents the segmented segment of the generated image, S represents the over-segmented segment, and T represents the segmentation model;
[0105] Step S53 maps this set of segments to the original image, generating an instance-level explanation of the model results.
[0106] like Fig. 9 , Fig.10 As shown, Fig. 9 This is a schematic diagram of the process of mapping IOU from 0.89105 to 0.84559. Fig. 9 This is a schematic diagram of the process of mapping IOU 0.95403 to 0.91483. You can see the counterfactual explanation results generated by the counterfactual module. Fig. 9 and Fig.10 In the figure, a is the input original image; b is the visualization of the segmented fragments generated by the SETC algorithm; c is the counterfactual explanation generated by the counterfactual module; d is the mapping of the counterfactual explanation on the original image.
[0107] The experimental results show that the present invention can efficiently utilize the patient's lymphoma ultrasound images and effectively improve the segmentation accuracy of lymphoma patients. The model can solve the boundary fuzziness and generalization problems of ultrasound images. At the same time, the model is explained at the instance level through counterfactual interpretation. The obtained segmentation results are more convincing for doctors' auxiliary diagnosis.
Claims
1. A semantic segmentation method for lymphoma ultrasound images based on self-attention mechanism and stable learning, characterized by include: Obtaining lymphoma ultrasound images and processing the images as training samples; Constructing a lymphoma ultrasound image segmentation network: the lymphoma ultrasound image segmentation network includes a data preprocessing module, a feature extraction network module, a self-attention mechanism module, a stable learning module, and a counterfactual explanation module; The self-attention mechanism module has the following specific operation steps: The last layer of features encoded by the encoder is input into the TSA module. The TSA module performs different linear transformations on the input feature map and the generated position embedding vector to generate three vectors: query (Q), key (K), and value (V) for attention calculation. Multiply the transpose of Q and K to get the similarity between the elements of Q and K, and divide it by the dimension d of vector Q k The square root of ensures the gradient of softmax and normalizes it through softmax to get a context attention map A. Multiply A and V to get the weighted value of attention. The formula is as follows: Add the return value of the attention mechanism module to the last layer feature element by element to obtain a fused feature map F for the decoder input; Stable learning module, the specific operation steps include: The feature map is input into the stable learning module, and the input features are mapped from low dimensions to high dimensions through random Fourier features to eliminate the correlation between features. The random Fourier feature formula is as follows: h is the high-dimensional spatial feature obtained by performing a random Fourier transform on the input low-dimensional feature x, ω is a random variable sampled from a standard normal distribution, and ωx represents the transformation of the random variable ω by multiplying it with x. is a random variable sampled from a uniform distribution; The optimal value w of the sample weight is obtained by calculating the minimum covariance between the two random variables * , the formula is as follows: In the formula n is the number of input batches, that is, the number of sample feature maps passed in, X i and X j are the feature maps of different samples in the sample space, w i and w j They are samples X i and X j The corresponding weighted weights; Through a learnable parameter α i To update the global weights and features, the weight update formula is as follows: X′ Gi =a i X Gi +(1-a i )X L , W′ Gi =a i W Gi +(1-a i )W L , X Gi and X L are the global sample features and the current sample features respectively, W Gi and W L are the global sample weight and the current sample weight respectively; Multiply the calculated optimal weight by the loss value of the sample to get the new loss for model training. The loss update formula is as follows: loss=SoftDiceLoss(SR,GT).view(1,-1).mm(w * ).view(1), SR and GT are the predicted value and true value given by the model respectively; The specific operation steps of the counterfactual explanation module include: Generate segmented fragments of the image using the fast displacement image segmentation algorithm By masking the segmented fragments, we find a set of irreducible fragments that can reduce the IoU score of the model the most. The formula is as follows: T(I\S) <T(I)(IoU reduce), I represents the segmented fragment of the generated image, S represents the over-segmented fragment, and T represents the segmentation model; This set of snippets is mapped to the original image to generate instance-level explanations of the model results.
2. The method for semantic segmentation of lymphoma ultrasound images based on self-attention mechanism and stable learning according to claim 1 is characterized in that The operation steps of the above data preprocessing module include: Step S11: cropping the ultrasound image scanned by the ultrasound instrument to remove the extra content added by the ultrasound scanning instrument; Step S12 uploads the cropped image to the PLabel labeling system for free labeling. After the labeling is completed, the image labeled in the system is submitted to experienced medical experts for review and correction, and the final labeling result is exported; Step S13 reads the exported json file to generate a real mask image; Step S14 adjusts the pixel spacing between the annotated image and the original image to obtain an image with a resolution of 256×256, crops the image of the region of interest, and adjusts the resolution to 512×512.
3. The method for semantic segmentation of lymphoma ultrasound images based on self-attention mechanism and stable learning according to claim 2 is characterized in that The above feature extraction network module operation steps include: Step S21 uses the encoder structure to perform downsampling to extract low-level features, thereby using the extracted spatial information and global information for accurate segmentation; In step S22, the decoder gradually restores the spatial information by upsampling and fuses the features extracted by the encoder during the encoding process to capture clearer object boundaries.