Breast cancer lesion localization method and device
By combining feature extraction and fusion processing of multiple ultrasound image modalities, the problem of low localization accuracy of single-modality images is solved, and high-precision localization of breast cancer lesions is achieved, especially the accurate segmentation of lesions with complex boundaries and small scale.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FOSHAN UNIVERSITY
- Filing Date
- 2026-04-09
- Publication Date
- 2026-07-14
AI Technical Summary
In existing technologies, when using single-modal images to locate breast cancer lesions, it is impossible to fully capture key features such as morphology, blood flow, and tissue stiffness, resulting in low positioning accuracy.
By combining B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images, multimodal joint features are generated through feature extraction and fusion processing. Improved segmentation model, cross-scale adaptive fusion mechanism, and dual-path attention enhancement mechanism are used to improve lesion segmentation accuracy. Weighted summation of multimodal features is achieved through gating network to obtain the location information of breast cancer lesions.
It achieves comprehensive capture of breast cancer lesions in terms of morphology, blood flow, and tissue stiffness, significantly improving the positioning accuracy, especially the segmentation ability of complex boundaries and small-scale lesions.
Smart Images
Figure CN122391100A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and in particular to a method and apparatus for locating breast cancer lesions. Background Technology
[0002] Breast cancer is one of the most common malignant tumors in women, and early screening is crucial for improving patient survival and prognosis. Related technologies utilize deep learning algorithms such as convolutional neural networks to locate breast cancer lesions. However, these technologies often only target a single modality of image, failing to comprehensively capture key features of breast cancer lesions in terms of morphology, blood flow, and tissue stiffness, resulting in low accuracy in lesion localization. Summary of the Invention
[0003] This application provides a method and apparatus for locating breast cancer lesions, which can effectively improve the accuracy of breast cancer lesion location.
[0004] On one hand, embodiments of this application provide a method for locating breast cancer lesions, including:
[0005] Acquire B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images containing breast cancer lesions;
[0006] Feature extraction processing is performed on the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image respectively to obtain the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image respectively;
[0007] The lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image are fused to obtain multimodal joint features;
[0008] Based on the multimodal joint features, the location information of the breast cancer lesion is obtained from any one of the B-mode ultrasound image, the color Doppler blood flow image, or the ultrasound elastography image.
[0009] Further, in one embodiment, the step of performing feature extraction processing on the B-mode ultrasound image, the color Doppler flow image, and the ultrasound elastography image respectively to obtain the lesion features corresponding to the B-mode ultrasound image, the color Doppler flow image, and the ultrasound elastography image respectively includes:
[0010] The target image is segmented to obtain a segmented image of the lesion;
[0011] The lesion region in the lesion segmentation image is extracted to obtain the lesion region image;
[0012] The lesion region image is subjected to feature extraction processing to obtain the lesion features corresponding to the target image;
[0013] The target image is any one of the B-mode ultrasound image, the color Doppler blood flow image, or the ultrasound elastography image.
[0014] Further, in one embodiment, the segmentation process of the target image to obtain a lesion segmentation image includes:
[0015] The target image is input into the improved segmentation model for segmentation processing to obtain the lesion segmentation image;
[0016] The improved segmentation model includes:
[0017] An encoder structure includes multiple encoders connected in sequence; each encoder is used to perform feature extraction and downsampling processing on the input of the encoder to obtain the output of the encoder;
[0018] The decoder structure includes multiple decoders connected sequentially, with a one-to-one correspondence between the multiple encoders and the multiple decoders. Each decoder performs cross-scale fusion skip connections with its corresponding encoder. The cross-scale fusion skip connections are used to fuse the encoded features of multiple scales in the output of the encoder to obtain multi-scale features, and to perform enhancement processing based on a multi-attention mechanism on the multi-scale features to obtain the skip connection output of the encoder. Each decoder is used to upsample the skip connection output of its corresponding encoder and the output of the previous decoder to obtain the output of the decoder.
[0019] The output layer is used to process the output of the last decoder to obtain the lesion segmentation image.
[0020] Furthermore, in one embodiment, the fusion processing of the encoded features at multiple scales in the output of the encoder to obtain multi-scale features includes:
[0021] The encoded features of each scale are scale aligned to obtain the aligned features of each scale.
[0022] By utilizing the importance weights of each scale, the alignment features of each scale are fused to obtain the multi-scale features.
[0023] Furthermore, in one embodiment, the enhancement processing of the multi-scale features based on a multi-attention mechanism to obtain the skip connection output includes:
[0024] The multi-scale features are subjected to multiple attention processing steps to obtain multiple initial attention weights;
[0025] The target attention weight is obtained by fusing the multiple initial attention weights.
[0026] The multi-scale features are enhanced using the target attention weights to obtain the skip connection output.
[0027] Further, in one embodiment, the step of extracting the lesion region from the lesion segmentation image to obtain a lesion region image includes:
[0028] Extract the minimum bounding rectangle of the lesion region from the lesion segmentation image;
[0029] Based on the segmented image of the lesion, calculate the morphological feature parameters of the lesion region;
[0030] Based on the morphological feature parameters, the minimum bounding rectangle is adjusted to generate the image of the lesion region.
[0031] Further, in one embodiment, the fusion processing of the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image to obtain multimodal joint features includes:
[0032] Gating processing is performed using the prior description vectors corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively, to obtain the target gating values corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively.
[0033] Using the target gating values corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively, the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image are fused to obtain the multimodal joint features.
[0034] Further, in one embodiment, the step of performing gating processing using the prior description vectors corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively, to obtain the target gating values corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, includes:
[0035] The lesion features and prior description vectors corresponding to the target image are input into the gating network to obtain the initial gating value;
[0036] Modality competition correction is performed on the initial gate value to obtain the target gate value corresponding to the target image;
[0037] The target image is any one of the B-mode ultrasound image, the color Doppler blood flow image, or the ultrasound elastography image.
[0038] Further, in one embodiment, the fusion processing of the lesion features corresponding to the B-mode ultrasound image, the color Doppler flow image, and the ultrasound elastography image, respectively, using the target gating values corresponding to the B-mode ultrasound image, the color Doppler flow image, and the ultrasound elastography image, to obtain the multimodal joint features, includes:
[0039] Using the target gating values corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively, the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image are weighted and summed to obtain the multimodal joint features.
[0040] On the other hand, embodiments of this application provide a breast cancer lesion localization device, including:
[0041] The acquisition module is used to acquire B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images containing breast cancer lesions;
[0042] The first processing module is used to perform feature extraction processing on the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image respectively, to obtain the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image respectively;
[0043] The second processing module is used to fuse the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image and the ultrasound elastography image respectively to obtain multimodal joint features.
[0044] The third processing module is used to perform identification processing based on the multimodal joint features to obtain the location information of the breast cancer lesion in any one of the B-mode ultrasound image, the color Doppler blood flow image, or the ultrasound elastography image.
[0045] According to the embodiments of this application, a method and apparatus for locating breast cancer lesions are first performed on each modality image to obtain the lesion features corresponding to each modality image. Then, the lesion features corresponding to each modality image are fused to fully explore the complementary information between different modalities in terms of morphology, blood flow, tissue stiffness, etc., and to make up for the missing feature information of each modality to obtain multimodal joint features. Finally, recognition processing is performed based on the multimodal joint features to obtain the location information of the breast cancer lesion in any modality image, thereby realizing the location processing of breast cancer lesions. In this way, the key features of breast cancer lesions in terms of morphology, blood flow, tissue stiffness, etc. can be comprehensively captured, thereby effectively improving the location accuracy of breast cancer lesions. Attached Figure Description
[0046] Figure 1 This is a flowchart of a method for locating breast cancer lesions provided in this application;
[0047] Figure 2 This is a schematic diagram of a method for locating breast cancer lesions provided in this application;
[0048] Figure 3 This is a structural diagram of a breast cancer lesion localization device provided in this application. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0050] The present application will be further described below with reference to the accompanying drawings and specific embodiments. The described embodiments should not be considered as limitations on the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.
[0051] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0053] To address the problems and deficiencies of related technologies, this application provides a method and apparatus for locating breast cancer lesions. It incorporates multimodal images, including B-mode ultrasound (Brightness-Mode Ultrasound) images, color Doppler flow imaging (CDFI) images, and ultrasound elastography (UE) images, to accurately locate breast cancer lesions. This approach comprehensively captures key features of breast cancer lesions in terms of morphology, blood flow, and tissue stiffness, thereby effectively improving the accuracy of breast cancer lesion location.
[0054] It should be noted that in various specific embodiments of this application, when processing is required based on data such as B-mode ultrasound images, color Doppler flow images, and ultrasound elastography images, the permission or consent of the subject will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require the acquisition of data such as B-mode ultrasound images, color Doppler flow images, and ultrasound elastography images, separate permission or consent from the subject will be obtained through pop-up windows or redirection to a confirmation page. Only after obtaining the subject's separate permission or consent will the necessary data such as B-mode ultrasound images, color Doppler flow images, and ultrasound elastography images for the proper functioning of the embodiments of this application be acquired.
[0055] It is important to emphasize that the B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images involved in the embodiments of this application are image data pre-stored in a database, and not image data acquired in real time from the patient. That is, the embodiments of this application do not use living human bodies as the object. Furthermore, the final output of the embodiments of this application is the lesion area in the image, which is provided to doctors for diagnostic reference, and is not the final diagnostic result. The final diagnosis of breast cancer still needs to be judged and decided by a doctor. In other words, the embodiments of this application do not have the direct purpose of obtaining a breast cancer diagnosis result. Therefore, the embodiments of this application do not belong to the methods of diagnosing and treating diseases.
[0056] The embodiments of this application will be described in detail below.
[0057] Reference Figure 1 The method for locating breast cancer lesions provided in this application includes the following steps S101-S104.
[0058] S101, acquires B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images containing breast cancer lesions.
[0059] In this step, multimodal images are acquired, including B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images. Each modality contains the breast cancer lesion to be located. Specifically, B-mode ultrasound images are used to characterize the morphological and structural information of the breast cancer lesion, color Doppler blood flow images are used to characterize the blood flow distribution information of the breast cancer lesion, and ultrasound elastography images are used to characterize the tissue stiffness information of the breast cancer lesion. The modalities are complementary.
[0060] S102, feature extraction processing is performed on B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images respectively to obtain the lesion features corresponding to the B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images respectively.
[0061] In this step, feature extraction processing is performed on each modal image to extract the morphological and structural features of breast cancer lesions from B-mode ultrasound images, the blood flow distribution features of breast cancer lesions from color Doppler blood flow images, and the tissue stiffness features of breast cancer lesions from ultrasound elastography images, thereby obtaining the lesion features corresponding to each modal image. This allows for the preliminary capture of key features associated with breast cancer lesions.
[0062] Optionally, before feature extraction, each modal image can be preprocessed, such as size standardization, grayscale normalization, and necessary data validity checks, to eliminate scale and grayscale distribution differences caused by different devices and acquisition conditions, thereby improving the stability and robustness of subsequent multimodal fusion.
[0063] S103, the lesion features corresponding to B-mode ultrasound images, color Doppler blood flow images and ultrasound elastography images are fused to obtain multimodal joint features.
[0064] In this step, after feature extraction is completed, the lesion features corresponding to each modality image are fused and interacted. The aim is to fully explore the complementary information between different modalities in terms of morphology, blood flow, tissue stiffness, etc., and to make up for the missing feature information of each modality, so as to obtain multimodal joint features. This can further capture the implicit features associated with breast cancer lesions and enrich their key features.
[0065] S104, based on multimodal joint feature recognition processing, obtains the location information of breast cancer lesions in any one of B-mode ultrasound images, color Doppler blood flow images, or ultrasound elastography images.
[0066] In this step, the multimodal joint features are input into the classification prediction layer for recognition processing, thereby obtaining the location information of the breast cancer lesion in any one of the following: B-mode ultrasound image, color Doppler blood flow image, or ultrasound elastography image. The classification prediction layer can be flexibly configured according to the actual situation; for example, it can be a multilayer perceptron or a Transformer, but it is not limited to these.
[0067] It is worth noting that during training, a first dataset, a second dataset, or a third dataset can be used. The first dataset includes several sample images and corresponding first label information for each sample image. The sample images refer to B-mode ultrasound images, color Doppler flow images, and ultrasound elastography images containing breast cancer lesions, and the first label information contains the location information of the breast cancer lesion in the B-mode ultrasound image. The second dataset includes several sample images and corresponding second label information for each sample image. The second label information contains the location information of the breast cancer lesion in the color Doppler flow image. The third dataset includes several sample images and corresponding third label information for each sample image. The third label information contains the location information of the breast cancer lesion in the ultrasound elastography image. Therefore, in testing or practical applications, if the first dataset is used during training, the final output location information is the location information of the breast cancer lesion in the B-mode ultrasound image; if the second dataset is used, the final output location information is the location information of the breast cancer lesion in the color Doppler flow image; and if the third dataset is used, the final output location information is the location information of the breast cancer lesion in the ultrasound elastography image.
[0068] Therefore, the embodiments of this application first extract features from each modal image to obtain the lesion features corresponding to each modal image. Then, the lesion features corresponding to each modal image are fused to fully explore the complementary information between different modalities in terms of morphology, blood flow, and tissue stiffness, and to make up for the missing feature information of each modality, so as to obtain multimodal joint features. Finally, based on the multimodal joint features, recognition processing is performed to obtain the location information of the breast cancer lesion in any modal image, thereby realizing the localization processing of breast cancer lesions. In this way, the key features of breast cancer lesions in terms of morphology, blood flow, and tissue stiffness can be comprehensively captured, thereby effectively improving the localization accuracy of breast cancer lesions.
[0069] The following will combine Figure 2 The above steps will be further explained.
[0070] In some embodiments, step S102 includes steps S201-S203, wherein the target image is any one of a B-mode ultrasound image, a color Doppler blood flow image, or an ultrasound elastography image.
[0071] S201, Perform segmentation processing on the target image to obtain a lesion segmentation image.
[0072] In this step, the target image is first segmented to preliminarily segment the lesion region in the target image so that key features associated with breast cancer lesions can be accurately captured in subsequent steps.
[0073] Furthermore, in some embodiments, step S201 above includes:
[0074] The target image is input into the improved segmentation model for segmentation processing to obtain a lesion segmentation image;
[0075] The improved segmentation model includes:
[0076] The encoder structure includes multiple encoders connected in sequence; each encoder is used to extract features and downsample the input of the encoder to obtain the output of the encoder.
[0077] The decoder structure includes multiple decoders connected sequentially, with multiple encoders corresponding one-to-one with multiple decoders. Each decoder performs cross-scale fusion skip connections with its corresponding encoder. The cross-scale fusion skip connections are used to fuse the encoded features of multiple scales in the encoder output to obtain multi-scale features, and then perform enhancement processing based on a multi-attention mechanism on the multi-scale features to obtain the skip connection output of the encoder. Each decoder is used to upsample the skip connection output of its corresponding encoder and the output of the previous decoder to obtain the decoder output.
[0078] The output layer is used to process the output of the last decoder to obtain the lesion segmentation image.
[0079] Furthermore, in some embodiments, the above-described fusion processing of multiple scale encoded features in the encoder output yields multi-scale features, including:
[0080] The encoded features at each scale are scale aligned to obtain the aligned features at each scale.
[0081] By utilizing the importance weights of each scale, the alignment features of each scale are fused to obtain multi-scale features.
[0082] Furthermore, in some embodiments, the above-described enhancement processing of multi-scale features based on a multi-attention mechanism to obtain a skip connection output includes:
[0083] Multiple attention processes are performed on the multi-scale features to obtain multiple initial attention weights;
[0084] Multiple initial attention weights are fused together to obtain the target attention weight;
[0085] The multi-scale features are enhanced using target attention weights to obtain the skip connection output.
[0086] Here, an improved segmentation model is used to segment the lesion region from the target image, thereby obtaining the lesion segmentation image. This improved segmentation model introduces a cross-scale adaptive fusion mechanism and a dual-path attention enhancement mechanism on the basis of the U-Net model to improve its segmentation ability for complex boundaries, small-scale lesions and low-contrast regions.
[0087] Specifically, the improved segmentation model comprises three parts: an encoder structure, a decoder structure, and an output layer, wherein:
[0088] The encoder structure comprises multiple encoders connected sequentially. The input to the first encoder is the target image, and the output of the previous encoder becomes the input to the next. Each encoder performs feature extraction and downsampling on the input, specifically through multiple convolution operations, batch normalization, and activation processing. The spatial resolution of the feature map is progressively reduced through downsampling, thus yielding the encoder's output. The number of convolution operations can be flexibly set according to the actual situation; for example, two convolution operations are possible, but not limited to this. Furthermore, the activation function used for activation processing can be flexibly set according to the actual situation; for example, the activation function could be the ReLU function, but not limited to this.
[0089] The decoder structure includes multiple decoders connected sequentially, with one-to-one correspondence between multiple encoders and multiple decoders. Each decoder performs cross-scale fusion skip connections with its corresponding encoder. These cross-scale fusion skip connections are used to fuse the encoded features of multiple scales in the encoder output to obtain multi-scale features. The multi-scale features are then enhanced using a multi-attention mechanism to obtain the skip connection output of the encoder. Each decoder is used to upsample the skip connection output of its corresponding encoder and the output of the previous decoder. Specifically, the skip connection output of its corresponding encoder and the output of the previous decoder are summed element-wise, and the element-wise sum is convolved to obtain fused features. Subsequently, the fused features are upsampled, for example, by bilinear interpolation or transposed convolution, to gradually restore the spatial resolution of the feature map, thereby obtaining the decoder output.
[0090] The output layer receives the output of the last decoder, performs a 1×1 convolution operation on the output, and then performs a sigmoid activation function operation to obtain the lesion segmentation image.
[0091] Optionally, the number of decoders and encoders can be flexibly set according to the actual situation, for example, the number of decoders and encoders can both be four, but it is not limited to this.
[0092] More specifically, to enhance the model's ability to identify complex boundaries, small-scale lesions, and low-contrast regions, a cross-scale adaptive fusion mechanism and a dual-path attention enhancement mechanism are introduced into the skip connection between the encoder and decoder, forming a cross-scale fusion skip connection.
[0093] For the cross-scale adaptive fusion mechanism, the encoder output includes encoded features from multiple scales. First, all encoded features are unified to the same spatial size through a scale alignment function, as shown in the following formula (1):
[0094] (1);
[0095] In equation (1), Indicates the first Alignment features at various scales; Indicates the first Encoding features at each scale; This indicates a scale alignment operation, such as an upsampling operation or a downsampling operation.
[0096] Then, the feature importance weights at each scale are calculated using the weight generation function, as shown in formula (2):
[0097] (2);
[0098] In equation (2), Indicates the first The importance weights of each scale are equal to the sum of the importance weights of all scales. This indicates a weight generation operation, such as one consisting of global average pooling and a fully connected layer; Indicates the number of scales.
[0099] Finally, by using the feature importance weights of each scale, the alignment features of each scale are weighted and summed to obtain the multi-scale features, as shown in the following formula (3):
[0100] (3);
[0101] In equation (3), Represents multi-scale features; This indicates element-wise multiplication.
[0102] Therefore, in the traditional U-Net model, before passing the encoder's output to the decoder via skip connections, the features at different scales of the encoder's output are simply concatenated, ignoring the contextual information provided by different scales. This results in low sensitivity of the model to complex features such as complex boundaries, small-scale lesions, and low-contrast regions, thus affecting the segmentation performance. To address this, this embodiment introduces a cross-scale adaptive fusion mechanism, which dynamically assigns weights to features at different scales based on their feature content. This allows the model to assign higher weights to high-value features when facing complex features such as complex boundaries, small-scale lesions, and low-contrast regions, preserving fine boundary details, while simultaneously fusing the global shape constraints provided by lower-value features. This results in segmented lesion regions that are both accurate and continuous, thereby effectively improving segmentation accuracy.
[0103] The dual-path attention enhancement mechanism aims to enhance the model's response to key lesion regions. First, multiple attention processing is performed on the multi-scale features to obtain multiple initial attention weights. Then, the multiple initial attention weights are broadcast-multiplied to obtain the target attention weight. Finally, the element-wise multiplication result of the target attention weight and the multi-scale features is determined as the skip connection output. The attention mechanism can be flexibly set according to the actual situation. In one example, the attention mechanism can be a channel attention mechanism and a spatial attention mechanism. That is, channel attention processing is performed on the multi-scale features to obtain channel attention weights, as shown in formula (4) below, and spatial attention processing is performed on the multi-scale features to obtain spatial attention weights, as shown in formula (5) below.
[0104] (4);
[0105] In equation (4), Indicates channel attention weights; This represents the global average pooling operation, used to compress feature maps into channel description vectors; and For learnable parameter matrices (fully connected layer); The activation function can be, for example, the ReLU activation function, but is not limited to this; This represents the Sigmoid function.
[0106] (5);
[0107] In equation (5), Indicates spatial attention weights; This represents the average pooling operation along the channel dimension; This represents a max pooling operation along the channel dimension; This indicates a channel splicing operation; This indicates a convolution operation.
[0108] Then attention fusion and feature modulation are performed, as shown in the following formula (6):
[0109] , (6);
[0110] In equation (6), Indicates the target attention weight; This indicates the encoder's skip connection output; This represents element-wise multiplication; This indicates broadcast multiplication.
[0111] Therefore, it is evident that in the traditional U-Net model, the encoder output is passed to the decoder via skip connections without feature enhancement. This results in low sensitivity to key lesion regions, thus affecting segmentation performance. To address this, this embodiment introduces a dual-path attention enhancement mechanism. This mechanism learns differentiated attention patterns for feature maps of different resolutions. For example, for low-resolution, high-semantic features, channel attention contributes more, focusing on key semantic channels; while for high-resolution, low-semantic features, spatial attention contributes more, focusing on locating detailed regions. The learned attention patterns are then arbitrated and fused to form the final high-precision attention weights, which are then used for feature enhancement. This enhances the model's responsiveness to key lesion regions, effectively improving segmentation accuracy.
[0112] Through the two mechanisms mentioned above, the improved segmentation model can adaptively enhance the lesion-related region before feature fusion while suppressing irrelevant background responses, thereby effectively improving segmentation accuracy.
[0113] Optionally, considering that the breast lesion area accounts for a small proportion of the entire ultrasound image, in order to avoid the background dominating the training process of the improved segmentation model due to the use of only cross-entropy loss, this embodiment preferably adopts a combined segmentation loss function of cross-entropy loss and Dice loss. This combined loss takes into account both pixel-level classification ability and region-level overlap optimization effect, which is beneficial to improving the segmentation stability of small lesion areas and blurred boundary areas, as shown in the following formula (7):
[0114] (7);
[0115] In equation (7), Represents the combined segmentation loss function; Represents the cross-entropy loss function; and This represents the weight value, which can be flexibly set according to the actual situation; The Dice loss function is represented by the following formula (8):
[0116] (8);
[0117] In equation (8), Indicates the predicted segmentation region; This indicates the actual labeled area.
[0118] S202, extract the lesion region from the lesion segmentation image to obtain the lesion region image.
[0119] In this step, based on the segmented lesion image, the lesion region is further extracted to generate a lesion region image.
[0120] Furthermore, in some embodiments, step S202 above includes:
[0121] Extract the minimum bounding rectangle of the lesion region from the lesion segmentation image;
[0122] Based on the segmented lesion image, calculate the morphological feature parameters of the lesion region;
[0123] Based on morphological feature parameters, the minimum bounding rectangle is adjusted to generate an image of the lesion area.
[0124] Here, after obtaining the lesion segmentation image (segmentation mask), this embodiment proposes a Dynamic Border Expansion Algorithm (DBEA) to construct the Region of Interest (ROI) in order to achieve a balance between lesion focusing and contextual information preservation. The specific process is as follows:
[0125] First, the minimum bounding rectangle of the lesion region is extracted based on the segmentation mask, as shown in the following formula (9):
[0126] , , (9);
[0127] In equation (9), This represents the smallest bounding rectangle. The width of the minimum bounding rectangle. It is the height of the smallest bounding rectangle.
[0128] Then, based on the segmentation mask, the morphological feature parameters of the lesion region are calculated. The morphological feature parameters include the area and perimeter of the lesion region, and the shape complexity index is defined accordingly, as shown in the following formula (10):
[0129] (10);
[0130] In equation (10), The shape complexity index is used to measure the degree of irregularity of the boundary of the lesion area; Indicates the perimeter of the lesion area; This indicates the area of the lesion.
[0131] Then, based on the shape complexity index, the boundary expansion coefficient of the lesion area is calculated as shown in the following formula (11):
[0132] (11);
[0133] In equation (11), Indicates the boundary spread coefficient; Indicates the basic expansion ratio; and This indicates the preset adjustment parameters.
[0134] Finally, based on the boundary expansion coefficient, the minimum bounding rectangle is adaptively expanded to generate the lesion region image, as shown in the following formula (12):
[0135] ,
[0136] (12);
[0137] In equation (12), This represents an image of the lesion area.
[0138] Therefore, compared with traditional minimum bounding rectangle clipping or fixed-ratio expansion methods, this embodiment introduces a dynamic boundary expansion mechanism based on lesion morphology features, which realizes adaptive adjustment of the ROI region. While preserving the surrounding tissue environment, edge invasion and posterior attenuation information of the lesion, it reduces irrelevant background interference and achieves a balance between lesion focusing and context preservation, thereby helping to improve the localization accuracy of breast cancer lesions.
[0139] S203, perform feature extraction processing on the lesion area image to obtain the lesion features corresponding to the target image.
[0140] In this step, the lesion region image is input into the feature extraction model for feature extraction, thereby obtaining the lesion features corresponding to the target image. The feature extraction model can be flexibly configured according to the actual situation; for example, it can be DenseNet, or other deep convolutional network structures, but it is not limited to these.
[0141] Furthermore, the feature extraction model can be set with multiple levels. In order to enhance the model's adaptability to lesions of different scales, multiple levels of output are preferably retained in the feature extraction stage, and modal feature representation is constructed through hierarchical aggregation to obtain lesion features, as shown in the following formula (13):
[0142] (13);
[0143] In equation (13), Indicates the characteristics of the lesion; Indicates the first The aggregate weights of each level are equal to one. Indicates the first Feature representation at each level; Indicates the number of levels.
[0144] Optionally, considering the limited number of breast image samples, a transfer learning strategy can be used to initialize the feature extraction model, that is, to use pre-trained parameters from a large dataset as initial weights, and then fine-tune them on the breast cancer lesion localization task; to reduce the risk of overfitting, batch normalization, Dropout, and early stopping strategies can also be introduced during the training process.
[0145] The above methods can effectively extract key features that are highly correlated with breast cancer lesions, thus providing a precise data foundation for subsequent processing.
[0146] In some embodiments, step S103 includes the following steps S301-S302.
[0147] S301 uses the prior description vectors corresponding to the B-mode ultrasound image, color Doppler blood flow image, and ultrasound elastography image respectively to perform gating processing, and obtains the target gating values corresponding to the B-mode ultrasound image, color Doppler blood flow image, and ultrasound elastography image respectively.
[0148] In this step, a lightweight dynamic gating network (CGN) that combines cross-modal contextual information, lesion prior information, and modal competition relationships is introduced to generate dynamic contribution weights for each modality, so as to achieve sample-level adaptive fusion based on this.
[0149] Furthermore, in some embodiments, step S301 above includes:
[0150] The lesion features and prior description vectors corresponding to the target image are input into the gating network to obtain the initial gating value;
[0151] Modal competition correction is performed on the initial gate value to obtain the target gate value corresponding to the target image;
[0152] The target image is any one of B-mode ultrasound image, color Doppler blood flow image, or ultrasound elastography image.
[0153] Here, firstly, the prior description vector corresponding to the target image is obtained, such as at least one of lesion size, edge complexity, morphological regularity, and texture inhomogeneity. Then, the lesion features and prior description vector corresponding to the target image are input into the gating network to obtain the initial gating value, as shown in the following formula (14):
[0154] (14);
[0155] In equation (14), Indicates the initial gate value; Represents the prior description vector; This represents a lightweight context-gated network, which can be any existing gating network. Here, lesion features and prior description vectors are used as inputs to the gating network, enabling it to have cross-modal context capabilities. Furthermore, it allows the gating network to rely not only on deep features but also on the structural features of the lesion region, thereby accurately generating gating values.
[0156] Then, modality competition correction is performed on the initial gate value, aiming to significantly increase the weight of a certain modality when its discriminative value in the current sample is higher, while the weight of redundant modalities is relatively suppressed, as shown in the following formula (15):
[0157] (15);
[0158] In equation (15), Indicates the first The corrected initial gating values for each mode; Indicates the first Initial gating values for each mode; Indicates the number of modes.
[0159] Finally, the corrected initial gate value is normalized to obtain the target gate value, as shown in the following formula (16):
[0160] (16);
[0161] In equation (16), Indicates the first The target gating value for each modality.
[0162] S302 utilizes the target gating values corresponding to B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images respectively to fuse the lesion features corresponding to B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images respectively, and obtains multimodal joint features.
[0163] In some embodiments, step S302 includes:
[0164] By using the target gating values corresponding to B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images, respectively, the lesion features corresponding to the B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images are weighted and summed to obtain multimodal joint features.
[0165] Here, firstly, the lesion features corresponding to each modality are projected, as shown in the following formula (17):
[0166] (17);
[0167] In equation (17), Indicates the first Lesion characteristics after projection of each modality; Indicates the first Lesion characteristics of each modality; Indicates the first Modal-specific projection matrices for each modality.
[0168] Then, based on the target gating values of each modality, the features are fused as shown in the following formula (18) to obtain the multimodal joint features:
[0169] (18);
[0170] In equation (18), This represents the joint features of multiple modalities.
[0171] It is worth noting that, in terms of the fusion mechanism, this embodiment emphasizes the following points: maintaining modality specificity, that is, each modality extracts features independently first; emphasizing information complementarity, explicitly utilizing the differences between different modalities in morphology, blood flow, and stiffness; controlling redundancy, avoiding duplicate information from dominating the final representation; and considering uncertainty, so that the contributions of different modalities in different samples have the ability to change dynamically.
[0172] Therefore, it is evident that different samples have varying degrees of dependence on different modalities. For example, some lesions primarily rely on the BUS modality for judgment, while others require the combination of CDFI blood flow information or UE stiffness information for identification. Thus, assigning the same weight to each modality consistently often fails to achieve optimal results. To address this, this embodiment introduces a dynamic weighting approach in feature fusion. This approach can reveal the differences in contribution of different modeling modalities to the localization of breast cancer lesions, adaptively adjusting the contribution ratio of different modalities accordingly. This method can fully leverage the complementary information between different modalities in terms of morphology, blood flow, and tissue stiffness, thereby compensating for the missing feature information in each modality. This enhances the expressive power of the fused features and the flexibility of classification, contributing to improved accuracy in the localization of breast cancer lesions.
[0173] In some embodiments, during the training phase, data augmentation methods such as rotation, flipping, translation, scaling, brightness perturbation, or random cropping are combined to expand the sample distribution and improve the model's generalization ability.
[0174] In some embodiments, to address the problem of imbalanced distribution of benign and malignant samples in breast cancer lesion datasets, the present invention preferably employs the SMOTE oversampling method to expand minority class samples. This method generates new synthetic samples by performing linear interpolation between minority class samples and their nearest neighbors, thereby increasing the distribution density of the minority class in the feature space and improving the model's ability to learn lesion features, as shown in the following formula (19):
[0175] (19);
[0176] In equation (19), Indicates a synthetic sample; Represents the nearest neighbor samples of minority class samples; Indicates minority class samples; A random number between 0 and 1. This method allows for the expansion of the minority class distribution without simply repeating samples.
[0177] In addition, refer to Figure 3 This application also provides a breast cancer lesion localization device, including:
[0178] The acquisition module 401 is used to acquire B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images containing breast cancer lesions;
[0179] The first processing module 402 is used to perform feature extraction processing on the B-mode ultrasound image, color Doppler blood flow image and ultrasound elastography image respectively, to obtain the lesion features corresponding to the B-mode ultrasound image, color Doppler blood flow image and ultrasound elastography image respectively;
[0180] The second processing module 403 is used to fuse the lesion features corresponding to the B-mode ultrasound image, color Doppler blood flow image and ultrasound elastography image respectively to obtain multimodal joint features.
[0181] The third processing module 404 is used for recognition processing based on multimodal joint features to obtain the location information of breast cancer lesions in any one of B-mode ultrasound images, color Doppler blood flow images, or ultrasound elastography images.
[0182] In summary, on the one hand, this application's embodiments introduce multimodal images such as B-mode ultrasound (Brightness-Mode Ultrasound) images, Color Doppler Flow Imaging (CDFI) images, and Ultrasound Elastography (UE) images, and accurately locate breast cancer lesions based on these images. This allows for the comprehensive capture of key features of breast cancer lesions in terms of morphology, blood flow, and tissue stiffness through multimodal data. On the other hand, this application's embodiments introduce a multi-scale feature fusion mechanism and attention enhancement unit into the U-Net model, improving the segmentation model's ability to identify complex boundaries, small-scale lesions, and low-contrast regions, thereby improving segmentation accuracy and lesion ROI quality. Furthermore, this application's embodiments introduce a dynamic weighting mechanism, which adaptively adjusts the contribution ratio of each modality based on the current sample, thereby avoiding the fusion rigidity problem caused by fixed weighting methods and improving the flexibility and generalization ability of multimodal fusion. Thus, the localization accuracy of breast cancer lesions can be effectively improved.
[0183] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0184] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.
[0185] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.
Claims
1. A method for locating breast cancer lesions, characterized in that, include: Acquire B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images containing breast cancer lesions; Feature extraction processing is performed on the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image respectively to obtain the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image respectively; The lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image are fused to obtain multimodal joint features; Based on the multimodal joint features, the location information of the breast cancer lesion is obtained from any one of the B-mode ultrasound image, the color Doppler blood flow image, or the ultrasound elastography image.
2. The method according to claim 1, characterized in that, The step involves performing feature extraction processing on the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image respectively to obtain the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, including: The target image is segmented to obtain a segmented image of the lesion; The lesion region in the lesion segmentation image is extracted to obtain the lesion region image; The lesion region image is subjected to feature extraction processing to obtain the lesion features corresponding to the target image; The target image is any one of the B-mode ultrasound image, the color Doppler blood flow image, or the ultrasound elastography image.
3. The method according to claim 2, characterized in that, The segmentation process of the target image to obtain the lesion segmentation image includes: The target image is input into the improved segmentation model for segmentation processing to obtain the lesion segmentation image; The improved segmentation model includes: An encoder structure includes multiple encoders connected in sequence; each encoder is used to perform feature extraction and downsampling processing on the input of the encoder to obtain the output of the encoder; The decoder structure includes multiple decoders connected sequentially, with a one-to-one correspondence between the multiple encoders and the multiple decoders. Each decoder performs cross-scale fusion skip connections with its corresponding encoder. The cross-scale fusion skip connections are used to fuse the encoded features of multiple scales in the output of the encoder to obtain multi-scale features, and to perform enhancement processing based on a multi-attention mechanism on the multi-scale features to obtain the skip connection output of the encoder. Each decoder is used to upsample the skip connection output of its corresponding encoder and the output of the previous decoder to obtain the output of the decoder. The output layer is used to process the output of the last decoder to obtain the lesion segmentation image.
4. The method according to claim 3, characterized in that, The process of fusing encoded features at multiple scales in the encoder output to obtain multi-scale features includes: The encoded features of each scale are scale aligned to obtain the aligned features of each scale. By utilizing the importance weights of each scale, the alignment features of each scale are fused to obtain the multi-scale features.
5. The method according to claim 3, characterized in that, The enhancement processing of the multi-scale features based on a multi-attention mechanism to obtain the skip connection output includes: The multi-scale features are subjected to multiple attention processing steps to obtain multiple initial attention weights; The target attention weight is obtained by fusing the multiple initial attention weights. The multi-scale features are enhanced using the target attention weights to obtain the skip connection output.
6. The method according to claim 2, characterized in that, The step of extracting the lesion region from the lesion segmentation image to obtain the lesion region image includes: Extract the minimum bounding rectangle of the lesion region from the lesion segmentation image; Based on the segmented image of the lesion, calculate the morphological feature parameters of the lesion region; Based on the morphological feature parameters, the minimum bounding rectangle is adjusted to generate the image of the lesion region.
7. The method according to claim 1, characterized in that, The process of fusing the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image to obtain multimodal joint features includes: Gating processing is performed using the prior description vectors corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively, to obtain the target gating values corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively. Using the target gating values corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively, the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image are fused to obtain the multimodal joint features.
8. The method according to claim 7, characterized in that, The process of gating the prior descriptive vectors corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image to obtain the target gating values corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively, includes: The lesion features and prior description vectors corresponding to the target image are input into the gating network to obtain the initial gating value; Modality competition correction is performed on the initial gate value to obtain the target gate value corresponding to the target image; The target image is any one of the B-mode ultrasound image, the color Doppler blood flow image, or the ultrasound elastography image.
9. The method according to claim 7, characterized in that, The multimodal joint features are obtained by fusing the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively, using the target gating values corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively, including: Using the target gating values corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image, respectively, the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image are weighted and summed to obtain the multimodal joint features.
10. A breast cancer lesion localization device, characterized in that, include: The acquisition module is used to acquire B-mode ultrasound images, color Doppler blood flow images, and ultrasound elastography images containing breast cancer lesions; The first processing module is used to perform feature extraction processing on the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image respectively, to obtain the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image, and the ultrasound elastography image respectively; The second processing module is used to fuse the lesion features corresponding to the B-mode ultrasound image, the color Doppler blood flow image and the ultrasound elastography image respectively to obtain multimodal joint features. The third processing module is used to perform identification processing based on the multimodal joint features to obtain the location information of the breast cancer lesion in any one of the B-mode ultrasound image, the color Doppler blood flow image, or the ultrasound elastography image.