A method, device, and medium for ultrasound breast tumor identification based on prior anatomical knowledge

Abstract

This invention proposes a method, device, and medium for ultrasound breast tumor identification based on prior anatomical knowledge. Specifically, it includes first segmenting the original breast ultrasound image using a segmentation algorithm; performing morphological processing on the segmentation results to obtain a breast region image; then, combining the original ultrasound image and the breast region image through channel synthesis to obtain a fused image; and finally, using an optimized target detection algorithm model to detect breast tumors on the fused image. The preprocessing method of fusion between the masked image based on prior knowledge and the original image significantly reduces interference from non-tumor regions, lowering the false positive rate for tumor identification. Simultaneously, the target detection algorithm is improved by utilizing large-scale convolution and programmed gradient information, enabling better acquisition of tumor and surrounding information at different scales, thus enhancing the tumor detection effect.

Description

Technical Field

[0001] This invention relates to machine learning technology, specifically to the field of machine learning that supports neural network models, and specifically to a technique for identifying breast tumors using neural network models. Background Technology

[0002] Breast cancer is currently the most common cancer among women worldwide and a leading cause of cancer death in women. Fortunately, early detection and timely treatment can significantly improve patient survival rates. Medical imaging is a useful tool for early breast cancer detection. Among various imaging methods, ultrasound imaging is currently the most widely used method for breast tumor detection and diagnosis in clinical practice. With the rapid development of deep learning technology, computer-aided examination of breast ultrasound images has also made rapid progress, assisting doctors in improving the accuracy, consistency, and reproducibility of diagnoses.

[0003] Object detection algorithms are one method for computer-aided examination of breast ultrasound images. These algorithms can obtain both tumor category information and the location of the tumor, and require simpler labels than segmentation algorithms. Therefore, object detection algorithms have certain advantages in computer-aided diagnosis of breast ultrasound images. However, object detection algorithms still face many unresolved problems in breast ultrasound images: 1. Various artifacts and noise exist in ultrasound images, which affect the final result of the detection algorithm; 2. Hypoechoic areas in the ducts of the breast, the fat above the breast, and the muscles below the breast are prone to false positives. While some mainstream object detection algorithms achieve high detection rates, they are also prone to misidentifying these areas as tumors, especially in normal tissue images. These false positives increase the workload of clinicians and reduce the effectiveness of computer-aided systems. Finally, it is difficult to distinguish between benign and malignant tumors using only ultrasound images. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention proposes a method, device and medium for ultrasound breast tumor identification based on prior anatomical knowledge. Specifically, it includes firstly segmenting the original breast ultrasound image using a segmentation algorithm model, then performing morphological processing on the segmentation result to obtain a breast region image, then combining the original ultrasound image and the breast region image through channel synthesis to obtain a fused image, and finally using an optimized target detection algorithm model to detect breast tumors in the fused image.

[0005] In a first aspect, the present invention proposes an ultrasound-guided method for identifying breast tumors based on prior anatomical knowledge, the method comprising the following steps:

[0006] Step S100: Use a segmentation model to segment the original breast ultrasound image to obtain a mask image of the breast region;

[0007] Step S200: The masked image of the breast region is fused with the original breast ultrasound image to obtain a fused image, specifically including:

[0008] S210. Calculate the mean value of the three RGB channels in the original breast ultrasound image to obtain the original single-channel breast ultrasound image.

[0009] S220. In the original single-channel breast ultrasound image, the pixel values ​​at positions where the mask is 1 are kept unchanged, and the positions at positions where the mask is 0 are replaced with the mean value of all pixel values ​​in the original single-channel breast ultrasound image, to obtain a fused single-channel breast ultrasound image; S230. The fused single-channel breast ultrasound image replaces the G channel in the three RGB channels of the original breast ultrasound image to obtain a fused image;

[0010] Step S300: Input the fused image into the target detection model for detection to obtain the tumor detection result.

[0011] Further, specifically, before channel fusion of the masked image of the breast region with the original breast ultrasound image to obtain a fused image, the method may further include the following steps:

[0012] Step S110: Perform one morphological opening operation on the mask image of the breast region, and then perform two dilation operations to obtain the preprocessed mask image of the breast region.

[0013] Preferably, step S200 may further be: ANDing the mask image of the breast region with the pixel value of one channel of the three channels of the original breast ultrasound image to obtain a fused image, wherein the channel is one of the R channel, G channel, and B channel.

[0014] Specifically, the target detection model includes a backbone extraction network, an enhanced feature extraction network, a feature decoupling head, and a programmable gradient information branch.

[0015] Specifically, in the ultrasound breast tumor identification method based on prior anatomical knowledge of the present invention, the method steps further include: before performing the ultrasound breast tumor identification method based on prior anatomical knowledge, step S400 is included, training the segmentation model and the target detection model using a training dataset.

[0016] Secondly, this invention proposes an ultrasound-guided breast tumor identification device based on prior anatomical knowledge, which specifically includes:

[0017] The segmentation module is used to segment the raw breast ultrasound image to obtain a mask image of the breast region;

[0018] The first preprocessing module is used to preprocess the masked image of the breast region;

[0019] The second preprocessing module is used to perform channel fusion of the masked image of the breast region and the original breast ultrasound image to obtain a fused image. Specifically, it includes: calculating the mean of the three RGB channels in the original breast ultrasound image to obtain an original single-channel breast ultrasound image; in the original single-channel breast ultrasound image, keeping the pixel values ​​unchanged at the positions where the mask is 1, and replacing the positions where the mask is 0 with the mean of all pixel values ​​in the original single-channel breast ultrasound image to obtain a fused single-channel breast ultrasound image; replacing the G channel of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain the fused image.

[0020] The prediction module is used to input the fused image into the target detection model to obtain the tumor detection result.

[0021] Thirdly, the present invention proposes an ultrasound imaging display system, comprising an ultrasound probe, a display, and an ultrasound breast tumor identification device based on prior anatomical knowledge as described above, wherein the segmentation module of the ultrasound breast tumor identification device based on prior anatomical knowledge is electrically connected to the ultrasound probe, and the display is electrically connected to the prediction module of the ultrasound breast tumor identification device based on prior anatomical knowledge, characterized in that it includes:

[0022] The ultrasound probe is used to acquire raw breast ultrasound images in real time;

[0023] The ultrasound breast tumor identification device based on prior anatomical knowledge is used to receive the original breast ultrasound image and output the tumor detection result of the original breast ultrasound image.

[0024] The display includes a first display area and a second display area, wherein the first display area is used to display the real-time breast ultrasound image;

[0025] The second display area is used to display the tumor detection results in real time.

[0026] Specifically, the second display area will simultaneously display the breast ultrasound image and overlay the tumor detection result onto the image.

[0027] Fourthly, the present invention proposes an electronic device, characterized in that it includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs including steps for performing the ultrasound breast tumor identification method based on prior anatomical knowledge.

[0028] Fifthly, the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the ultrasound breast tumor identification method based on prior anatomical knowledge as described in any one of the claims.

[0029] Based on the present invention, a method, device, and medium for ultrasound breast tumor identification based on prior anatomical knowledge achieve the following beneficial technical effects:

[0030] (1) In this invention, U-Net is used to segment the image to obtain the breast region image. The prior anatomical knowledge of the breast and breast tumor is integrated into the breast tumor detection algorithm, so that the model can better focus on the breast tumor and its surrounding area, reduce the false positives of detection, and improve the accuracy of the algorithm in classifying the benign and malignant tumors.

[0031] (2) This invention reduces the false positive rate of ultrasound breast tumor detection while ensuring the accuracy and detection rate of tumor identification. To achieve this goal, this invention incorporates prior knowledge of breast anatomy into the target detection algorithm, improving existing target detection algorithms. Specifically, this invention uses U-Net to segment the image to obtain a mask image of the breast region, and then performs channel fusion between the mask image of the breast region and the original image. This preprocessing method of fusing the mask image with the original image based on prior knowledge can greatly reduce interference from non-tumor regions and reduce the false positive rate of tumor identification. At the same time, this preprocessing method also shows that prior knowledge of anatomy can improve the performance of breast tumor detection algorithms. In addition, this invention improves the target detection algorithm by using large-scale convolution and programmed gradient information, which can better obtain information about tumors and their surroundings at different scales, improving the effect of tumor detection. Finally, this invention demonstrates the effectiveness and superiority of the algorithm through a large number of experiments, and has broad application prospects. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic diagram of the hierarchical structure of the original breast ultrasound image provided in an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of different types of raw breast ultrasound images provided in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of a masked image of the breast region provided in an embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of a fused single-channel breast ultrasound image provided in an embodiment of the present invention.

[0037] Figure 5 This is a schematic diagram of a fused image obtained by channel fusion of a masked image of the breast region and an original breast ultrasound image, as provided in an embodiment of the present invention.

[0038] Figure 6 This is a schematic diagram of a preprocessed masked image of the breast region provided in an embodiment of the present invention.

[0039] Figure 7 This is an overall schematic diagram of the original-mask-fusion of three types of breast ultrasound images provided in the embodiments of the present invention.

[0040] Figure 8 This is a schematic diagram of the network architecture of the target detection model provided in an embodiment of the present invention.

[0041] Figure 9 These are schematic diagrams (left) of the MBConv-CA module of the backbone feature extraction network of the breast tumor target detection model provided in this embodiment of the invention, and schematic diagram (right) of the CA module of the MBConv-CA module.

[0042] Figure 10 This is a schematic diagram of the backbone feature extraction network of the target detection model for breast tumors provided in this embodiment of the invention.

[0043] Figure 11 This is a schematic diagram of the UniRepLK-SPPF module, which is part of the enhanced feature extraction network for the breast tumor detection model provided in this embodiment of the invention.

[0044] Figure 12 This is a schematic diagram of the enhanced feature extraction network of the target detection model for breast tumors provided in this embodiment of the invention.

[0045] Figure 13 This is a schematic diagram of the feature decoupling head of the target detection model for breast tumors provided in an embodiment of the present invention.

[0046] Figure 14 This is a schematic diagram showing the structure of the tumor region marked by the positioning frame provided in the embodiment of the present invention. Detailed Implementation

[0047] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0048] In view of the shortcomings of existing technologies, this invention proposes a method, device, and medium for ultrasound breast tumor identification based on prior anatomical knowledge. Its main objective is to incorporate the anatomical relationship between the breast and the tumor into the target detection algorithm, which can significantly improve the performance of the target detection algorithm in ultrasound breast images. It reduces the false positive rate while maintaining high detection and accuracy. Specifically, ultrasound breast images can be divided into the fat layer, breast layer, muscle layer, and pleural cavity layer, such as... Figure 1 The diagram illustrates the hierarchical structure of a raw breast ultrasound image. Most tumors are located within the breast tissue, while a small percentage protrude into the fat. Breast tissue is generally easier to identify than breast tumors. Therefore, this invention first utilizes a segmentation algorithm to identify the breast region (i.e., prior knowledge), then processes the breast ultrasound image using the obtained segmentation information, and finally uses a target detection algorithm model to identify breast tumors in the processed ultrasound image. This reduces interference from irrelevant areas outside the breast, allowing the model to focus more on the breast and tumor regions, reducing false positives, improving the accuracy of benign / malignant classification, and enhancing the performance of breast tumor detection.

[0049] Furthermore, the target detection model of this invention incorporates programmable gradient information and large-kernel convolution, which can improve the detection performance of breast tumors. Programmable gradients introduce more gradient information during training, allowing the model to be trained better and thus improving the detection performance of breast tumors. Large-kernel convolution can expand the receptive field of neurons, allowing the network to pay attention to more surrounding information. This is quite effective for breast tumor identification, as a certain amount of surrounding information about the breast tumor can improve the accuracy of breast tumor identification.

[0050] Specifically, an ultrasound-based breast tumor identification method based on prior anatomical knowledge includes first segmenting the original breast ultrasound image using a segmentation algorithm model, then performing morphological processing on the segmentation results to obtain a breast region image, then combining the original ultrasound image and the breast region image through channel synthesis to obtain a fused image, and finally using an optimized target detection algorithm model to detect breast tumors in the fused image.

[0051] For example, this invention uses a public dataset and a private dataset. The private dataset serves as the training dataset, and method comparisons and ablation experiments are conducted on it. The private dataset comes from the China-Japan Friendship Hospital in Beijing. With the approval of the hospital's ethics committee, all patients signed informed consent forms. The private dataset contains images of 1878 patients, including 406 images of malignant tumors, 592 images of benign tumors, and 880 images of normal patients. All results were confirmed by biopsy or follow-up results. During training, 200 tumor images and 880 normal tissue images were used as the test set, 98 tumor images as the validation set, and the remaining 700 tumor images as the training set. The ratio of benign to malignant tumors in both the test and validation sets was 1:1. The public dataset, BrEaST with tumor-level labels, was used as the external test set. This dataset comes from 256 patients and includes 98 images of malignant tumors, 154 images of benign tumors, and 4 images of normal tissue. All tumors were confirmed by biopsy or subsequent care results.

[0052] In the following embodiments of the present invention, the specific exemplary technical solutions are for illustrative purposes only, and the implementation of the technical solutions is not limited thereto.

[0053] First Embodiment

[0054] In one embodiment, the present invention provides an ultrasound-based breast tumor identification method based on prior anatomical knowledge. This method can be applied to offline ultrasound images or to breast ultrasound images acquired by ultrasound equipment. The ultrasound images used for training and testing the network model can both be derived from the aforementioned dataset. The method includes the following steps:

[0055] Step S100: Use a segmentation model to segment the original breast ultrasound image to obtain a mask image of the breast region.

[0056] Specifically, this invention is based on prior anatomical knowledge: breast tumors are mainly located within or adjacent to the breast tissue layer. Therefore, this invention utilizes a segmentation algorithm to first segment the breast region, obtaining the actual breast region. The original breast ultrasound image corresponding to this breast region can include both normal tissue images and images of the breast region containing tumors.

[0057] Specifically, the segmentation model of the present invention can be a segmentation algorithm capable of accurately segmenting the breast region. In one embodiment, the segmentation algorithm may include: 1. Based on experience, the doctor uses the breast region as a constraint in the ultrasound breast image to remove interference from external information in the original ultrasound image, and uses random forest to detect and segment the breast region in the original ultrasound image; 2. By pre-modeling the breast anatomical structure and using neutral connections, different layers are decomposed in the original breast ultrasound image to obtain the breast region; 3. A convolutional neural network is used to segment the breast ultrasound image into skin layer, fat layer, and breast layer on the three-dimensional breast ultrasound image; 4. A semi-supervised semantic segmentation of the breast anatomical structure is performed using a semi-supervised framework. Simultaneously, the segmentation algorithm of the present invention can also be an improved segmentation algorithm, using the deep learning U-Net segmentation network. Since this network is very suitable for medical image segmentation, the present invention preferably uses the U-Net segmentation network. For breast ultrasound images, the breast structure is relatively fixed, its distribution in the image follows a specific pattern, the semantics are simple and clear, and low-level semantic features can provide good information. U-Net effectively utilizes low-level semantic features through skip-connections during segmentation, making it suitable for ultrasound breast segmentation. Therefore, in one embodiment, preferably, the present invention uses the U-Net segmentation network model. The U-Net network is trained using breast region labels (the labels are pixel-level labels with two values, corresponding to breast regions and non-breast regions respectively). The labels are manually annotated by junior doctors on the original breast ultrasound images and corrected by senior doctors. The U-Net segmentation network model is obtained by training the U-Net with the original breast ultrasound images and corresponding labels in the training set.

[0058] In one embodiment, before the segmentation model segments the breast region to obtain an image, to ensure that the size of the input raw breast ultrasound image is adapted to the segmentation model, or to unify the size of the raw breast ultrasound image with the parameters of the segmentation model, this step may further include: normalizing the size of the raw breast ultrasound image. Since the original ultrasound breast images vary in size, they are initially normalized to a uniform size; for example, the images can be uniformly adjusted to 224*224 pixels. Each raw breast ultrasound image includes three RGB channels. In embodiments of the present invention, the original breast ultrasound image, the mask image of the breast region, and the single-channel breast ultrasound image and fused image mentioned below can all be set to the same size.

[0059] Specifically, such as Figure 2As shown, this is a schematic diagram of different types of raw breast ultrasound images. From left to right, there are two images each of normal breast ultrasound, breast ultrasound containing benign tumors, and breast ultrasound containing malignant tumors (corresponding to normal, benign, and malignant types). The raw breast ultrasound images of different types are input into the segmentation network model. The two channels output by the segmentation network model are transformed by the softmax function to obtain a binary image of the background and target (breast) with channel 1. This binary image is the mask image of the breast region. For example, as shown... Figure 3 The image shown is a masked image of the breast region. In the masked image, the breast region will be marked as 1, and the non-breast region will be the background region, marked as 0.

[0060] Step S200: The masked image of the breast region is fused with the original breast ultrasound image to obtain a fused image.

[0061] Specifically, after obtaining the mask image of the breast region, the fused image containing the breast region is obtained using the mask.

[0062] In this step, to reduce interference from irrelevant areas outside the breast in the original breast ultrasound image and allow the model to focus more on the breast and tumor regions in the original breast ultrasound image, thus reducing false positives, the channel fusion process is performed based on the fact that each original breast ultrasound image includes three RGB channels. Specifically, this involves ANDing the mask image of the breast region with the pixel values ​​of each of the three channels of the original breast ultrasound image to obtain a fused image; or, ANDing the mask image of the breast region with the pixel values ​​of one channel (i.e., R channel, G channel, B channel) of the three channels of the original breast ultrasound image to obtain a fused image.

[0063] In one embodiment, this embodiment also proposes an improved channel fusion method for breast ultrasound images. The channel fusion in step S200 specifically includes:

[0064] S210. Calculate the mean value of the three RGB channels in the original breast ultrasound image to obtain the original single-channel breast ultrasound image.

[0065] S220. In the original single-channel breast ultrasound image, the pixel values ​​at positions where the mask is 1 are kept unchanged, while the positions at positions where the mask is 0 are replaced with the mean value of all pixel values ​​in the original single-channel breast ultrasound image, resulting in a fused single-channel breast ultrasound image. Specifically, the purpose of the mean replacement method is to keep the mean grayscale value of the breast region image and the grayscale image unchanged, reducing the impact of grayscale value changes on the subsequent detection network, such as... Figure 4 The diagram shows a fused single-channel breast ultrasound image.

[0066] S230. Replace the pixel value of one of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain the fused image. Specifically, replace the pixel value of one of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain the fused image. Figure 4 The image of the breast region shown replaces the G channel in the RGB channels of the original breast ultrasound image to obtain a fused image, as shown below. Figure 5 As shown in the figure. Finally, the fused image is fed into the improved target detection network of this invention to obtain the detection results of breast tumors.

[0067] The resulting fused image is an RGB three-channel image. In one embodiment, to obtain a fused single-channel breast ultrasound image, step S220 may further involve performing a bitwise AND operation between the original single-channel breast ultrasound image and the mask image of the breast region to obtain the fused single-channel breast ultrasound image. Simultaneously, step S230 may involve replacing the 'r' channel of the RGB three channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain the fused image, or replacing the 'b' channel of the RGB three channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain the fused image. Through the improved fusion processing method in this embodiment, based on the segmentation result in step S220, the background region is set to the mean grayscale value of the breast ultrasound image. This eliminates the influence of the background region while maintaining the overall grayscale mean of the image, avoiding excessive interference. Meanwhile, in step S230, the fused single-channel breast ultrasound image will replace the g channel of the three RGB channels in the original breast ultrasound image. The resulting fused image also maintains the overall grayscale mean and has better adaptability to the subsequent network detection model.

[0068] In existing technologies, directly applying existing target detection algorithms to ultrasound breast images for tumor prediction results in a significant false positive problem. Hypoechoic areas in the fat and muscle layers above the breast are easily misidentified as tumors, leading to false positives. Therefore, to address this issue, this invention does not directly use breast region images for breast tumor detection. There are two main reasons for this: first, breast region segmentation results are not always entirely reliable and may miss some tumor areas, leading to false negatives; second, retaining only the breast region results in excessive loss of information about the surrounding tumor areas, potentially making it harder for the target detection algorithm to distinguish between ducts and tumors within the breast region, thus causing false positives. Therefore, this invention performs channel fusion between a masked image of the breast region and the original breast ultrasound image. This fusion method suppresses information from non-breast regions while retaining information about the breast tumor and its surrounding areas. This approach, while suppressing non-tumor information, preserves more information about the tumor and its surrounding areas, aiding in subsequent tumor detection and classification.

[0069] In one embodiment, before step S200, which involves channel fusion of the masked image of the breast region with the original breast ultrasound image to obtain a fused image, the ultrasound breast tumor identification method based on prior anatomical knowledge may further include the following steps:

[0070] Step S110: Preprocess the mask image of the breast region. Specifically, the preprocessing includes morphological processing, noise reduction, etc. Specifically, in one embodiment, the mask image of the breast region is subjected to one morphological opening operation and then two dilation operations to obtain the preprocessed mask image of the breast region.

[0071] Specifically, a morphological opening operation is performed on the binary image using a 5x5 elliptical kernel provided by the OpenCV library to remove noise from the background region. Two dilation operations are then performed to ensure that tumor areas are not missed as much as possible within the breast region. The number of morphological processing operations can be adjusted according to the actual spiculation of the image; this invention is not limited to the single opening operation and two dilation operations described above. After these three morphological operations, a preprocessed mask image of the breast region is obtained. Furthermore, in one embodiment, this mask image can be simultaneously restored to the size of the original ultrasound breast image during this step.

[0072] Combination such as Figure 3 and 4 As shown, because Figure 2 The masked image of the mid-mammary gland region contained some noise points and edge spikes. Preprocessing improved this situation and minimized the omission of mammary gland areas. This was achieved by preprocessing the masked image of the mammary gland region. Figure 6 The image shown is a masked image of the breast region after preprocessing. This process removes noise from the area surrounding the tumor, preventing missed areas and better preserving information about the tumor and surrounding regions, which is beneficial for subsequent tumor detection and classification.

[0073] In one embodiment, optionally, to assess whether tumor regions would be missed after U-Net and morphological post-processing, the present invention calculated a recall rate of 98.89% for the tumor regions. Based on the recall rate and actual results, we find that the tumor regions are generally identified, but a few false negatives still exist, mainly occurring in the peripheral areas of the tumor, such as... Figure 7 As shown in the last column. Specifically, Figure 7The BUS image preprocessing results for specific breast regions are shown from left to right, illustrating the effects of preprocessing on normal tissue, benign tumors, and malignant tumors. From top to bottom, the original image, the preprocessed image after erasing non-breast areas, and the channel-fused image are displayed. This, in turn, verifies that the channel fusion method we ultimately adopted is effective and necessary. It can reduce interference from irrelevant areas and avoid missing or omitting tumor areas.

[0074] Step S300: Input the fused image into the target detection model for detection to obtain the tumor detection result.

[0075] Specifically, tumor detection results can primarily include outputs for malignant tumors, benign tumors, and normal patients. In another embodiment, the detection results can also be categorized into two groups: tumor and normal. In this invention, to determine whether a tumor is benign or malignant, the main approach is to specifically improve the target detection model to determine the benignity or malignancy of breast tumors and reduce false positive results.

[0076] Specifically, the object detection model is optimized based on the Yolox framework, and the network architecture diagram is as follows: Figure 8 As shown. The object detection model adopts the anchor-free mechanism of Yolox, which has a larger and more flexible solution space, making it extremely effective for breast tumors with large spatial scale variations. In addition, this invention has specifically optimized the remaining network structures to make the network more suitable for breast tumor detection. Specifically, the object detection model of this invention includes a backbone extraction network, an enhanced feature extraction network, a feature decoupling head, and a programmable gradient information branch. The backbone feature extraction network is constructed using stacked MBConv-CA modules. The enhanced feature extraction network uses SPPF optimized with GELAN and the large-kernel convolutional network UniRepLKNet. Furthermore, the network uses an invertible branch based on programmable gradient information to optimize the training of the backbone network. Combined with... Figure 8 In this model, the output of the object detection model can also include multiple channels. For example, the output channels of the object detection model can be seven channels, representing the x, y, w, and h values ​​of the breast tumor localization box, the confidence score of the localization box containing the object, and the probability of benign and malignant categories, respectively. Therefore, in one embodiment, the tumor detection result can also be the x, y, w, and h values ​​of the breast tumor localization box, the confidence score of the localization box containing the object, and the probability of benign and malignant categories.

[0077] In this invention, the backbone feature extraction network uses the EfficientNet backbone network as the backbone feature extraction network for the object detection model. Different layers of EfficientNet are formed by stacking Mobile Inverted Residual Bottleneck Blocks (MBConv) of different scales. In one embodiment, MBConv-CA is used to stack and form EfficientNet, and the structure diagram of MBConv-CA is shown below. Figure 9 As shown on the left, the CA module replaces the SE module. Compared to the SE module, the CA module focuses on both the channel and spatial information of the feature map. The structure of the CA module is shown below. Figure 9 As shown on the right, not only channel information but also orientation-related positional information is considered. Improvements to the backbone network make it more suitable for breast tumor detection. The CA module implements a channel attention mechanism in both width W and height H, simultaneously focusing on both spatial and channel information. Conv represents a convolutional layer, Depwise Conv represents a depthwise separable convolutional layer, BatchNorm represents a normalization operation, Linear represents a fully connected layer, and Sigmoid represents the Sigmoid activation function.

[0078] In one embodiment, specifically, as an example, the size of the input fused image to the backbone extraction network is adjusted to 3*512*512 (c*h*w), and the output sizes of out1, out2, and out3 are 320*16*16, 112*32*32, and 40*64*64, respectively. The backbone extraction network's role is to initially extract feature information at different scales from the fused image, while the MBConv-CA module's role is to assist the backbone feature extraction network in better extracting features. The MBConv-CA module at different locations has different input and output channels, and the input is downsampled with a stride of 2. For example... Figure 10 The diagram shows the structure of the backbone feature extraction network of the target detection model for breast tumors. The backbone extraction network of the breast tumor detection network is mainly composed of stacked MBConv-CA modules. The three feature layers out1, out2, and out3 are responsible for detecting tumors of large, medium, and small sizes, respectively.

[0079] In one embodiment, the purpose of enhancing the feature extraction network is to fuse the features extracted by the backbone network through upsampling and subsampling to further extract the feature information of the image. Specifically, the enhanced feature extraction network uses GELAN and SPPF (UniRepLK-SPPF) optimized based on UniRepLKNet. GELAN uses traditional convolution operators to achieve better parameter utilization, while UniRepLK-SPPF uses large kernel convolution, and its structure is as follows. Figure 11 As shown, UniRepLK-SPPF gives the network a larger receptive field, enabling it to better acquire information about the tumor and surrounding tissues, thus aiding in the detection, localization, and classification of breast tumors. Figure 11 As shown in the diagram, the UniRepLK-SPPF module structure diagram shows that MaxPool represents maximum pooling and UniRepLKNetBlock represents large kernel convolution operation.

[0080] In one embodiment, GELAN is specifically responsible for extracting features from the input image to generate multi-scale feature maps. For tumor images, multi-scale feature maps better reflect tumor information because tumor size itself is multi-scale; therefore, GELAN is used in this invention. Figure 12 The diagram shows the structure of the enhanced feature extraction network in the target detection model for breast tumors. It illustrates the overall structure of the enhanced feature extraction network. The input of the enhanced feature extraction network is the output of the backbone feature extraction network. The output is the result of enhancing the feature layers of the backbone feature extraction network. The three feature layers out1, out2, and out3 are responsible for detecting tumors of large, medium, and small sizes, respectively.

[0081] In one embodiment, the purpose of the feature decoupling head is to decode the output of the enhanced feature extraction network into the network's localization and classification results. Specifically, the input of the feature decoupling head is the output of the enhanced feature network, with output sizes of 7*16*16, 7*32*32, and 7*64*64, where 16, 32, and 64 represent the number of equal divisions of the initial image in the length and width directions, respectively, responsible for detecting large, medium, and small breast tumors. Seven output channels can be set, representing the x, y, w, and h values ​​of the breast tumor localization box, the confidence score of the localization box containing the object, and the probability of benign or malignant classification. In the decoupling head, the regression prediction of the localization box and the prediction of the confidence score and classification are performed separately because they are essentially regression and classification problems, respectively. Figure 13The diagram shows the structure of the feature decoupling head of the target detection model for breast tumors. The input to this detection head is the output of the enhanced feature extraction network, and the output is the location and category information of the breast tumor. After decoding, the final detection result can be obtained. The detection head separates the category classification, the determination of whether an object is contained, and the regression of the detection box. The solid boxes from left to right correspond to the above three operations respectively.

[0082] In one embodiment, combined with Figure 8 In this model, the programmable gradient information branch has two inputs: the output of the backbone extraction network and the resized breast image. The output is the output of the decoupling head. The Fuse module upsamples the parameters of different scales from the outputs of different Linear layers to make them equal. The programmable gradient information branch provides additional gradient information for one branch, helping the backbone extraction network converge faster and better, thereby improving the performance of the object detection network.

[0083] In one embodiment, optionally, before performing the ultrasound breast tumor identification method based on prior anatomical knowledge, the present invention may also include step S400: training the segmentation model and the target detection model using a training dataset.

[0084] Exemplarily, the dataset used in this invention comprises a public dataset and a private dataset. The private dataset serves as the training dataset, and method comparisons and ablation experiments are conducted on the private dataset. The private dataset originates from the China-Japan Friendship Hospital in Beijing. With the approval of the hospital's ethics committee, all patients signed informed consent forms. The private dataset contains images of 1878 patients, including 406 images of malignant tumors, 592 images of benign tumors, and 880 images of normal patients. All results were confirmed by biopsy or follow-up results. During training, 200 tumor images and 880 normal tissue images were used as the test set, 98 tumor images as the validation set, and the remaining 700 tumor images as the training set. The ratio of benign tumors in the test and validation sets was 1:1. The public dataset, BrEaST with tumor-level labels, was used as the external test set. This dataset consists of images from 256 patients, including 98 malignant tumors, 154 benign tumors, and 4 normal tissue images. All tumors were confirmed by biopsy or subsequent care results. The segmentation network was trained using labels for breast regions (the labels are pixel-level labels with two values, corresponding to breast regions and non-breast regions respectively). The labels were manually annotated by junior doctors using original breast ultrasound images and then corrected by senior doctors.

[0085] Specifically, in one embodiment, to balance the dataset, the number of malignant tumor images in the training set is first increased to the same level as the number of benign tumor images using spatial domain filtering, frequency domain filtering, and adding random noise. The labels of the increased images are kept consistent with their corresponding categories. During training, data augmentation techniques such as flipping, rotating, adding noise, cutout, and Mosica are used to aid network training.

[0086] Specifically, the original breast ultrasound image size can be adjusted to 224*224 as needed, and then U-Net is trained. The SGD optimizer is used, with momentum set to 0.937 and weight_decay set to 5e-4. The initial learning rate is 0.01, and the termination learning rate is 0.0001. A linear descent strategy is used to adjust the learning rate during training. The loss function for U-Net training is the sum of two loss functions:

[0087] L seg-train =αL dice +βL bce (1)

[0088] Where L dice L represents the Dice loss function. bce Let represent the BCE loss function. In the formula, α takes the value of 0.4, and β takes the value of 0.6.

[0089] Furthermore, segmented images of three categories are obtained through the segmentation model. These segmented images are then combined with their corresponding category labels to continue training the target detection model. Specifically, based on the size design of the target detection model, and according to actual design needs, the image can optionally be adjusted to 512*512 using bilinear interpolation. However, this invention is not limited to this size; the actual ultrasound image size can be dynamically adjusted according to specific requirements.

[0090] Specifically, all original breast ultrasound images from the training set are uniformly input into the segmentation model, and the fused image obtained by fusing the masked image output by U-Net with the original breast ultrasound images is uniformly input into the object detection network for training. The optimizer and learning rate settings used for training are the same as those for the U-Net network. The loss function for training the object detection model is the sum of two classification losses and one regression loss:

[0091] L detect-train =L obj-focal +L cls-bce +L giou (2)

[0092] Where L giouL represents the GIOU regression loss adjusted for tumor boundaries. cls-bce L represents the BCE classification loss adjusted for tumor categories. obj-focal This represents the Focal classification loss for determining whether an object is included. The Focal loss function is as follows:

[0093] L obj-focal =-α(1-p t ) γ log(p t (3)

[0094] Among them, α=0.25, γ=2, p t When the label is 1, it equals the network prediction probability p; when the label is 0, p... t =1-p.

[0095] This invention is based on the principle of programmable gradient information and utilizes reversible network branches to aid network training, improve the network's ability to identify breast tumors, reduce false positive rates, and increase the identification rate of benign and malignant tumors. Specifically, during training, the feature parameters output by the backbone extraction network are input into the reversible network branches for training. These reversible branches provide the backbone extraction network with more gradient information, assisting its training and resulting in better weight parameters. Removing the reversible network branches during prediction does not affect the original network's prediction results or speed.

[0096] This invention, based on prior anatomical knowledge, aims to reduce the false positive rate of ultrasound-guided breast tumor detection while maintaining accuracy and detection rate. To achieve this, it incorporates prior knowledge of breast anatomy into the target detection algorithm, improving existing algorithms. Specifically, it uses U-Net to segment images to obtain breast region images, then fuses these images with the original image through channel fusion. This image preprocessing significantly reduces interference from non-tumor areas, lowering the false positive rate for tumor detection. This preprocessing method also demonstrates that prior anatomical knowledge can enhance the performance of breast tumor detection algorithms. Furthermore, the invention improves the target detection model using large-scale convolution and programmed gradient information, enabling better acquisition of tumor and surrounding information at different scales, thus improving tumor detection effectiveness. Finally, extensive experiments demonstrate the effectiveness and superiority of our algorithm, showcasing its broad application prospects.

[0097] Example 2

[0098] The present invention also provides another embodiment, proposing an ultrasound breast tumor identification device based on prior anatomical knowledge, comprising:

[0099] The segmentation module is used to segment the raw breast ultrasound image to obtain a mask image of the breast region;

[0100] The first preprocessing module is used to preprocess the masked image of the breast region;

[0101] The second preprocessing module is used to perform channel fusion of the masked image of the breast region and the original breast ultrasound image to obtain a fused image. Specifically, it includes: calculating the mean of the three RGB channels in the original breast ultrasound image to obtain an original single-channel breast ultrasound image; in the original single-channel breast ultrasound image, keeping the pixel values ​​unchanged at the positions where the mask is 1, and replacing the positions where the mask is 0 with the mean of all pixel values ​​in the original single-channel breast ultrasound image to obtain a fused single-channel breast ultrasound image; replacing the G channel of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain the fused image.

[0102] The prediction module is used to input the fused image into the target detection model to obtain the tumor detection result.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device and module described above can be referred to the corresponding process in the aforementioned method embodiment one, and will not be repeated here.

[0104] Example 3

[0105] The present invention also provides another embodiment, proposing an ultrasound imaging display system comprising an ultrasound probe, a display, and an ultrasound breast tumor identification device based on prior anatomical knowledge as described above. The segmentation module of the ultrasound breast tumor identification device based on prior anatomical knowledge is electrically connected to the ultrasound probe, and the display is electrically connected to the prediction module of the ultrasound breast tumor identification device based on prior anatomical knowledge. The system is characterized by comprising:

[0106] The ultrasound probe is used to acquire raw breast ultrasound images in real time;

[0107] The ultrasound breast tumor identification device based on prior anatomical knowledge is used to receive the original breast ultrasound image and output the tumor detection result of the original breast ultrasound image.

[0108] The display includes a first display area and a second display area, wherein the first display area is used to display the real-time breast ultrasound image;

[0109] The second display area is used to display the tumor detection results in real time.

[0110] In the embodiments of the present invention, the specific structure and working process of the ultrasound breast tumor identification device based on prior anatomical knowledge are consistent with those in the two embodiments described above, and will not be repeated here.

[0111] In an embodiment of the present invention, for example, the first display area and the second area are linked. Based on the original breast ultrasound image acquired in real time by the ultrasound probe, the first display area displays the breast ultrasound image in real time, and the second display area displays the breast ultrasound image synchronously and outputs the tumor detection result on the image. This can help doctors determine the benign or malignant result of the breast tumor corresponding to the currently acquired breast ultrasound image in real time.

[0112] For example, tumor detection results may primarily include outputs for malignant tumors, benign tumors, and normal patients. In another embodiment, the breast region corresponding to the above results in the original breast ultrasound image can be displayed as an output box or a positioning box. Figure 14 As shown, the tumor region is marked and displayed using a bounding box. In one embodiment, the x, y, w, h coordinates of the breast tumor bounding box, the confidence level of the object contained in the bounding box, and the probability of benign or malignant categories can also be displayed next to the bounding box. In another embodiment, the tumor detection result can also be the x, y, w, h coordinates of the breast tumor bounding box, the confidence level of the object contained in the bounding box, and the probability of benign or malignant categories.

[0113] The implementation of the above embodiments can assist doctors in adjusting the direction and angle of the probe, helping them to accurately determine whether a tumor exists in the breast area and to promptly assist in the determination of whether the tumor is benign or malignant.

[0114] Example 4

[0115] The present invention also provides another embodiment, which proposes an electronic device including: a processor 1 and a memory 2.

[0116] The memory 2 is used to store computer programs.

[0117] The memory 2 includes various media capable of storing program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card, or optical disk.

[0118] The processor 1 is connected to the memory 2 and is used to execute the computer program stored in the memory 2 so that the processor can implement the above-mentioned ultrasound breast tumor identification method based on prior anatomical knowledge when executing the computer program.

[0119] Preferably, the processor 1 may be a central processing unit (CPU); or it may be an application-specific integrated circuit (ASIC).

[0120] Example 5

[0121] The present invention also provides another embodiment, namely, a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the ultrasound breast tumor identification method based on prior anatomical knowledge as described above.

[0122] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0123] Those skilled in the art will recognize that the modules, units, and / or method steps of the various embodiments described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0124] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another device or system, or some features may be ignored or not executed.

[0125] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated unit described above can be implemented in hardware or as a software functional unit.

[0126] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying breast tumors using ultrasound based on prior anatomical knowledge, characterized in that, The method includes the following steps: Step S100: Use a segmentation model to segment the original breast ultrasound image to obtain a mask image of the breast region; Step S200: The masked image of the breast region is fused with the original breast ultrasound image to obtain a fused image, specifically including: S210. Calculate the mean value of the three RGB channels in the original breast ultrasound image to obtain the original single-channel breast ultrasound image. S220. In the original single-channel breast ultrasound image, the pixel values ​​at positions where the mask is 1 are kept unchanged, and the positions at positions where the mask is 0 are replaced with the mean value of all pixel values ​​in the original single-channel breast ultrasound image, to obtain a fused single-channel breast ultrasound image; S230. Replace one of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain a fused image; Step S300: Input the fused image into the target detection model for detection to obtain tumor detection results; the tumor detection results include tumor localization boxes and the confidence level, benign and malignant categories corresponding to the localization boxes; The target detection model includes a backbone extraction network, an enhanced feature extraction network, a feature decoupling head, and a programmable gradient information branch. The backbone feature extraction network uses EfficientNet as the backbone feature extraction network for the target detection model. Different layers of EfficientNet are formed by stacking Mobile Inverted Residual Bottleneck Blocks of different scales. The enhanced feature extraction network uses GELAN and an SPPF structure optimized based on UniRepLKNet, with its input being the output of the backbone feature extraction network. The feature decoupling head takes the output of the enhanced feature extraction network as its input and decodes the output into localization and classification results. The programmable gradient information provides gradient information for one branch, helping the backbone extraction network converge faster and better. The programmable gradient information branch has two inputs: one is the output of the backbone extraction network, and the other is the original breast ultrasound image after size adjustment. Its output is the output of the feature decoupling head.

2. The ultrasound breast tumor identification method based on prior anatomical knowledge according to claim 1, characterized in that, Before fusing the masked image of the breast region with the original breast ultrasound image to obtain a fused image, the method includes the following steps: Step S110: Perform one morphological opening operation on the mask image of the breast region, and then perform two dilation operations to obtain the preprocessed mask image of the breast region.

3. The ultrasound breast tumor identification method based on prior anatomical knowledge according to claim 1 or 2, characterized in that, Before performing the ultrasound breast tumor identification method based on prior anatomical knowledge, the method includes step S400, which involves training the segmentation model and the target detection model using a training dataset.

4. An ultrasound-guided breast tumor identification device based on prior anatomical knowledge, specifically comprising: The segmentation module is used to segment the original breast ultrasound image to obtain a mask image of the breast region; The first preprocessing module is used to preprocess the masked image of the breast region; The second preprocessing module is used to perform channel fusion of the masked image of the breast region and the original breast ultrasound image to obtain a fused image. Specifically, it includes: calculating the mean of the three RGB channels in the original breast ultrasound image to obtain an original single-channel breast ultrasound image; in the original single-channel breast ultrasound image, keeping the pixel values ​​unchanged at the positions where the mask is 1, and replacing the positions where the mask is 0 with the mean of all pixel values ​​in the original single-channel breast ultrasound image to obtain a fused single-channel breast ultrasound image; replacing one of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain the fused image. The prediction module is used to input the fused image into the target detection model to obtain tumor detection results; the tumor detection results include tumor bounding boxes and the confidence level, benign and malignant categories corresponding to the bounding boxes. The target detection model includes a backbone extraction network, an enhanced feature extraction network, a feature decoupling head, and a programmable gradient information branch. The backbone feature extraction network uses EfficientNet as the backbone feature extraction network for the target detection model. Different layers of EfficientNet are formed by stacking Mobile Inverted Residual Bottleneck Blocks of different scales. The enhanced feature extraction network uses GELAN and an SPPF structure optimized based on UniRepLKNet, with its input being the output of the backbone feature extraction network. The feature decoupling head takes the output of the enhanced feature extraction network as its input and decodes the output into localization and classification results. The programmable gradient information provides gradient information for one branch, helping the backbone extraction network converge faster and better. The programmable gradient information branch has two inputs: one is the output of the backbone extraction network, and the other is the original breast ultrasound image after size adjustment. Its output is the output of the feature decoupling head.

5. An ultrasound imaging display system, comprising an ultrasound probe, a display, and an ultrasound breast tumor identification device based on prior anatomical knowledge according to claim 4, wherein the segmentation module of the ultrasound breast tumor identification device based on prior anatomical knowledge is electrically connected to the ultrasound probe, and the display is electrically connected to the prediction module of the ultrasound breast tumor identification device based on prior anatomical knowledge, characterized in that, include: The ultrasound probe is used to acquire raw breast ultrasound images in real time; The ultrasound breast tumor identification device based on prior anatomical knowledge is used to receive the original breast ultrasound image and output the tumor detection result of the original breast ultrasound image. The display includes a first display area and a second display area, wherein the first display area is used to display the real-time breast ultrasound image; The second display area is used to display the tumor detection results in real time.

6. The ultrasound imaging display system according to claim 5, characterized in that, The breast ultrasound image is simultaneously displayed on the second display area, and the tumor detection result is superimposed on the image.

7. An electronic device, characterized in that, It includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, wherein the one or more programs include steps for performing the ultrasound breast tumor identification method based on prior anatomical knowledge as described in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the ultrasound breast tumor identification method based on prior anatomical knowledge as described in any one of claims 1-3.

Images