Ultrasonic breast tumor identification method and device based on prior anatomical knowledge, and medium

Abstract

The invention provides an ultrasonic breast tumor recognition method and device based on prior anatomical knowledge and a medium, and the method specifically comprises the steps: firstly segmenting an original breast ultrasonic image through a segmentation algorithm model, and carrying out the morphological processing of a segmentation result, thereby obtaining a breast region image; then performing channel synthesis on the original ultrasonic image and the breast region image to obtain a fused image, and finally performing breast tumor detection on the fused image by using the optimized target detection algorithm model. The interference of a non-tumor area can be greatly reduced through a prior knowledge mask image and original image fusion preprocessing mode, and the false positive rate of tumor recognition is reduced. Meanwhile, a target detection algorithm is improved by using large-scale convolution and programming gradient information, tumors of different scales and peripheral information can be better obtained, and the tumor detection effect is improved.

Description

Technical Field

[0001] The present invention relates to machine learning technology, and in particular to the field of machine learning supporting neural network models, and in particular to a technology for identifying breast tumors using neural network models. Background Art

[0002] Currently, breast cancer is the most common tumor disease in women around the world and the leading cause of cancer death in women. Fortunately, early detection of tumors and timely treatment can significantly improve the survival rate of patients. Medical imaging is a useful tool for early detection of breast cancer. Among various imaging methods, ultrasound imaging is currently the most widely used method for breast tumor detection and diagnosis in clinical practice. With the rapid development of deep learning technology, computer-aided examination of ultrasound breast images has also made rapid progress, which can assist doctors in improving the accuracy, consistency and repeatability of diagnosis.

[0003] The target detection algorithm is one of the methods to realize computer-aided inspection of ultrasound breast images. The target detection algorithm can obtain both the category information of the tumor and the location area of ​​the tumor, and the labels it requires are simpler than those of the segmentation algorithm. Therefore, the target detection algorithm has certain advantages in computer-aided diagnosis of ultrasound breast images. However, there are still many unresolved problems in the target detection algorithm in ultrasound breast images: 1. There are various artifacts and noises in ultrasound images, which will affect the final effect of the detection algorithm; 2. The ducts in the breast, the fat above the breast, and the low echo areas in the muscles under the breast are prone to false positive identification. Some of the current mainstream target detection algorithms can easily misidentify these areas as tumors while achieving a high detection rate, especially in normal tissue images. These false positive problems will increase the workload of clinicians and reduce the working effect of computer-aided systems. Finally, it is difficult to distinguish the benign and malignant nature of certain tumors only through B-ultrasound images. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes an ultrasonic breast tumor recognition method, device, and medium based on prior anatomical knowledge, which specifically includes first using a segmentation algorithm model to segment the original breast ultrasound image, performing morphological processing on the segmentation result to obtain a breast region image, then performing channel synthesis on the original ultrasound image and the breast region image to obtain a fused image, and finally using an optimized target detection algorithm model to detect breast tumors in the fused image.

[0005] In a first aspect, the present invention provides a method for ultrasonic breast tumor recognition based on prior anatomical knowledge, the method comprising the steps of:

[0006] Step S100, using a segmentation model to segment the original breast ultrasound image to obtain a mask image (mask) of the breast area;

[0007] Step S200, performing channel fusion on the mask image of the breast region and the original breast ultrasound image to obtain a fused image, specifically includes:

[0008] S210, calculating the average values ​​of the three RGB channels in the original breast ultrasound image to obtain an original single-channel breast ultrasound image;

[0009] S220, in the original single-channel breast ultrasound image, keep the pixel value unchanged according to the position with 1 in the mask image mask, and replace the position with 0 in the mask with the mean value of all pixel values ​​in the original single-channel breast ultrasound image to obtain a fused single-channel breast ultrasound image; S230, replace the G channel of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain a fused image;

[0010] Step S300: input the fused image into a target detection model for detection to obtain a tumor detection result.

[0011] Further, specifically, before performing channel fusion on the mask image of the breast area and the original breast ultrasound image to obtain a fused image, the method may further include the steps of:

[0012] Step S110: performing a morphological opening operation on the mask image of the breast region once, and then performing two dilation operations to obtain a preprocessed mask image (mask) of the breast region.

[0013] Preferably, the step S200 may also be: obtaining a fused image by adding pixel values ​​of one of the three channels of the mask image of the breast area and the original breast ultrasound image, wherein the channel is one of the R channel, the G channel and the B channel.

[0014] Specifically, the target detection model includes a backbone extraction network, an enhanced feature extraction network, a feature decoupling head, and a programmable gradient information branch.

[0015] Specifically, in an ultrasonic breast tumor recognition method based on prior anatomical knowledge of the present invention, the method steps also include: before performing the ultrasonic breast tumor recognition method based on prior anatomical knowledge, including step S400, using a training data set to train the segmentation model and the target detection model.

[0016] In a second aspect, the present invention proposes an ultrasonic breast tumor recognition device based on prior anatomical knowledge, which specifically comprises:

[0017] A segmentation module is used to segment the original breast ultrasound image to obtain a mask image (mask) of the breast area;

[0018] A first preprocessing module, used for preprocessing the mask image of the breast area;

[0019] The second preprocessing module is used to perform channel fusion on the mask image of the breast area and the original breast ultrasound image to obtain a fused image; specifically comprising: calculating the average value of the three RGB channels in the original breast ultrasound image to obtain the original single-channel breast ultrasound image; in the original single-channel breast ultrasound image, keeping the pixel value unchanged according to the position with 1 in the mask image mask, and replacing the position with 0 in the mask with the average value of all pixel values ​​in the original single-channel breast ultrasound image to obtain a fused single-channel breast ultrasound image; replacing the G channel of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain a fused image;

[0020] The prediction module is used to input the fused image into the target detection model for detection to obtain a tumor detection result.

[0021] In a third aspect, the present invention proposes an ultrasound imaging display system, which includes an ultrasound probe, a display, and the ultrasound breast tumor recognition device based on prior anatomical knowledge as described above, wherein the segmentation module of the ultrasound breast tumor recognition device based on prior anatomical knowledge is electrically connected to the ultrasound probe, and the display is electrically connected to the prediction module of the ultrasound breast tumor recognition device based on prior anatomical knowledge, and is characterized in that it includes:

[0022] The ultrasonic probe is used to collect original breast ultrasonic images in real time;

[0023] The ultrasonic breast tumor recognition device based on prior anatomical knowledge is used to receive the original breast ultrasound image and output the tumor detection result of the original breast ultrasound image;

[0024] The display comprises a first display area and a second display area, wherein the first display area is used to display the real-time breast ultrasound image;

[0025] The second display area is used to display the tumor detection result in real time.

[0026] Specifically, the second display area will synchronously display the breast ultrasound image and overlay the tumor detection result on the image.

[0027] In a fourth aspect, the present invention proposes an electronic device, characterized in that it includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include steps for executing the ultrasonic breast tumor identification method based on prior anatomical knowledge.

[0028] In a fifth aspect, the present invention proposes a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any one of the methods for ultrasonic breast tumor identification based on prior anatomical knowledge are implemented.

[0029] The ultrasonic breast tumor identification method, device, and medium based on prior anatomical knowledge of the present invention achieve the following beneficial technical effects:

[0030] (1) In the present invention, U-Net is used to segment the image to obtain a breast region image, and the prior anatomical knowledge of the breast and breast tumors is integrated into the breast tumor detection algorithm, so that the model can pay more attention to the breast tumor and its surrounding area, reduce the false positive detection, and improve the accuracy of the algorithm in classifying the benign and malignant tumors.

[0031] (2) The present invention reduces the false positive rate of ultrasonic breast tumor detection, while ensuring the accuracy and detection rate of tumor identification. In order to achieve this goal, the present invention incorporates the prior knowledge of breast anatomy into the target detection algorithm to improve the existing target detection algorithm. Specifically, the present invention uses U-Net to segment the image to obtain a mask image of the breast area, and then performs channel fusion on the mask image of the breast area and the original image. The preprocessing method of fusing the mask image and the original image based on prior knowledge can greatly reduce the interference of non-tumor areas and reduce the false positive rate of tumor identification. At the same time, this preprocessing method also shows that the prior knowledge of anatomy can improve the performance of the breast tumor detection algorithm. In addition, the present invention uses large-scale convolution and programmed gradient information to improve the target detection algorithm, which can better obtain tumors and surrounding information of different scales and improve the effect of tumor detection. Finally, the present invention proves the effectiveness and superiority of the algorithm of the present invention through a large number of experiments, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 It is a schematic diagram of the hierarchical structure of the original breast ultrasound image provided by an embodiment of the present invention.

[0034] Figure 2 It is a schematic diagram of different types of original breast ultrasound images provided by an embodiment of the present invention.

[0035] Figure 3 is a schematic diagram of a mask image of a breast region provided by an embodiment of the present invention.

[0036] Figure 4 is a schematic diagram of a fused single-channel breast ultrasound image provided by an embodiment of the present invention.

[0037] Figure 5 It is a schematic diagram of a fused image obtained by channel fusion of a mask image of a breast region and an original breast ultrasound image provided by an embodiment of the present invention.

[0038] Figure 6 It is a schematic diagram of a mask image of a breast region after preprocessing provided by an embodiment of the present invention.

[0039] Figure 7 It is an overall schematic diagram of original-mask-fusion of three types of breast ultrasound images provided by an embodiment of the present invention.

[0040] Figure 8 Schematic diagram of the network architecture of the target detection model provided by an embodiment of the present invention.

[0041] Fig. 9 It is a structural schematic diagram of the MBConv-CA module of the backbone feature extraction network of the target detection model of breast tumor provided by an embodiment of the present invention (left) and a structural schematic diagram of the CA module of the MBConv-CA module (right).

[0042] Fig.10 It is a structural diagram of the backbone feature extraction network of the target detection model of breast tumor provided by an embodiment of the present invention.

[0043] Fig.11 It is a structural diagram of the UniRepLK-SPPF module of the enhanced feature extraction network of the breast tumor detection model provided by an embodiment of the present invention.

[0044] Fig.12 It is a schematic diagram of the structure of the enhanced feature extraction network of the target detection model of breast tumors provided by an embodiment of the present invention.

[0045] Fig.13 It is a schematic diagram of the structure of a feature decoupling head of a breast tumor target detection model provided by an embodiment of the present invention.

[0046] Fig.14 This is a schematic diagram of a structure in which a tumor area is marked and displayed in a positioning frame manner provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0048] In view of the shortcomings of the prior art, the present invention proposes an ultrasonic breast tumor recognition method, device, and medium based on prior anatomical knowledge. Its main goal is to integrate the anatomical position relationship between the breast and the tumor into the target detection algorithm, which can significantly improve the performance of the target detection algorithm in ultrasonic breast images. While reducing the false positive rate of detection, it also has a high detection rate and accuracy. Specifically, the ultrasonic breast image can be divided into the fat layer, breast layer, muscle layer, and chest layer, such as Figure 1 As shown, a schematic diagram of the hierarchical structure of the original breast ultrasound image. Most tumors are located inside the breast, a small number of tumors will protrude into the fat, and the breast is relatively easier to identify breast tumors. Therefore, the present invention first uses a segmentation algorithm to first identify the breast area (i.e., prior knowledge), then uses the obtained segmentation information to process the breast ultrasound image, and finally uses the target detection algorithm model to identify breast tumors on the processed ultrasound image. In this way, the interference of irrelevant areas outside the breast can be reduced, allowing the model to pay more attention to the breast and tumor areas, reducing the false positives of identification, improving the accuracy of benign and malignant classification, and improving the performance of breast tumor detection.

[0049] In addition, the target detection model of the present invention also incorporates programmable gradient information and large kernel convolution into the target detection model, which can improve the effect of breast tumor detection. The programmable gradient introduces more back-propagation gradient information in training, so that the model is better trained, thereby improving the detection effect of breast tumors. Large kernel convolution can expand the receptive field of neurons, allowing the network to pay attention to more peripheral information. This is quite effective for the identification of breast tumors, because a certain amount of peripheral information of breast tumors can improve the recognition accuracy of breast tumors.

[0050] Specifically, an ultrasonic breast tumor recognition method based on prior anatomical knowledge includes first using a segmentation algorithm model to segment the original breast ultrasound image, performing morphological processing on the segmentation result to obtain a breast region image, then performing channel synthesis on the original ultrasound image and the breast region image to obtain a fused image, and finally using an optimized target detection algorithm model to detect breast tumors in the fused image.

[0051] Exemplarily, the data set in the present invention uses a public data set and a private data set. The private data set is used as a training data set, and the comparison of methods and ablation experiments are all performed on the private data set. The private data set comes from Beijing Friendship Hospital of China and Japan. Approved by the hospital ethics committee, all patients signed informed consent. The private data set contains a total of 1,878 patient images, of which 406 are malignant tumor images, 592 are benign tumor images, and 880 are normal patient images. All results are confirmed by biopsy results or follow-up results. During the training process, 200 tumor images and 880 normal tissue images were used as test sets, 98 tumor images were used as validation sets, and the remaining 700 tumor images were used as training sets. The ratio of benign tumors to malignant tumors in the test set and validation set is 1:1. The public data set is BrEaST with tumor grade labels, which is used as an external test set. The data set comes from 256 patients, including 98 malignant tumors, 154 benign tumors, and 4 normal tissue images. All tumors have been confirmed by biopsy or follow-up care results.

[0052] In the following embodiments of the present invention, the specific exemplary technical solutions are for the purpose of examples, and the implementation methods of the technical solutions are not limited thereto.

[0053] First embodiment

[0054] In one embodiment, the present invention provides an ultrasonic breast tumor recognition method based on prior anatomical knowledge. The method can be applied to offline ultrasound images or to breast ultrasound images collected by ultrasound equipment. The ultrasound images for training and testing the network model in the method can come from the aforementioned data set. The method includes the following steps:

[0055] Step S100: Use the segmentation model to segment the original breast ultrasound image to obtain a mask image (mask) of the breast area.

[0056] Specifically, the present invention is based on prior anatomical knowledge: breast tumors are mainly located in the breast layer, or close to the breast area. Therefore, specifically, the present invention uses a segmentation algorithm to first segment the breast area, and the segmentation obtains a real breast area, and the original breast ultrasound image corresponding to the breast area needle can include a normal tissue image and a breast area with a tumor image.

[0057] Specifically, the segmentation model of the present invention can be a segmentation algorithm that can accurately segment the breast region. In one embodiment, the segmentation algorithm may include: 1. The doctor uses the breast region as a constraint in the ultrasound breast image through experience, removes the interference of external information from the original ultrasound image, and uses random forest to detect and segment in the breast region of the original ultrasound image; 2. By pre-modeling the breast anatomical structure and using neutral connections to decompose different layers in the original breast ultrasound image, the breast region is obtained; 3. The breast ultrasound image is segmented into skin layer, fat layer and breast layer on the three-dimensional breast ultrasound image using a convolutional neural network; 4. The semi-supervised framework is used to perform semi-supervised semantic segmentation on the breast anatomical structure. At the same time, the segmentation algorithm of the present invention can also be an improved segmentation algorithm, using a deep learning U-Net segmentation network, which is very suitable for the segmentation of medical images based on the network. Therefore, preferably, the present invention uses a U-Net segmentation network. For breast ultrasound images, the breast structure is relatively fixed, the distribution in the image has a specific rule, the semantics are simple and clear, and low-level semantic features can also provide information well. U-Net makes good use of low-level semantic features through skip-connection in the segmentation process and is suitable for ultrasound breast segmentation. Therefore, in one embodiment, preferably, the present invention selects a U-Net segmentation network model. The U-Net network is trained using labels of breast areas (labels are pixel-level labels with two values, corresponding to breast areas and non-breast areas, respectively). The labels are manually annotated original breast ultrasounds by junior doctors and corrected by senior doctors. The U-Net segmentation network model is obtained by training U-net using the original breast ultrasound images of the training set and the corresponding labels.

[0058] In one embodiment, before the segmentation model is segmented to obtain an image of the breast area, in order to ensure that the size of the input original breast ultrasound image adapts to the segmentation model, or, the size of the original breast ultrasound image is unified with the parameters of the segmentation model. This step may also include: normalizing the size of the original breast ultrasound image. Since the original ultrasound breast images are of different sizes, the images will be normalized at the beginning to adjust the images to a uniform size. Exemplarily, the images can be uniformly adjusted to a size of 224*224. Each original breast ultrasound image includes three RGB channels. In an embodiment of the present invention, the sizes of the original breast ultrasound image, the mask image of the breast area, and the single-channel breast ultrasound image and the fused image involved below can all be set to the same size.

[0059] Specifically, Figure 2As shown in FIG. 1 , it is a schematic diagram of different types of original breast ultrasound images, including two breast ultrasound images of normal tissue, two breast ultrasound images containing benign tumors, and two breast ultrasound images containing malignant tumors (corresponding to the three types of normal, benign, and malignant) from left to right. The original breast ultrasound images of different types are input into the segmentation network model, and the two channels output from the segmentation network model are transformed by the softmax function to obtain a binary image of the background and the target (breast) with a channel of 1. The binary image is the mask image of the breast area. For example, Figure 3 The image shown is a mask image of the breast area. In the mask image, the breast area is marked as 1, and all non-breast areas are background areas and are marked as 0.

[0060] Step S200: performing channel fusion on the mask image of the breast area and the original breast ultrasound image to obtain a fused image.

[0061] Specifically, after the mask image mask of the breast area is acquired, the mask is used to obtain a fused image containing the breast area.

[0062] In this step, in order to reduce the interference of irrelevant areas outside the breast in the original breast ultrasound image, the model is made to pay more attention to the breast and tumor areas in the original breast ultrasound image, thereby reducing the false positives of recognition. Based on the fact that each original breast ultrasound image includes three RGB channels, the channel fusion processing specifically includes respectively ANDing the mask image of the breast area with the pixel values ​​of the three channels of the original breast ultrasound image to obtain a fused image; or, ANDing the mask image of the breast area with the pixel values ​​of one of the three channels of the original breast ultrasound image (i.e., R channel, G channel, and B channel) to obtain a fused image;

[0063] In one implementation, this embodiment further proposes an improved channel fusion method for breast ultrasound images. The channel fusion in step S200 specifically includes:

[0064] S210, calculating the average values ​​of the three RGB channels in the original breast ultrasound image to obtain an original single-channel breast ultrasound image;

[0065] S220, in the original single-channel breast ultrasound image, keep the pixel value unchanged according to the position with 1 in the mask image mask, and replace the position with 0 in the mask with the mean value of all pixel values ​​in the original single-channel breast ultrasound image to obtain a fused single-channel breast ultrasound image; specifically, the purpose of the mean value replacement method is to keep the grayscale value mean of the breast area image and the grayscale image unchanged, so as to reduce the impact of the grayscale value change on the subsequent detection network, such as Figure 4 As shown, a schematic diagram of a fused single-channel breast ultrasound image is obtained.

[0066] S230, replacing the pixel value of one of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain a fused image. Figure 4 The breast region image shown replaces the G channel of the RGB three channels of the original breast ultrasound image to obtain a fused image, such as Figure 5 Finally, the fused image is sent to the improved target detection network of the present invention to obtain the detection result of breast tumor.

[0067] Among them, the obtained fused image is an RGB three-channel image. In one embodiment, in order to obtain a fused single-channel breast ultrasound image, step S220 can also be to perform an AND operation on the original single-channel breast ultrasound image and the mask image of the breast area to obtain a fused single-channel breast ultrasound image; at the same time, step S230 can also be: replace the R channel of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain a fused image, or replace the B channel of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain a fused image. Through the improved fusion processing method in this embodiment, according to the segmentation result in step S220, the background area is set to the mean of the grayscale value of the breast ultrasound image, which can not only eliminate the influence of the background area, but also maintain the grayscale mean of the entire image to avoid excessive interference. At the same time, in step S230, the fused single-channel breast ultrasound image replaces the g channel of the three RGB channels in the original breast ultrasound image. The obtained fused image also maintains the overall grayscale mean of the image and has better adaptability to the subsequent network detection model.

[0068] In the prior art, when the existing target detection algorithm is directly applied to the ultrasound breast image, there will be a prominent problem of false positive problem in the process of predicting tumors. It is easy to misidentify some low echo areas in the fat and muscle layers above the breast as tumors, resulting in false positive problems. Therefore, in order to solve the problem of false positives, the present invention does not directly use the breast area image to detect breast tumors. There are two main reasons: first, the segmentation result of the breast area is not necessarily completely reliable, and some tumor areas may be missed, resulting in missed detection; second, only retaining the breast area leads to excessive information loss in some tumor peripheral areas, which may make it more difficult for the target detection algorithm to distinguish between ducts and tumors in the breast area, thereby leading to false positive problems. Therefore, the present invention performs channel fusion of the mask image of the breast area with the original breast ultrasound image. This fusion method suppresses the information of the non-breast area and retains the information of the breast tumor and its surrounding breast area. In addition to suppressing the information of the non-tumor area, this method retains more information of the tumor and the surrounding area, which is helpful for the subsequent detection and classification of tumors.

[0069] In one embodiment, before step S200, channel fusion of the mask image of the breast region and the original breast ultrasound image to obtain a fused image, the ultrasound breast tumor recognition method based on prior anatomical knowledge may further include the following steps:

[0070] Step S110, preprocessing the mask image of the breast region. Specifically, the preprocessing includes morphological processing, noise removal processing, etc. Specifically, in one embodiment, the mask image of the breast region is subjected to one morphological opening operation and then two dilation operations to obtain a preprocessed mask image of the breast region.

[0071] Specifically, a morphological opening operation is performed on the binary image using an elliptical kernel of size 5*5 provided by the opencv library to remove noise points in the background area. Then two dilation operations are performed to ensure that the tumor area is not missed in the breast area as much as possible. Specifically, the number of morphological processing times can be adjusted and set according to the burr situation of the actual image itself, and the present invention is not limited to the above-mentioned one opening operation and two dilation operations. After three morphological operations, a mask image (mask) of the pre-processed breast area is obtained. In addition, in one embodiment, the mask image can be restored to the size of the original ultrasound breast image at the same time in this step.

[0072] Combined with Figure 3 and 4 As shown, because Figure 2 There are some noise points and edge burrs in the mask image of the breast area. Preprocessing can improve this situation and omit the breast area as little as possible. Figure 6 The figure shows the mask image of the breast area after preprocessing. The noise of the surrounding information of the tumor area is removed, and at the same time, it can also avoid missing some tumor areas and causing missed detection, and better retain the information of the tumor and the surrounding area, which is helpful for the subsequent detection and classification of tumors.

[0073] In one embodiment, optionally, in order to evaluate whether the tumor area will be missed after U-Net and morphological post-processing, the present invention calculates the recall rate of the tumor area to be 98.89%. According to the recall rate and the actual effect, we can get that the tumor area is basically recognized, but there are still a few misses, which mainly occur in the surrounding area of ​​the tumor, such as Figure 7 Specifically, Figure 7The BUS image preprocessing results of a specific breast area in the figure show the preprocessing effects of normal tissue, benign tumors, and malignant tumors from left to right. From top to bottom, the original image, the preprocessing effect of erasing the non-breast area, and the channel fused image are shown. This in turn verifies that the channel fusion method we finally adopted is effective and necessary. It can reduce the interference of irrelevant areas and avoid missing tumor areas.

[0074] Step S300: input the fused image into a target detection model for detection to obtain a tumor detection result.

[0075] Specifically, the tumor detection results may mainly include the output results of malignant tumors, benign tumors, and normal patients. In another embodiment, the detection results may also be two categories: tumor and normal. In the present invention, in order to achieve the judgment of benign and malignant tumors, the judgment of benign and malignant breast tumors is mainly based on targeted improvements to the target detection model, and the recognition result of reducing false positives is achieved.

[0076] Specifically, the target detection model is optimized based on the Yolox framework. The network architecture is shown in the figure below: Figure 8 As shown. The target detection model adopts the anchor free mechanism of Yolox, which has a larger and more flexible solution space, which is extremely effective for breast tumors with large spatial scale variations. In addition, the present invention has carried out targeted optimization of the remaining network structures to make the network more suitable for breast tumor detection. Specifically, the target detection model of the present invention includes a backbone extraction network, an enhanced feature extraction network, a feature decoupling head, and a programmable gradient information branch. Among them, the backbone feature extraction network is stacked using MBConv-CA modules. In the enhanced feature extraction network part, GELAN and the large kernel convolutional network UniRepLKNet are used to optimize the SPPF. In addition, the network also uses reversible branches based on programmable gradient information to optimize the training of the backbone network. Combined with Figure 8 In the example, the output of the target detection model may also include multiple channels. For example, the output channels of the target detection model may be 7 channels, which respectively represent the x, y, w, h of the breast tumor localization frame, the confidence that the localization frame contains an object, and the probability of benign and malignant categories. Therefore, in one embodiment, the tumor detection result may also be the x, y, w, h of the breast tumor localization frame, the confidence that the localization frame contains an object, and the probability of benign and malignant categories.

[0077] The backbone feature extraction network uses the backbone extraction network of EfficientNet as the backbone feature extraction network of the target detection model. Different layers of EfficientNet are formed by stacking Mobile Inverted Residual Bottleneck Block (MBConv) of different scales. In the present invention, in one embodiment, MBConv-CA is used to stack EfficientNet. The structure diagram of MBConv-CA is as follows: Fig. 9 As shown on the left, the SE module is replaced by the CA module. Compared with the SE module, the CA module focuses on both the channel and spatial information of the feature map. The structure of the CA module Fig. 9 As shown on the right, not only channel information is considered, but also direction-related position information. The improvement of the backbone network makes it more suitable for breast tumor detection. The CA module implements the channel attention mechanism in both width W and height H, paying attention to both spatial information and channel information. Conv stands for convolutional layer, Depwise Conv stands for depth-separable convolutional layer, BatchNorm stands for normalization operation, Linear stands for fully connected layer, and Sigmoid stands for Sigmoid activation function.

[0078] In one embodiment, specifically, as an example, the size of the input fused image of the backbone extraction network is adjusted to 3*512*512 (c*h*w), and the output sizes of out1, out2, and out3 are 320*16*16, 112*32*32, and 40*64*64, respectively. The role of the backbone extraction network is to preliminarily extract feature information of different scales in the fused image, and the role of the MBConv-CA module is to help the backbone feature extraction network to better extract features. The input and output channels of the MBConv-CA modules at different positions are different, and the input is downsampled with a step size of 2. Fig.10 As shown in the figure, the structural diagram of the backbone feature extraction network of the target detection model of breast tumors, the backbone extraction network of the detection network of breast tumors, the backbone extraction network is mainly composed of stacked MBConv-CA modules, and the output out1, out2, and out3 three feature layers are responsible for detecting large, medium, and small levels of tumors respectively.

[0079] In one embodiment, the purpose of strengthening the feature extraction network is to fuse the features extracted by the backbone network through up and down sampling to further extract the feature information of the image. Specifically, the enhanced feature extraction network uses GELAN and SPPF (UniRepLK-SPPF) optimized based on UniRepLKNet. GELAN uses traditional convolution operators to achieve better parameter utilization, and UniRepLK-SPPF uses large kernel convolution, whose structure is as follows: Fig.11 As shown in Figure 2. UniRepLK-SPPF enables the network to have a larger receptive field, which can better obtain information about the tumor and its surroundings, and help detect, locate and classify breast tumors. Fig.11 As shown in the figure, the UniRepLK-SPPF module structure diagram, MaxPool represents maximum pooling, and UniRepLKNetBlock represents large kernel convolution operation.

[0080] In one embodiment, specifically, GELAN is responsible for extracting features from the input image and generating a multi-scale feature map. For tumor images, the multi-scale feature map can better reflect the information of the tumor, because the tumor size itself is multi-scale. Therefore, GELAN is used in the present invention. Fig.12 As shown in the figure, the structural diagram of the enhanced feature extraction network of the target detection model of breast tumors shows the overall structure of the enhanced feature extraction network. The input of the enhanced feature extraction network is the output of the trunk feature extraction network, and the output is the result of enhancing the feature layer of the output of the trunk feature extraction network. The three feature layers of the output out1, out2, and out3 are also responsible for detecting large, medium, and small tumors respectively.

[0081] In one embodiment, the purpose of the feature decoupling head is to decode the output of the enhanced feature extraction network into the positioning and classification results of the network. Specifically, the input of the feature decoupling head is the output of the enhanced feature network, and the output sizes are 7*16*16, 7*32*32 and 7*64*64, respectively. 16, 32, 64 represent how many equal parts the initial image is divided in length and width, respectively, and are responsible for the detection of large, medium and small breast tumors, respectively. The output can be set to 7 channels, and the 7 channels represent the x, y, w, h of the breast tumor positioning box, the confidence that the positioning box contains an object, and the probability of benign and malignant categories. In the decoupling head, the regression prediction of the positioning box and the confidence and category prediction are performed separately, because the two are essentially regression problems and classification problems, respectively, and therefore are performed separately. Fig.13Figure 1 shows the structure of the feature decoupling head of the target detection model for breast tumors. The input of the detection head is the output of the enhanced feature extraction network, and the output is the location and category information of the breast tumor. After decoding, the final detection result can be obtained. The detection head performs the classification of categories, the judgment of whether objects are included, and the regression of the detection box separately. The solid line boxes correspond to the above three operations from left to right.

[0082] In one embodiment, in combination Figure 8 In the example, the input of the programmable gradient information branch consists of two parts: one is the output of the backbone extraction network, and the other is the resized breast image. The output is the output of the decoupling head. The function of the Fuse module is to adjust the parameters of different scales of the output of different Linear layers to the same size by upsampling. The function of the programmable gradient information branch is to provide more gradient information of a branch to help the backbone extraction network converge faster and better, thereby improving the effect of the target detection network.

[0083] In one embodiment, optionally, before performing the ultrasonic breast tumor recognition method based on prior anatomical knowledge, the present invention may optionally further include step S400, training the segmentation model and the target detection model using a training data set.

[0084] Exemplarily, the dataset of the present invention uses a public dataset and a private dataset. The private dataset is used as a training dataset, and the comparison of the methods and ablation experiments are all performed on the private dataset. The private dataset comes from the Beijing Friendship Hospital of China and Japan. Approved by the hospital ethics committee, all patients signed informed consent. The private dataset contains a total of 1,878 patient images, of which 406 are malignant tumor images, 592 are benign tumor images, and 880 are normal patient images. All results are confirmed by biopsy results or follow-up results. During the training process, 200 tumor images and 880 normal tissue images were used as test sets, 98 tumor images were used as validation sets, and the remaining 700 tumor images were used as training sets. The ratio of benign tumors in the test set and validation set is 1:1. The public dataset is BrEaST with tumor grade labels, which is used as an external test set. The dataset comes from 256 patients, including 98 malignant tumors, 154 benign tumors, and 4 normal tissue images. All tumors have been confirmed by biopsy or follow-up care results. The segmentation network is trained using breast region labels (the labels are pixel-level labels with two values, corresponding to breast and non-breast regions respectively). The labels are manually annotated original breast ultrasounds by junior doctors and corrected by senior doctors.

[0085] Specifically, in one embodiment, in order to balance the data set, the number of malignant tumor images in the training set is first expanded to the same number as the number of benign tumor images by using spatial filtering, frequency filtering, and adding random noise. The labels of the expanded images are kept synchronized with the corresponding categories. During the training process, data enhancement methods such as flipping, rotating, adding noise, Cutout and Mosica are used to help network training.

[0086] Specifically, according to actual needs, the size of the original breast ultrasound image can be adjusted to 224*224, and then the U-Net is trained. The SGD optimizer is used, the momentum is set to 0.937, and the weight_decay is set to 5e-4. The initial learning rate is 0.01, the termination learning rate is 0.0001, and the linear descent strategy is used to adjust the learning rate during training. The loss function of U-Net training is the sum of two loss functions:

[0087] L seg-train =αL dice +βL bce (1)

[0088] Where L dice represents the Dice loss function, L bce represents the BCE loss function, where α is 0.4 and β is 0.6.

[0089] Furthermore, three types of segmented images are obtained through the segmentation model, and the target detection model is further trained in the network in combination with the category labels corresponding to the segmented images. Specifically, according to the size design of the target detection model and the actual design needs, the image is optionally adjusted to 512*512 by bilinear interpolation. Of course, the present invention is not limited to this size, and the size setting of the actual ultrasound image can be dynamically adjusted according to actual needs.

[0090] Specifically, all original breast ultrasound images in the training set are uniformly input into the segmentation model, and the mask image output by U-Net is uniformly fused with the original breast ultrasound image to obtain the fused image; each fused image is input into the target detection network for training. The settings of the optimizer and learning rate used in training are the same as those of the U-net network. The loss function of the target detection model training is the sum of two classification losses and one regression loss:

[0091] L detect-train =L obj-focal +L cls-bce +L giou (2)

[0092] Where L giourepresents the GIOU regression loss for adjusting the tumor boundary, L cls-bce represents the BCE classification loss adjusted for tumor categories, L obj-focal Represents the Focal classification loss of whether an object is contained. The Focal loss function is as follows:

[0093] L obj-focal = -α(1-p t ) γ log(p t ) (3)

[0094] Among them, α=0.25, γ=2, p t When the label is 1, it is equal to the network prediction probability p, and when the label is 0, p t =1-p.

[0095] The present invention is based on the principle of programmable gradient information and utilizes reversible network branches to assist network training, thereby improving the network's ability to identify breast tumors, reducing false positive rates, and improving benign and malignant identification rates. Specifically, during training, the characteristic parameters output by the backbone extraction network are input into the reversible network branches for training, and the backbone extraction network is given more gradient information through the reversible network branches to assist in the training of the backbone extraction network to obtain better weight parameters. Removing the reversible network branches during prediction does not affect the results and speed of the original network prediction.

[0096] The ultrasonic breast tumor recognition method based on prior anatomical knowledge mainly reduces the false positive rate of ultrasonic breast tumor detection, while ensuring the accuracy and detection rate of tumor recognition. In order to achieve this goal, the present invention integrates the prior knowledge of breast anatomy into the target detection algorithm to improve the existing target detection algorithm. Specifically, the present invention uses U-Net to segment the image to obtain a breast area image, and then the breast area image and the original image are channel-fused. Such an image preprocessing method can greatly reduce the interference of non-tumor areas and reduce the false positive rate of tumor recognition. This preprocessing method also shows that the prior knowledge of anatomy can improve the performance of the breast tumor detection algorithm. At the same time, the present invention uses large-scale convolution and programming gradient information to improve the target detection model, which can better obtain tumors and surrounding information of different scales and improve the effect of tumor detection. Finally, we have proved the effectiveness and superiority of our algorithm through a large number of experiments, and it has broad application prospects.

[0097] Embodiment 2

[0098] The present invention also provides another embodiment, which proposes an ultrasonic breast tumor identification device based on prior anatomical knowledge, comprising:

[0099] A segmentation module is used to segment the original breast ultrasound image to obtain a mask image (mask) of the breast area;

[0100] A first preprocessing module, used for preprocessing the mask image of the breast area;

[0101] The second preprocessing module is used to perform channel fusion on the mask image of the breast area and the original breast ultrasound image to obtain a fused image; specifically comprising: calculating the average value of the three RGB channels in the original breast ultrasound image to obtain the original single-channel breast ultrasound image; in the original single-channel breast ultrasound image, keeping the pixel value unchanged according to the position with 1 in the mask image mask, and replacing the position with 0 in the mask with the average value of all pixel values ​​in the original single-channel breast ultrasound image to obtain a fused single-channel breast ultrasound image; replacing the G channel of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain a fused image;

[0102] The prediction module is used to input the fused image into the target detection model for detection to obtain a tumor detection result.

[0103] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiment 1, and will not be repeated here.

[0104] Embodiment 3

[0105] The present invention also provides another embodiment. The present invention proposes an ultrasound imaging display system, which includes an ultrasound probe, a display, and the ultrasound breast tumor recognition device based on prior anatomical knowledge as described above, wherein the segmentation module of the ultrasound breast tumor recognition device based on prior anatomical knowledge is electrically connected to the ultrasound probe, and the display is electrically connected to the prediction module of the ultrasound breast tumor recognition device based on prior anatomical knowledge, and is characterized in that it includes:

[0106] The ultrasonic probe is used to collect original breast ultrasonic images in real time;

[0107] The ultrasonic breast tumor recognition device based on prior anatomical knowledge is used to receive the original breast ultrasound image and output the tumor detection result of the original breast ultrasound image;

[0108] The display comprises a first display area and a second display area, wherein the first display area is used to display the real-time breast ultrasound image;

[0109] The second display area is used to display the tumor detection result in real time.

[0110] In the embodiment of the present invention, the specific structure and working process of the ultrasonic breast tumor identification device based on prior anatomical knowledge are consistent with those in the above two embodiments, and will not be repeated here.

[0111] In an embodiment of the present invention, exemplarily, the first display area and the second area have a linkage relationship. According to the original breast ultrasound image acquired in real time by the ultrasound probe, the first display area displays the breast ultrasound image in real time, and the second display area synchronously displays the breast ultrasound image and superimposes the output tumor detection results on the image, which can help doctors determine in real time the benign or malignant results of the breast area tumor corresponding to the currently acquired breast ultrasound image.

[0112] For example, the tumor detection results may mainly include the output results of malignant tumors, benign tumors, and normal patients. In another embodiment, the breast area corresponding to the above results in the original breast ultrasound image may be displayed in the form of an output box or a positioning box. Fig.14 As shown, the tumor area is marked and displayed in the form of a positioning frame. In one embodiment, the x, y, w, h of the breast tumor positioning frame, the confidence of the positioning frame containing the object, and the probability of benign and malignant categories can also be displayed beside the positioning frame. In another embodiment, the tumor detection result can also be the x, y, w, h of the breast tumor positioning frame, the confidence of the positioning frame containing the object, and the probability of benign and malignant categories.

[0113] Through the implementation of the above embodiments, doctors can be assisted in adjusting the direction and movement angle of the probe, which helps doctors to accurately determine whether a tumor exists in the breast area and timely assist in the benign or malignant judgment of the tumor.

[0114] Embodiment 4

[0115] The present invention also provides another implementation mode. The present invention proposes an electronic device, including: a processor 1 and a memory 2.

[0116] The memory 2 is used to store computer programs.

[0117] The memory 2 includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0118] The processor 1 is connected to the memory 2 and is used to execute the computer program stored in the memory 2, so that the processor implements the above-mentioned ultrasonic breast tumor recognition method based on prior anatomical knowledge when executing the computer program.

[0119] Preferably, the processor 1 may be a central processing unit (CPU for short); or an application specific integrated circuit (ASIC for short).

[0120] Embodiment 5

[0121] The present invention also provides another embodiment, namely, providing a computer-readable storage medium, which stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the ultrasonic breast tumor identification method based on prior anatomical knowledge as described above.

[0122] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0123] Those of ordinary skill in the art will appreciate that the modules, units and / or method steps of the various embodiments described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0124] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device or system, or some features can be ignored or not executed.

[0125] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0126] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for ultrasonic breast tumor recognition based on prior anatomical knowledge, characterized in that: The method comprises the steps of: Step S100, using a segmentation model to segment the original breast ultrasound image to obtain a mask image (mask) of the breast area; Step S200, performing channel fusion on the mask image of the breast region and the original breast ultrasound image to obtain a fused image, specifically includes: S210, calculating the average values ​​of the three RGB channels in the original breast ultrasound image to obtain an original single-channel breast ultrasound image; S220, in the original single-channel breast ultrasound image, keep the pixel value unchanged according to the position with 1 in the mask image mask, and replace the position with 0 in the mask with the mean value of all pixel values ​​in the original single-channel breast ultrasound image to obtain a fused single-channel breast ultrasound image; S230, replace one of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain a fused image; Step S300: input the fused image into a target detection model for detection to obtain a tumor detection result.

2. The ultrasonic breast tumor recognition method based on prior anatomical knowledge according to claim 1 is characterized in that: Before performing channel fusion on the mask image of the breast region and the original breast ultrasound image to obtain a fused image, the method may further include the steps of: Step S110: performing a morphological opening operation on the mask image of the breast region once, and then performing two dilation operations to obtain a preprocessed mask image (mask) of the breast region.

3. The ultrasonic breast tumor recognition method based on prior anatomical knowledge according to claim 1 or 2, characterized in that: The step S200 may also be: obtaining a fused image by adding pixel values ​​of one of the three channels of the mask image of the breast area and the original breast ultrasound image, wherein the channel is one of the R channel, the G channel, and the B channel.

4. The ultrasonic breast tumor recognition method based on prior anatomical knowledge according to claim 1 is characterized in that: The target detection model includes a backbone extraction network, an enhanced feature extraction network, a feature decoupling head, and a programmable gradient information branch.

5. The ultrasonic breast tumor recognition method based on prior anatomical knowledge according to claim 4 is characterized in that: Before performing the ultrasonic breast tumor recognition method based on prior anatomical knowledge, the method includes step S400 of training the segmentation model and the target detection model using a training data set.

6. An ultrasonic breast tumor recognition device based on prior anatomical knowledge, which specifically comprises: A segmentation module is used to segment the original breast ultrasound image to obtain a mask image (mask) of the breast area; A first preprocessing module, used for preprocessing the mask image of the breast area; The second preprocessing module is used to perform channel fusion on the mask image of the breast area and the original breast ultrasound image to obtain a fused image; specifically comprising: calculating the average value of the three RGB channels in the original breast ultrasound image to obtain the original single-channel breast ultrasound image; in the original single-channel breast ultrasound image, keeping the pixel value unchanged at the position with a value of 1 in the mask image mask, and replacing the position with a value of 0 in the mask with the average value of all pixel values ​​in the original single-channel breast ultrasound image to obtain a fused single-channel breast ultrasound image; replacing one of the three RGB channels in the original breast ultrasound image with the fused single-channel breast ultrasound image to obtain a fused image; The prediction module is used to input the fused image into the target detection model for detection to obtain a tumor detection result.

7. An ultrasonic imaging display system, comprising an ultrasonic probe, a display and the ultrasonic breast tumor recognition device based on prior anatomical knowledge as described above, wherein the segmentation module of the ultrasonic breast tumor recognition device based on prior anatomical knowledge is electrically connected to the ultrasonic probe, and the display is electrically connected to the prediction module of the ultrasonic breast tumor recognition device based on prior anatomical knowledge, characterized in that: include: The ultrasonic probe is used to collect original breast ultrasonic images in real time; The ultrasonic breast tumor recognition device based on prior anatomical knowledge is used to receive the original breast ultrasound image and output the tumor detection result of the original breast ultrasound image; The display comprises a first display area and a second display area, wherein the first display area is used to display the real-time breast ultrasound image; The second display area is used to display the tumor detection result in real time.

8. The ultrasonic imaging display system according to claim 7, characterized in that: The breast ultrasound image is synchronously displayed in the second display area and the tumor detection result is superimposed and output on the image.

9. An electronic device, characterized in that: It includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include steps for executing the ultrasonic breast tumor recognition method based on prior anatomical knowledge as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by the processor, implements the step of ultrasonic breast tumor identification based on prior anatomical knowledge as described in any of claims 1-5, wherein the second display area will synchronously display the breast ultrasound image and superimpose the output tumor detection result on the image.

Images