A method and related device for segmenting breast nodules

Abstract

The present application discloses a method for segmenting breast nodules and a related device. The method includes inputting a breast image to be segmented into a feature module, determining an initial prediction probability map and a target feature map of the breast image through the feature module; inputting the initial prediction probability map into the edge rendering module, controlling the edge rendering module to determine an uncertainty map corresponding to the breast image based on the initial prediction probability, obtaining a number of problem pixels in the uncertainty map, predicting a candidate prediction probability of each problem pixel based on the initial prediction probability map and the target feature map, and determining a target area corresponding to the breast image based on the candidate prediction probability of each problem pixel and the initial prediction probability map. The present application improves the prediction accuracy of the problem pixels by obtaining an uncertainty map of the initial prediction probability map, selecting a number of problem pixels based on the uncertainty map, and then re-predicting the problem pixels, thereby improving the accuracy of breast nodule segmentation.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing, and in particular to a breast nodule segmentation method and related devices. Background Art

[0002] Early diagnosis of breast cancer is key to improving cure rates and reducing breast cancer mortality. Ultrasound imaging is widely used in clinical breast cancer testing due to its real-time, radiation-free, and low-cost nature. Numerous studies have shown that the contour and texture characteristics of tumors in two-dimensional ultrasound images are important indicators for physicians to determine whether a tumor is benign or malignant.

[0003] Currently, the benign or malignant nature of a tumor is generally determined by experienced doctors who directly examine the image and extract information about the tumor area based on internal echoes, shape, boundaries, and other features. However, manually demarcating the boundaries of the tumor area in breast ultrasound images is a very time-consuming and highly repetitive task. In addition, due to differences in clinical experience, different doctors are prone to inter-observer differences in the determination of nodule areas. Therefore, it is very necessary to simplify labor costs and reduce the misdiagnosis rate through computer-aided diagnosis. Among them, breast lesion segmentation based on ultrasound images is a commonly used basic preprocessing step for computer-aided diagnosis and quantitative analysis of breast cancer. By automatically obtaining the boundaries of the tumor and assisting doctors in automatically measuring the size of the tumor, it is the key to determining whether the tumor is benign or malignant.

[0004] Currently, the segmentation methods for breast ultrasound images mainly include the following categories: (1) Compared with traditional machine learning methods, including threshold-based methods, contour-based methods, and region-based methods, these methods only consider the shallow visual information of the image, such as brightness, edges, texture, contours, etc. When the boundaries of the nodule area in the image are discontinuous or unclear, the shallow image features often cannot accurately represent the target area. (2) Deep learning-based methods, especially convolutional neural networks (CNNs), use convolution operators to extract high-level semantic information from the input image and achieve segmentation tasks by predicting the category of pixel points. However, due to the repeated downsampling operations used in classic CNN networks, image segmentation tasks are often accompanied by the loss of position information, and it is difficult to obtain a sufficiently large output resolution, which in turn affects the prediction accuracy.

[0005] Therefore existing technology still needs to be improved and improved. Summary of the Invention

[0006] The technical problem to be solved by this application is to provide a breast nodule segmentation method and related devices in response to the deficiencies of the existing technology.

[0007] In order to solve the above technical problems, a first aspect of an embodiment of the present application provides a method for segmenting breast nodules. The method applies a segmentation model, and the segmentation model includes a feature module and an edge rendering module. The method includes:

[0008] Inputting the mammary gland image to be segmented into the feature module, and determining an initial prediction probability map and a target feature map of the mammary gland image by the feature module;

[0009] The initial prediction probability map and the target feature map are input into the edge rendering module, the edge rendering module is controlled to determine the uncertainty map corresponding to the breast image based on the initial prediction probability map, a number of problem pixels in the uncertainty map are obtained, the candidate prediction probability of each problem pixel is predicted based on the initial prediction probability map and the target feature map, and the target area corresponding to the breast image is determined based on the candidate prediction probability of each problem pixel and the initial prediction probability map, wherein the uncertainty map is used to reflect the credibility of the initial prediction probability map.

[0010] The method for segmenting breast nodules, wherein the feature module includes a feature extraction unit, a fusion unit, and an initial prediction unit; inputting the breast image to be segmented into the feature module, and determining the initial prediction probability map and the target feature map of the breast image by the feature module specifically includes:

[0011] Inputting the breast image to be segmented into a feature extraction unit, and determining a plurality of first feature maps of the breast image by the feature extraction unit, wherein the plurality of feature maps have different image sizes;

[0012] Inputting a plurality of first feature maps into a fusion unit, and determining a second feature map through the fusion unit;

[0013] The second feature map is input into the initial prediction unit, and the initial prediction probability map of the breast image is determined by the initial prediction unit.

[0014] In the breast nodule segmentation method, inputting the plurality of first feature maps into a fusion unit and determining the second feature map by the fusion unit specifically includes:

[0015] Inputting the plurality of first feature maps into a fusion unit, and concatenating the plurality of first feature maps through the fusion unit to obtain a third feature map;

[0016] A plurality of fourth feature maps are determined by the fusion unit and the third feature map, and the plurality of fourth feature maps are concatenated to obtain a second feature map, wherein the receptive fields of the fourth feature maps are different.

[0017] In the breast nodule segmentation method, the step of controlling the edge rendering module to determine the uncertainty map corresponding to the breast image based on the initial prediction probability map specifically includes:

[0018] Obtaining the predicted probability of each pixel in each channel in the initial predicted probability map;

[0019] For each pixel, select the maximum prediction probability and the second largest prediction probability from all prediction probabilities corresponding to the pixel, and calculate the uncertainty of the pixel based on the maximum prediction probability and the second largest prediction probability;

[0020] An uncertainty map is formed based on the uncertainty of each pixel in the initial prediction probability map.

[0021] In the breast nodule segmentation method, the step of obtaining several problematic pixels in the uncertainty map is as follows:

[0022] In the uncertainty map, several problem pixels are selected in descending order of uncertainty.

[0023] In the breast nodule segmentation method, during the training of the segmentation model, obtaining a number of problematic pixels in the uncertainty map specifically includes:

[0024] Randomly selecting a first number of first pixel points in the uncertainty map, and sequentially selecting a second number of second pixel points from the first number of first pixel points in descending order of uncertainty;

[0025] A third number of third pixel points are selected from the uncertain map, and the second number of second pixel points and the third number of third pixel points selected are used as several problem pixel points in the uncertain map, wherein the number of the several problem pixel points is less than the first number.

[0026] In the breast nodule segmentation method, the step of predicting the candidate prediction probability of each problem pixel based on the initial prediction probability map and the target feature map specifically includes:

[0027] Selecting fine-grained feature points corresponding to each problem pixel in the target feature map;

[0028] Determine the corrected feature points corresponding to each problem pixel based on the fine-grained feature points corresponding to each problem pixel and the initial prediction probability corresponding to each problem pixel;

[0029] Based on the corresponding correction feature points of each problem pixel, the candidate prediction probability of each problem pixel is predicted.

[0030] A second aspect of an embodiment of the present application provides a breast nodule segmentation device, the segmentation device comprising:

[0031] a feature module for determining an initial prediction probability map based on the breast image;

[0032] An edge rendering module is used to determine the uncertainty map corresponding to the breast image based on the initial prediction probability block, obtain several problem pixels in the uncertainty map, predict the candidate prediction probability of each problem pixel based on the initial prediction probability map and the target feature map, and determine the target area corresponding to the breast image based on the candidate prediction probability of each problem pixel and the initial prediction probability map.

[0033] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in any of the breast nodule segmentation methods described above.

[0034] A fourth aspect of an embodiment of the present application provides a terminal device, comprising: a processor, a memory, and a communication bus; the memory stores a computer-readable program executable by the processor;

[0035] The communication bus realizes the connection and communication between the processor and the memory;

[0036] When the processor executes the computer-readable program, the steps in any of the above-described methods for segmenting breast nodules are implemented.

[0037] Beneficial effect: Compared with the prior art, the present application provides a method for segmenting breast nodules and related devices, the method comprising inputting the breast image to be segmented into the feature module, determining the initial prediction probability map and the target feature map of the breast image through the feature module; inputting the initial prediction probability map into the edge rendering module, controlling the edge rendering module to determine the uncertainty map corresponding to the breast image based on the initial prediction probability map, obtaining several problem pixels in the uncertainty map, predicting the candidate prediction probability of each problem pixel based on the initial prediction probability map and the target feature map, and determining the target area corresponding to the breast image based on the candidate prediction probability of each problem pixel and the initial prediction probability map. The present application can improve the prediction accuracy of the problem pixels by obtaining the uncertainty map of the initial prediction probability map, selecting several problem pixels based on the uncertainty map, and then re-predicting the problem pixels, thereby improving the accuracy of breast nodule segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without inventive work.

[0039] Figure 1 This is a flow chart of the breast nodule segmentation method provided in this application.

[0040] Figure 2 This is a flow chart of the principle of the breast nodule segmentation method provided in this application.

[0041] Figure 3 This is a schematic diagram of the structural principle of the breast nodule segmentation device provided in this application.

[0042] Figure 4 This is a schematic diagram of the structure of the terminal device provided in this application. DETAILED DESCRIPTION

[0043] This application provides a method and related apparatus for segmenting breast nodules. To make the purpose, technical solutions, and effects of this application more clear and explicit, the application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to explain this application and are not intended to limit this application.

[0044] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0046] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0047] The inventors have discovered through research that early diagnosis of breast cancer is key to improving cure rates and reducing breast cancer mortality. Ultrasound imaging is widely used in clinical breast cancer testing due to its real-time, radiation-free, and low-cost nature. Numerous studies have shown that the contour and texture characteristics of tumors in two-dimensional ultrasound images are important indicators for physicians to determine whether a tumor is benign or malignant.

[0048] Currently, the benign or malignant nature of a tumor is generally determined by experienced doctors who directly examine the image and extract information about the tumor area based on internal echoes, shape, boundaries, and other features. However, manually demarcating the boundaries of the tumor area in breast ultrasound images is a very time-consuming and highly repetitive task. In addition, due to differences in clinical experience, different doctors are prone to inter-observer differences in the determination of nodule areas. Therefore, it is very necessary to simplify labor costs and reduce the misdiagnosis rate through computer-aided diagnosis. Among them, breast lesion segmentation based on ultrasound images is a commonly used basic preprocessing step for computer-aided diagnosis and quantitative analysis of breast cancer. By automatically obtaining the boundaries of the tumor and assisting doctors in automatically measuring the size of the tumor, it is the key to determining whether the tumor is benign or malignant.

[0049] Currently, the segmentation methods for breast ultrasound images mainly include the following categories: (1) Compared with traditional machine learning methods, including threshold-based methods, contour-based methods, and region-based methods, these methods only consider the shallow visual information of the image, such as brightness, edges, texture, contours, etc. When the boundaries of the nodule area in the image are discontinuous or unclear, the shallow image features often cannot accurately represent the target area. (2) Deep learning-based methods, especially convolutional neural networks (CNNs), use convolution operators to extract high-level semantic information from the input image and achieve segmentation tasks by predicting the category of pixel points. However, due to the repeated downsampling operations used in classic CNN networks, image segmentation tasks are often accompanied by the loss of position information, and it is difficult to obtain a sufficiently large output resolution, which in turn affects the prediction accuracy.

[0050] In order to solve the above problems, in an embodiment of the present application, the breast image to be segmented is input into the feature module, and the initial prediction probability map and the target feature map of the breast image are determined by the feature module; the initial prediction probability map is input into the edge rendering module, and the edge rendering module is controlled to determine the uncertainty map corresponding to the breast image based on the initial prediction probability map, obtain several problem pixels in the uncertainty map, predict the candidate prediction probability of each problem pixel based on the initial prediction probability map and the target feature map, and determine the target area corresponding to the breast image based on the candidate prediction probability of each problem pixel and the initial prediction probability map. The present application improves the prediction accuracy of the problem pixels by obtaining the uncertainty map of the initial prediction probability map, selecting several problem pixels based on the uncertainty map, and then re-predicting the problem pixels, thereby improving the accuracy of breast nodule segmentation.

[0051] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0052] This embodiment provides a method for segmenting breast nodules. The method utilizes a segmentation model, which is a trained neural network model, to segment breast images to identify nodule regions within the breast images. The segmentation model may include a feature module and an edge rendering module, wherein the feature module is connected to the edge rendering module. The feature module receives as input the breast image to be segmented, and outputs the feature module including an initial predicted probability map and a target feature map. The edge rendering module receives as input the initial predicted probability map and the target feature map determined by the feature module, and outputs the target region.

[0053] This embodiment provides a method for segmenting breast nodules. Figure 1 and 2 As shown, the method includes:

[0054] S10. Inputting the breast image to be segmented into the feature module, and determining an initial prediction probability map and a target feature map of the breast image through the feature module.

[0055] Specifically, the breast image is a breast ultrasound image. The breast image may be acquired by an ultrasound device connected to an electronic device running the method for segmenting breast nodules provided in this embodiment, or may be sent by an external device to the electronic device running the method for segmenting breast nodules provided in this embodiment, or may be acquired via a network. Of course, in actual applications, the ultrasound device may also be directly configured with the method for segmenting breast nodules provided in this embodiment. When acquiring a breast ultrasound image, the ultrasound device may directly segment the breast ultrasound image to obtain a nodule mask image, and then synchronously display the nodule mask image and the breast ultrasound image.

[0056] The feature module is used to provide the edge rendering module with a target feature map and an initial prediction probability map. That is, after the breast image to be segmented is input into the feature module, the feature module will determine the target feature map and the initial prediction probability map based on the breast image, and use the target feature map and the initial prediction probability map as input items of the edge rendering module, so that the edge rendering module corrects the prediction probabilities of some pixels in the prediction probability map to improve the segmentation accuracy of the breast image. The feature module may include a deep convolutional neural network. During the training process of the feature module, the input items of the feature module may be training breast images and nodule annotated areas. The feature module sequences the training breast images and nodule annotated areas to achieve feature extraction of unlabeled breast images and determine the initial prediction probability map.

[0057] In one implementation of this embodiment, Figure 2 As shown, the feature module includes a feature extraction unit, a fusion unit and an initial prediction unit. The breast image to be segmented is input into the feature module, and the initial prediction probability map and the target feature map of the breast image are determined by the feature module. Specifically, the steps include:

[0058] Inputting the mammary gland image to be segmented into a feature extraction unit, and determining a plurality of first feature maps and a target feature map of the mammary gland image by the feature extraction unit;

[0059] Inputting a plurality of first feature maps into a fusion unit, and determining a second feature map through the fusion unit;

[0060] The second feature map is input into the initial prediction unit, and the initial prediction probability map of the breast image is determined by the initial prediction unit.

[0061] Specifically, the target feature map is a feature map determined in the process of the feature extraction unit acquiring a plurality of first feature maps. The target feature map may be included in the plurality of first feature maps or may not be included in the plurality of first feature maps. The feature extraction unit is used to determine a plurality of first feature maps of a breast image, and each feature map in the plurality of first feature maps has a different image size. It is understandable that each feature map is obtained by outputting a different network layer of the feature extraction unit, so that the feature dimensions of each feature map are different. For example, Figure 2 As shown, the feature extraction unit uses the ResNet network, and the first feature maps are the features of the first 7*7 convolution block of the ResNet network, the output features of the first, second, and fourth stacking layers (i.e. Figure 2 64-dimensional, 256-dimensional, 512-dimensional, and 2048-dimensional features in

[15] .

[0062] The second feature map is high-order feature information determined by the fusion unit based on the first feature maps. The second feature map integrates multi-layer and multi-scale features, so that the second feature map carries multi-level and multi-scale information of the breast image, thereby improving the accuracy of the initial prediction probability map. In one implementation of this embodiment, the inputting of the first feature maps into the fusion unit and the determination of the second feature map by the fusion unit specifically include:

[0063] Inputting the plurality of first feature maps into a fusion unit, and concatenating the plurality of first feature maps through the fusion unit to obtain a third feature map;

[0064] A plurality of fourth feature maps are determined by the fusion unit and the third feature map, and the plurality of fourth feature maps are concatenated to obtain a second feature map, wherein the receptive fields of the fourth feature maps are different.

[0065] Specifically, since the image sizes of the several first feature maps are different, before splicing the several first feature maps, it is necessary to adjust the image sizes of the several first feature maps so that the image sizes of the adjusted first feature maps are the same. For example, the image size of each first feature map is adjusted to a preset image size, or one feature map is selected from each first feature map as the target feature map, and the image sizes of the other feature maps are adjusted to the image size of the target feature map, wherein the target feature map is any feature map among the several first feature maps. In addition, after adjusting the image sizes of the first feature maps to be the same, the first feature maps are spliced in the channel direction to obtain a third feature map. That is, the number of channels of the third feature map is equal to the sum of the number of channels of the first feature maps.

[0066] After obtaining the third feature map, the third feature map is convolved to obtain the fifth feature map, and then several different operations are performed on the fifth feature map to obtain several fourth feature maps with different receptive fields. For example, the fifth feature map is convolved using 1*1 ordinary convolution, 3*3 atrous convolution with a hole rate of 12, 3*3 atrous convolution with a hole rate of 24, and 3*3 atrous convolution with a hole rate of 36, respectively, to obtain four fourth feature maps under different receptive fields. In addition, after obtaining several fourth feature maps, the several fourth feature maps are spliced in a channel manner, and the spliced feature maps are convolved to obtain the second feature map, wherein 1*1 ordinary convolution can be used when performing the convolution operation on the spliced feature maps. In this way, the second feature map can obtain more multi-level and multi-scale information that is highly important to breast nodules, so that the accuracy of the initial prediction unit in determining the initial prediction probability map of the breast image can be improved.

[0067] The initial prediction unit is used to predict the initial prediction probability map of the breast image based on the second feature map, wherein the initial prediction unit may include a convolution layer, the second feature map is input into the convolution layer, and the initial prediction probability map of the breast image is output through the convolution layer. In a specific implementation, Figure 2 As shown, the initial prediction unit can use a 3*3 ordinary convolution.

[0068] S20. Input the initial prediction probability map into the edge rendering module, control the edge rendering module to determine the uncertainty map corresponding to the breast image based on the initial prediction probability map, obtain several problem pixels in the uncertainty map and predict the candidate prediction probability of each problem pixel based on the initial prediction probability map and the target feature map, and determine the target area corresponding to the breast image based on the candidate prediction probability of each problem pixel and the initial prediction probability map.

[0069] Specifically, the edge rendering module is used to correct the prediction probabilities of some pixels in the initial prediction probability map based on the target feature map and the initial prediction probability map, and determine the target area corresponding to the breast image based on the corrected initial prediction probability map. The uncertainty map is used to reflect the credibility of the initial prediction probability map, wherein the number of channels in the uncertainty map is 1, the image size is equal to the image size of the initial prediction probability, and the value of each pixel in the uncertainty map is used to reflect the credibility of the pixel corresponding to the pixel in the initial prediction probability map. In other words, the uncertainty map reflects the credibility of each pixel in the initial prediction probability map.

[0070] In an implementation of this embodiment, controlling the edge rendering module to determine the uncertainty map corresponding to the breast image based on the initial prediction probability map specifically includes:

[0071] Obtaining the predicted probability of each pixel in each channel in the initial predicted probability map;

[0072] For each pixel, select the maximum prediction probability and the second largest prediction probability from all prediction probabilities corresponding to the pixel, and calculate the uncertainty of the pixel based on the maximum prediction probability and the second largest prediction probability;

[0073] An uncertainty map is formed based on the uncertainty of each pixel in the initial prediction probability map.

[0074] Specifically, the initial prediction probability map includes several channels, and the feature module is configured with several default categories. The channels correspond one-to-one to the default categories, and each channel is used to reflect the predicted probability that the predicted category of a pixel in the breast image is the default category corresponding to that channel. That is, for a pixel, the predicted probability of the pixel in each channel is the probability that the pixel belongs to the default category corresponding to each channel. The predicted probability of the pixel in each channel is between 0 and 1, and the sum of the predicted probabilities of the pixel in each channel is 1. Therefore, after obtaining the initial prediction probability map, the predicted probability of each pixel in each channel can be obtained to obtain the predicted probability of the pixel in each default category. For example, if the default categories configured by the feature module include foreground and background categories, and the initial prediction probability map has two channels, then pixel A can obtain predicted probability a and predicted probability b. Prediction probability a corresponds to the foreground category, and prediction probability b corresponds to the background category. Prediction probability a reflects the probability that pixel A is foreground, and prediction probability b reflects the probability that pixel A is background.

[0075] After obtaining the predicted probabilities for each pixel, the predicted probabilities are arranged in descending order to obtain a predicted probability sequence. The first two predicted probabilities in the predicted probability sequence are then selected to obtain the maximum predicted probability and the number predicted probability. It is understood that the maximum predicted probability is the maximum value among the predicted probabilities, and the second-largest predicted probability is the second-largest value among the predicted probabilities. After obtaining the maximum and second-largest predicted probabilities, uncertainty can be calculated based on the maximum and second-largest predicted probabilities. The uncertainty can be equal to the product of the difference between the maximum and second-largest predicted probabilities and negative one. In other words, the uncertainty is equal to the second-largest predicted probability minus the maximum predicted probability. This embodiment calculates uncertainty using the maximum and second-largest predicted probabilities so that the uncertainty can represent the confidence level of the predicted category corresponding to the pixel. A higher uncertainty indicates a lower confidence level of the predicted category corresponding to the pixel, and conversely, a lower uncertainty indicates a higher confidence level of the predicted category corresponding to the pixel. Of course, in practical applications, other methods can be used to calculate uncertainty, such as using the maximum prediction probability and the third largest prediction probability, or the maximum prediction probability and the minimum prediction probability, or directly using the probability of the predicted category corresponding to the pixel as the uncertainty. The uncertainty determination process provided in this embodiment is a typical implementation method, which can more accurately reflect the credibility of the predicted category corresponding to the pixel.

[0076] After obtaining the uncertainty corresponding to each pixel point, each uncertainty is arranged according to the position of the corresponding pixel point in the initial prediction probability map to form an uncertainty map, so that each pixel point in the uncertainty map is used to reflect the credibility of the pixel point corresponding to the pixel point in the initial prediction probability map, wherein the image size of the uncertainty map is equal to the image size of the initial prediction probability map.

[0077] The method for obtaining several problem pixels during the testing and use of the segmentation model is the same as the method for obtaining the problem pixels during the training process of the segmentation model. During the testing and use process, several problem pixels with high uncertainty can be selected. Therefore, the acquisition of several problem pixels in the uncertainty map is specifically as follows: several problem pixels are selected in the uncertainty map in descending order of uncertainty. That is to say, after obtaining the uncertainty of each pixel in the initial prediction probability map, each pixel can be sorted from high to low according to the uncertainty, and then the problem pixels can be selected from front to back to obtain several problem pixels.

[0078] During the training process of the segmentation model, obtaining a number of problem pixels in the uncertainty map specifically includes:

[0079] Randomly selecting a first number of first pixel points in the uncertainty map, and sequentially selecting a second number of second pixel points from the first number of first pixel points in descending order of uncertainty;

[0080] A third number of third pixel points are selected in the uncertainty map, and the second number of second pixel points and the third number of third pixel points selected are used as a number of problem pixel points in the uncertainty map.

[0081] Specifically, the first number is greater than or equal to the fourth number of the plurality of problem pixels to be selected, and the sum of the second number and the third number is equal to the fourth number of the plurality of problem pixels to be selected, wherein the second number is less than the fourth number, and the third number is less than the fourth number. Furthermore, the first number can be any number greater than or equal to the fourth number. In one implementation, the first number is a multiple of the fourth number. It is understood that the first number is equal to K * the fourth number N, where K is a preset parameter multiple and K is a positive integer.

[0082] The first number of first pixel points are randomly selected from the uncertainty map, and the second number of second pixel points are selected from the first number of first pixels, that is, any second pixel point is included in the first number of first pixel points, and there are pixel points in the first number of first pixel points that are not second pixel points. In this embodiment, the second number of second pixel points are selected in descending order of uncertainty. The selection process can be to sort the first number of pixel points in descending order of uncertainty to obtain a first pixel point sequence, and then select the first second number of first pixel points in the first pixel point sequence, and use the selected second number of first pixel points as the second number of second pixel points. After obtaining the second number of second pixel points, a third number of third pixel points are selected from the uncertainty map to obtain a number of problem pixel points, wherein the third number of third pixel points can be obtained by uniform sampling, and the third number of third pixel points can include pixel points included in the second number of second pixel points.

[0083] For example: assuming that the breast image is a 256*256*3 breast ultrasound image, the breast image passes through the feature module to obtain an initial prediction probability map of 32*32*c, where c is the number of default categories configured by the feature module (for example, if only nodules and background areas in the ultrasound image are considered, c=2, if the nodules are classified as benign or malignant, then c=3), the fourth number of problem pixels N=256, the multiple K=2, and the ratio of the second preset number to the fourth number β=0.75. Then, after obtaining the 32*32 uncertainty map, 2*256 first pixels are randomly selected on the uncertainty map, and then 0.75*256 second pixels with the highest uncertainty are selected from the 2*256 first pixels, and then 0.25*256 third pixels are uniformly sampled from the entire uncertainty map to obtain the 256 most problematic pixels.

[0084] In addition, the number of problem pixels selected during the training process can be different from the number of problem pixels selected during the testing and verification process. The number of problem pixels selected during the training process can be greater than the number of problem pixels selected during the testing and verification process, so that the training process can quickly learn the characteristics of the problem pixels, thereby improving the model training speed. The number of problem pixels can be reduced during the testing and verification process, which can reduce the amount of computation of the edge rendering module.

[0085] In one implementation of this embodiment, predicting the candidate prediction probability of each problem pixel based on the initial prediction probability map and the target feature map specifically includes:

[0086] Selecting fine-grained feature points corresponding to each problem pixel in the target feature map;

[0087] Determine the corrected feature points corresponding to each problem pixel based on the fine-grained feature points corresponding to each problem pixel and the initial prediction probability corresponding to each problem pixel;

[0088] Based on the corresponding correction feature points of each problem pixel, the candidate prediction probability of each problem pixel is predicted.

[0089] Specifically, the target feature map is a feature map determined by the feature module in the process of obtaining the initial prediction probability map. The feature map can be any feature map extracted by the feature module. For example, the feature map is a second feature map obtained by the fusion module fusing several first feature maps, or one of several first feature maps, or a third feature map obtained by splicing several first feature maps. In one implementation of this embodiment, the feature module uses ResNet as the feature extraction unit in the feature module, and the target feature module is a feature map extracted by the second layer of ResNet. For example, the target feature map is Figure 2 The 512-dimensional feature map in the image is obtained by selecting the feature map output by the second layer as the target feature map. This embodiment enables the target feature map to carry high-order global semantic information. In actual experimental results, it is also proved that the feature map output by the second layer is more suitable for the existing ultrasound data. Of course, for different data, feature maps output by other layers can also be used, or feature maps of two or more layers can be spliced and fused to form a target feature map.

[0090] After obtaining the target feature map, the fine-grained feature points corresponding to each problem pixel are selected in the target feature map, wherein the pixel position of each problem pixel in the initial prediction probability map is the same as the pixel position of its corresponding fine-grained feature point in the target feature map. In other words, the fine-grained feature points corresponding to each problem pixel are selected based on the pixel position. In addition, before selecting the fine-grained feature points based on the pixel position, the image size of the target feature map can be adjusted to the image size of the prediction probability map, and then the fine-grained feature points corresponding to each problem pixel are selected. For example, the target feature map is Figure 2 In the 512-dimensional feature map, each problem pixel can be selected into a 1*1*512 fine-grained feature point.

[0091] The corrected feature points are obtained by combining the problem pixel points and the fine-grained feature points. That is to say, after obtaining the fine-grained feature points corresponding to each problem pixel point, the initial prediction probability corresponding to each problem pixel point and the fine-grained feature point corresponding to each problem pixel point are spliced in the channel direction to obtain the modified feature points, and then the prediction probability of the problem pixel point is predicted based on the modified feature point to obtain the candidate prediction probability of each problem pixel point, wherein the prediction probability of the problem pixel point based on the modified feature point can be predicted by a lightweight multi-layer perceptron, for example, a lightweight multi-layer perceptron including 1*1 convolution and ReLu activation.

[0092] In one implementation of this embodiment, when determining the target region corresponding to the breast image based on the candidate prediction probabilities of each problematic pixel and the initial prediction probability map, the candidate prediction probability of each problematic pixel is used to replace the initial prediction probability map corresponding to each problematic pixel in the initial prediction probability map to obtain a candidate prediction probability map. The target region corresponding to the breast image is then determined based on the candidate prediction probability map. Furthermore, during the training phase, due to the gradient backpropagation mechanism, the edge rendering module is only executed on the initial prediction probability map obtained through the feature module. This is because the gradient backpropagation mechanism during network training prevents multiple amplifications of the prediction results before sampling. Therefore, the edge rendering module is executed once on the initial prediction probability map during the training phase. During the testing phase, after obtaining the initial prediction probability map, the edge rendering module is executed on the initial prediction probability map. Furthermore, during the upsampling process of the initial prediction probability map, the edge rendering module is executed on each upsampled prediction probability map. This is because the gradient backpropagation problem is not involved, allowing for multiple cycles of amplification and re-prediction of point categories, further improving the accuracy of the predicted target region.

[0093] For example, suppose that during the test phase, the image size of a breast image is 256*256*3, the number of problem pixels N=4096, and the image size of the initial prediction probability map is 32*32. The initial prediction probability map is linearly upsampled by a factor of 2 three times to a prediction probability map of the same size as the breast image. After running the edge rendering module on the initial prediction probability map, the edge rendering module is run on the upsampled prediction probability map obtained after each upsampling. All pixels in the 64*64 upsampled prediction probability map are selected as problem pixels, and 4096 pixels in the 128*128 and 256*256 upsampled prediction probability maps are also selected as problem pixels.

[0094] In one implementation of this embodiment, the segmentation model can perform data preprocessing on the training images in the training sample set during training. The data preprocessing mainly includes standardization, normalization, histogram equalization, etc. For example, during image standardization, the image will be subtracted from the mean and then divided by the standard deviation to make the data conform to the standard normal distribution; during normalization, all pixels in the image will be subtracted from the minimum pixel value and then divided by the difference between the maximum pixel value and the minimum pixel value. The final pixel value range of the image will change from 0 to 255 to 0 to 1. The pixel value range of 0 to 1 is more suitable for subsequent neural network processing; and the role of histogram equalization is mainly to enhance the contrast of the image, making the image clearer, etc. In some cases, the image differences between the data are large and the distribution is uneven. If the data is not preprocessed, it may have a certain impact on subsequent training: for example, it may limit the accuracy, convergence speed, and generalization ability of the model. Of course, other preprocessing methods can also be used in actual applications, which will not be described one by one here.

[0095] In one implementation of this embodiment, after acquiring the target region, post-processing can be performed to make the predicted target region more closely aligned with clinical reality. This post-processing can include filling holes in the nodule region predicted by the framework and removing small connected regions in non-nodule regions. Of course, other post-processing methods can also be used in actual applications, which will not be detailed here.

[0096] In summary, this embodiment provides a method for segmenting breast nodules, which includes inputting the breast image to be segmented into the feature module, determining the initial prediction probability map and the target feature map of the breast image through the feature module; inputting the initial prediction probability map into the edge rendering module, controlling the edge rendering module to determine the uncertainty map corresponding to the breast image based on the initial prediction probability map, obtaining several problem pixels in the uncertainty map, predicting the candidate prediction probability of each problem pixel based on the initial prediction probability map and the target feature map, and determining the target area corresponding to the breast image based on the candidate prediction probability of each problem pixel and the initial prediction probability map. This application improves the prediction accuracy of the problem pixels by obtaining the uncertainty map of the initial prediction probability map, selecting several problem pixels based on the uncertainty map, and then re-predicting the problem pixels, thereby improving the accuracy of breast nodule segmentation.

[0097] Based on the above-mentioned breast nodule segmentation method, this embodiment provides a breast nodule segmentation device, such as Figure 3 As shown, the segmentation device includes:

[0098] A feature module 100 is configured to determine an initial prediction probability map based on the breast image;

[0099] The edge rendering module 200 is used to determine the uncertainty map corresponding to the breast image based on the initial prediction probability block, obtain several problem pixels in the uncertainty map, predict the candidate prediction probability of each problem pixel based on the initial prediction probability map and the target feature map, and determine the target area corresponding to the breast image based on the candidate prediction probability of each problem pixel and the initial prediction probability map.

[0100] Based on the above-mentioned breast nodule segmentation method, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the breast nodule segmentation method as described in the above-mentioned embodiment.

[0101] Based on the above-mentioned breast nodule segmentation method, the present application also provides a terminal device, such as Figure 4 As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0102] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0103] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0104] The memory 22 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal device. In addition, the memory 22 may include high-speed random access memory and non-volatile memory. For example, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

[0105] In addition, the specific process of loading and executing the multiple instructions in the storage medium and the processor in the terminal device has been described in detail in the above method and will not be described here one by one.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for segmenting breast nodules, characterized in that: The method applies a segmentation model, the segmentation model includes a feature module and an edge rendering module, and the method includes: Inputting the mammary gland image to be segmented into the feature module, and determining an initial prediction probability map and a target feature map of the mammary gland image by the feature module; Inputting the initial prediction probability map and the target feature map into the edge rendering module, controlling the edge rendering module to determine an uncertainty map corresponding to the breast image based on the initial prediction probability map, obtaining a number of problem pixels in the uncertainty map, predicting a candidate prediction probability for each problem pixel based on the initial prediction probability map and the target feature map, and determining a target area corresponding to the breast image based on the candidate prediction probability for each problem pixel and the initial prediction probability map, wherein the uncertainty map is used to reflect the credibility of the initial prediction probability map; The feature module includes a feature extraction unit, a fusion unit, and an initial prediction unit; the step of inputting the breast image to be segmented into the feature module and determining the initial prediction probability map and the target feature map of the breast image through the feature module specifically includes: Inputting the breast image to be segmented into a feature extraction unit, and determining a plurality of first feature maps of the breast image by the feature extraction unit, wherein the plurality of feature maps have different image sizes; Inputting a plurality of first feature maps into a fusion unit, and determining a second feature map through the fusion unit; Inputting the second feature map into the initial prediction unit, and determining an initial prediction probability map of the breast image by the initial prediction unit; The controlling the edge rendering module to determine the uncertainty map corresponding to the breast image based on the initial prediction probability map specifically includes: Obtaining the predicted probability of each pixel in each channel in the initial predicted probability map; For each pixel, select the maximum prediction probability and the second largest prediction probability from all prediction probabilities corresponding to the pixel, and calculate the uncertainty of the pixel based on the maximum prediction probability and the second largest prediction probability; An uncertainty map is formed based on the uncertainty of each pixel in the initial prediction probability map.

2. The method for segmenting breast nodules according to claim 1, wherein: Inputting the plurality of first feature maps into the fusion unit and determining the second feature map by the fusion unit specifically includes: Inputting the plurality of first feature maps into a fusion unit, and concatenating the plurality of first feature maps through the fusion unit to obtain a third feature map; A plurality of fourth feature maps are determined by the fusion unit and the third feature map, and the plurality of fourth feature maps are concatenated to obtain a second feature map, wherein the receptive fields of the fourth feature maps are different.

3. The method for segmenting breast nodules according to claim 1, wherein: The step of obtaining several problematic pixels in the uncertainty map is as follows: In the uncertainty map, several problem pixels are selected in descending order of uncertainty.

4. The method for segmenting breast nodules according to claim 1, wherein: During the training process of the segmentation model, obtaining a number of problem pixels in the uncertainty map specifically includes: Randomly selecting a first number of first pixel points in the uncertainty map, and sequentially selecting a second number of second pixel points from the first number of first pixel points in descending order of uncertainty; A third number of third pixel points are selected from the uncertain map, and the second number of second pixel points and the third number of third pixel points selected are used as several problem pixel points in the uncertain map, wherein the number of the several problem pixel points is less than the first number.

5. The method for segmenting breast nodules according to claim 1, wherein: The predicting of the candidate prediction probability of each problem pixel based on the initial prediction probability map and the target feature map specifically includes: Selecting fine-grained feature points corresponding to each problem pixel in the target feature map; Determine the corrected feature points corresponding to each problem pixel based on the fine-grained feature points corresponding to each problem pixel and the initial prediction probability corresponding to each problem pixel; Based on the corresponding correction feature points of each problem pixel, the candidate prediction probability of each problem pixel is predicted.

6. A breast nodule segmentation device, characterized in that: Applied to implement the steps in the breast nodule segmentation method according to any one of claims 1 to 5, the segmentation device comprises: a feature module for determining an initial prediction probability map based on the breast image; An edge rendering module is used to determine the uncertainty map corresponding to the breast image based on the initial prediction probability block, obtain several problem pixels in the uncertainty map, predict the candidate prediction probability of each problem pixel based on the initial prediction probability map and the target feature map, and determine the target area corresponding to the breast image based on the candidate prediction probability of each problem pixel and the initial prediction probability map.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the breast nodule segmentation method according to any one of claims 1 to 5.

8. A terminal device, characterized in that: include: processor, memory, and communication bus; The memory stores a computer-readable program executable by the processor; The communication bus realizes the connection and communication between the processor and the memory; When the processor executes the computer-readable program, the processor implements the steps in the breast nodule segmentation method according to any one of claims 1 to 5.

Images