A medical image segmentation method and system

Abstract

The present application proposes a medical image segmentation method and system, which relates to the field of image processing technology. The method comprises: obtaining a breast ultrasound image training set; each breast ultrasound image in the breast ultrasound image training set includes an associated lesion annotation image and a lesion prediction image, and the lesion prediction image is a prediction image obtained by sending the breast ultrasound image to a pre-trained image segmentation model for image segmentation. Then, the annotation and prediction images are compared, and the missed detection and false detection areas are identified by using set operations, and the corresponding label images are generated; based on the above images, the prediction features of the lesions, missed detection areas and false detection areas are extracted. Then, the topological interaction processing technology is used to integrate multiple prediction features to optimize the model performance. Finally, the integrated multiple prediction features are fused to generate a comprehensive fused prediction feature, thereby improving the segmentation accuracy of the breast ultrasound image and reducing the missed detection and false detection of similar tissues of breast tumors.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a medical image segmentation method and system. Background Art

[0002] Breast tumor segmentation in breast ultrasound images is one of the key steps in computer-aided diagnosis and quantitative analysis of breast cancer. Due to the interference of factors such as speckle noise and ultrasound artifacts, the diversity of tumor morphology, and the difference between benign and malignant tumors, breast tumor segmentation faces huge challenges. Although deep learning networks have been used to segment breast tumors, which have improved the segmentation accuracy of breast tumors to varying degrees, the inventors have found that the existing breast segmentation methods have serious missed detections and false detections due to factors such as blurred boundaries and heterogeneous structures. Summary of the invention

[0003] The purpose of the present application is to provide a medical image segmentation method and system, which can improve the segmentation accuracy of breast tumors and reduce missed detection and false detection of similar tissues to breast tumors.

[0004] This application is implemented as follows:

[0005] In a first aspect, the present application provides a medical image segmentation method, comprising the following steps:

[0006] A breast ultrasound image training set is obtained; each breast ultrasound image in the breast ultrasound image training set includes an associated lesion annotation image and a lesion prediction image, and the lesion prediction image is a prediction image obtained by sending the breast ultrasound image to a pre-trained image segmentation model for image segmentation. Set operations are performed on the associated lesion annotation image and the lesion prediction image to obtain corresponding missed detection area label images and false detection area label images. Lesion prediction features, and corresponding missed detection area prediction features and false detection area prediction features are obtained based on the lesion prediction features, and corresponding missed detection area prediction features and false detection area prediction features. Topological interaction processing is performed on the lesion prediction features, and corresponding missed detection area prediction features and false detection area prediction features to obtain multiple prediction features. The multiple prediction features are fused to obtain fused prediction features.

[0007] Further, based on the aforementioned scheme, the set operation processing is performed according to the associated lesion annotation image and lesion prediction image to obtain the corresponding missed detection area label image and false detection area label image, including: using the lesion annotation image to subtract the common area of ​​the lesion annotation image and the lesion prediction image to obtain the missed detection area label image; using the lesion prediction image to subtract the common area of ​​the lesion annotation image and the lesion prediction image to obtain the false detection area label image.

[0008] Further, based on the above scheme, the topological interaction processing is performed according to the lesion prediction features, and the corresponding missed detection area prediction features and misdetected area prediction features to obtain multiple prediction features, including:

[0009] The lesion prediction features, and the corresponding missed detection region prediction features and false detection region prediction features are sent to the topological information interaction module to obtain multiple prediction features. The topological information interaction module is constructed according to the following steps: convolution operations are performed on the lesion prediction features, and the corresponding missed detection region prediction features and false detection region prediction features to obtain multiple prediction layer features; multiple prediction layer features are interacted according to a preset interaction mode to obtain multiple interactive prediction features, and the preset interaction mode includes one or more modes of no interaction, interaction with missed detection regions and false detection regions separately, and interaction with missed detection regions and false detection regions at the same time.

[0010] Further, based on the above scheme, the topological information interaction module includes an encoder network, a decoder network and a topological information interaction unit. The encoder network uses a predefined backbone network to perform convolution operations on the lesion prediction features, and the corresponding features of the missed detection area prediction features and the false detection area prediction features, respectively, to obtain multiple prediction layer features. The decoder network gradually restores the image resolution and fuses multi-scale features through deconvolution, convolution and jump connection operations to output features for predicting lesion areas, missed detection areas and false detection areas. The topological information interaction unit is used to input the features of the predicted lesion areas, missed detection areas and false detection areas, and interact according to a preset interaction method to obtain multiple interactive prediction features, and the preset interaction method includes one or more of no interaction, interaction with missed detection areas and false detection areas separately, and interaction with missed detection areas and false detection areas at the same time.

[0011] Furthermore, based on the aforementioned scheme, the method further includes: converting the fused prediction features into a fused prediction image, and calculating the loss between the fused prediction image and the lesion annotation image; the loss includes a basic loss and a local smoothing loss.

[0012] In a second aspect, the present application provides a medical image segmentation system, comprising:

[0013] The data acquisition module is used to acquire a breast ultrasound image training set; each breast ultrasound image in the breast ultrasound image training set includes an associated lesion annotation image and a lesion prediction image, and the lesion prediction image is a prediction image obtained by sending the breast ultrasound image into a pre-trained image segmentation model for image segmentation. The first processing module is used to perform set operation processing based on the associated lesion annotation image and the lesion prediction image to obtain the corresponding missed detection area label image and the false detection area label image. The second processing module is used to obtain the lesion prediction feature, the corresponding missed detection area prediction feature and the false detection area prediction feature based on the lesion prediction image, the corresponding missed detection area label image and the false detection area label image. The third processing module is used to perform topological interaction processing based on the lesion prediction feature, the corresponding missed detection area prediction feature and the false detection area prediction feature to obtain multiple prediction features. The fourth processing module is used to perform fusion processing on the multiple prediction features to obtain fusion prediction features.

[0014] In a third aspect, the present application provides an electronic device, comprising a memory for storing one or more programs; a processor; when the one or more programs are executed by the processor, a method as described in any one of the first aspects is implemented.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method as described in any one of the above-mentioned first aspects.

[0016] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:

[0017] The present application proposes a medical image segmentation method, which can effectively improve the segmentation accuracy of breast tumors and reduce the missed detection and false detection of similar tissues of breast tumors by introducing the determination of missed detection and false detection areas and topological interaction processing steps. Among them, by determining the missed detection and false detection areas of lesions, it is easy to accurately understand the problems existing in the model during the segmentation process and make targeted improvements. And through topological interaction processing, the spatial relationship and structural information in the image can be fully utilized to extract features, thereby improving the expressiveness and robustness of the features. Finally, by fusing multiple prediction features, a more comprehensive and accurate prediction result can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is a flow chart of an embodiment of a medical image segmentation method of the present application;

[0020] Figure 2 A schematic diagram of performing set operation processing according to associated lesion annotation images and lesion prediction images in one embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of the structure of a topology information interaction module in an embodiment of the present application;

[0022] Figure 4 This is a schematic diagram of the structure of a topology information interaction unit in an embodiment of the present application;

[0023] Figure 5 This is a structural block diagram of an embodiment of a medical image segmentation system of the present application;

[0024] Figure 6 A structural block diagram of an electronic device provided in an embodiment of the present application.

[0025] Icons: 101, data acquisition module; 102, first processing module; 103, second processing module; 104, third processing module; 105, fourth processing module; 201, processor; 202, memory; 203, communication interface. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] Example 1

[0029] Breast ultrasound images are an indispensable tool in the clinical diagnosis and evaluation of breast diseases. However, due to the complexity of breast tissue and the inherent characteristics of ultrasound images, accurately segmenting breast ultrasound images is a challenging task. Although existing breast segmentation methods have improved the segmentation accuracy of breast tumors to varying degrees, the similar intensity distribution, cascades, and shadows of surrounding tissues still lead to serious missed detections and false detections.

[0030] To this end, the present application proposes a medical image segmentation method, which effectively improves the segmentation accuracy of breast tumors and reduces missed detection and false detection of similar tissues of breast tumors by introducing the determination of missed detection and false detection areas and topological interaction processing steps.

[0031] See also Figure 1 , the medical image segmentation method comprises the following steps:

[0032] Step S101: Obtain a breast ultrasound image training set; each breast ultrasound image in the breast ultrasound image training set includes an associated lesion annotation image and a lesion prediction image, and the lesion prediction image is a predicted image obtained by sending the breast ultrasound image to a pre-trained image segmentation model for image segmentation.

[0033] Among them, each breast ultrasound image is equipped with a tumor area image manually annotated by a medical expert. These images are called lesion annotation images and are used as reference standards for segmentation results. When training an image segmentation model, you can first collect a series of training data, and then divide the training set and test set into a ratio of 7:3. For example, if there are data from 1,000 patients, 700 cases are used as training sets and the remaining 300 cases are used as test sets. Then, the data of the training set (700 test data) can be sent to the deeplab v3 network for training. After the model converges, use the test set data (300 test data) for testing, and calculate the predicted image of each input image (the segmentation prediction image of the breast tumor of the 300 test images).

[0034] Step S102: performing set operation processing on the associated lesion annotation image and lesion prediction image to obtain corresponding missed detection region label image and misdetected region label image.

[0035] Step S103: Obtain lesion prediction features and corresponding missed-detection region prediction features and false-detection region prediction features according to the lesion prediction image and the corresponding missed-detection region label image and false-detection region label image.

[0036] Exemplarily, the lesion annotation image and the lesion prediction image can be compared pixel by pixel. The pixel value in the lesion annotation image indicates whether the pixel belongs to the breast tumor area (for example, 1 represents the tumor area and 0 represents the non-tumor area), while the pixel value in the lesion prediction image is the model's prediction result for the pixel. By calculating the difference between the lesion annotation image and the lesion prediction image, we can obtain the missed area label image, that is, the part of the lesion annotation image that belongs to the tumor area but is omitted in the prediction image. Similarly, by calculating the difference between the lesion prediction image and the lesion annotation image, we can obtain the misdetected area label image, that is, the non-tumor part of the lesion prediction image that is mistakenly predicted as a tumor area.

[0037] like Figure 2 As shown, Figure 2 The figure shows a schematic diagram of performing set operation processing according to the associated lesion annotation image and lesion prediction image in one embodiment of the present application. In one implementation of the present application, the set operation processing according to the associated lesion annotation image and lesion prediction image to obtain the corresponding missed detection area label image and false detection area label image includes: using the lesion annotation image to subtract the common area of ​​the lesion annotation image and the lesion prediction image to obtain the missed detection area label image; using the lesion prediction image to subtract the common area of ​​the lesion annotation image and the lesion prediction image to obtain the false detection area label image. At the same time, the missed detection area label image and the false detection area label image can also be checked by a clinician to improve the reliability of the determined missed detection area label image and the false detection area label image.

[0038] Exemplarily, when the associated lesion annotation image and the lesion prediction image are processed by set operations to obtain the corresponding missed detection area label image and the false detection area label image, the specific operation can compare the lesion annotation image and the lesion prediction image at the pixel level. It should be noted that for each breast ultrasound image, it is necessary to compare the value of each pixel on its lesion annotation image and the lesion prediction image. The missed detection area label image and the false detection area label image can be used to visualize the effect of model segmentation and help evaluate the performance of the model. By calculating the size or number of pixels of the missed detection area and the false detection area, the accuracy and robustness of the model in tumor detection can be quantified.

[0039] Specifically, in the lesion annotation image, the tumor area is usually marked with a specific pixel value (such as 1, representing the tumor area; 0, representing the non-tumor area). These pixel values ​​represent the real tumor area. The lesion prediction image is generated by the image segmentation model, in which the pixels predicted by the model as the tumor area are also marked with a specific value (it can also be 1, indicating that it is predicted as a tumor; 0, indicating that it is predicted as a non-tumor). In addition, the missed detection area refers to a set of pixels that are marked as tumor areas in the annotated image but are not marked in the predicted image (that is, they are predicted to be incorrectly marked as non-tumor areas). Through logical operations (such as: annotated image AND NOT predicted image), these pixels can be found and a missed detection area label image can be generated. In the missed detection area label image, these pixels are usually marked (such as red or highlighted). The false detection area is the opposite. It refers to a set of pixels that are incorrectly marked as tumor areas in the predicted image, but do not actually exist in the annotated image. Through logical operations (such as: predicted image AND NOT annotated image), these pixels can be found and a false detection area label image can be generated. In the misdetected region label image, these pixels will also be marked (e.g., highlighted in blue or a different color).

[0040] Please continue reading Figure 1 , step S104: performing topological interaction processing according to the lesion prediction features, and the corresponding missed detection region prediction features and misdetected region prediction features to obtain multiple prediction features;

[0041] Step S105: fusing the multiple prediction features to obtain a fused prediction feature.

[0042] In the above steps, after determining the prediction features of the missed detection area and the prediction features of the false detection area, topological interaction processing is used to extract multiple prediction features. Among them, topological interaction processing is a method for extracting features by using the spatial relationship and structural information between different regions in the image. Specifically, the lesion prediction features, the missed detection area prediction features and the false detection area prediction features can be used as inputs and sent to a topological information interaction module. The module first extracts features from the three input images respectively to obtain multiple prediction layer features. Then, according to the preset interaction mode, these prediction layer features are interactively processed. The specific interaction mode may include no interaction, interaction with the missed detection area alone, interaction with the false detection area alone, and interaction with the missed detection area and the false detection area at the same time. Through interactive processing, multiple interactive prediction features can be obtained, which not only contain the information in the prediction image, but also incorporate the information of the missed detection and false detection areas, thereby improving the expressiveness and robustness of the features.

[0043] Next, multiple interactive prediction features are fused to obtain fused prediction features. Fusion processing can be performed in a variety of ways, such as weighted average, maximum fusion, and convolution fusion. Through fusion processing, useful information from multiple prediction features can be integrated to form a more comprehensive and accurate prediction result. The fused prediction feature will be used as the final output to represent the tumor area in the breast ultrasound image.

[0044] Finally, based on the aforementioned scheme, in one implementation of the present application, the method further includes: converting the fused prediction features into a fused prediction image, and calculating the loss between the fused prediction image and the lesion annotation image; the loss includes a basic loss and a local smoothing loss.

[0045] Among them, the fused prediction feature is usually a multi-dimensional feature vector or feature map, which contains rich information extracted from the original image. In order to convert these features into a predicted image that can be directly compared with the annotated image, an appropriate decoder or post-processing step is required. This usually involves converting the feature map into a pixel-level prediction of the same size as the original image. The loss function is used to quantify the difference between the model prediction and the true annotation and drive the training process of the model. The basic loss measures the global difference between the fused prediction image and the annotated image. Exemplarily, the basic loss function can include cross entropy loss (for classification tasks) or mean square error loss (for regression tasks). In addition, the local smoothness loss is intended to encourage the predicted image to maintain smoothness within the local area, that is, the prediction results between adjacent pixels should be continuous. Because tumor regions usually have continuous and smooth boundaries, the problem of discontinuous predicted values ​​that may exist in the space of the segmentation results can be improved by controlling the local smoothness loss. Specifically, the local smoothness loss can be achieved by calculating the difference between the prediction values ​​of adjacent pixels, such as using gradient loss or Laplace loss.

[0046] In summary, in the above-described embodiments, by introducing the determination of missed detection and false detection area and the topology interaction processing step, the segmentation accuracy of breast tumors can be effectively promoted, and the missed detection and false detection of similar tissues of breast tumors can be reduced.Wherein, by determining the missed detection and false detection area, it is convenient to accurately understand the problems of the model in the segmentation process, and improve it in a targeted manner.And by topology interaction processing, the spatial relationship and structural information in the image can be fully utilized to extract features, thereby improving the expressive power and robustness of features.Finally, by fusing multiple prediction features, a more comprehensive and accurate prediction result can be obtained, for the diagnosis and treatment of breast diseases provide more reliable basis.

[0047] Based on the above scheme, in one implementation of the present application, the topological interaction processing is performed according to the lesion prediction features, and the corresponding missed detection area prediction features and the false detection area prediction features to obtain multiple prediction features, including: sending the lesion prediction features, and the corresponding missed detection area prediction features and the false detection area prediction features to the topological information interaction module to obtain multiple prediction features. Wherein, the topological information interaction module is constructed according to the following steps: convolution operations are performed on the features corresponding to the lesion prediction features, and the corresponding missed detection area prediction features and the false detection area prediction features, respectively, to obtain multiple prediction layer features; multiple prediction layer features are interacted according to a preset interaction mode to obtain multiple interactive prediction features, and the preset interaction mode includes one or more modes of no interaction, interaction with missed detection areas and false detection areas separately, and interaction with missed detection areas and false detection areas at the same time.

[0048] In the above implementation, firstly, by performing a convolution operation on each image, feature extraction can be performed, and the obtained multiple prediction layer features can capture information at different abstract levels from low to high, such as edges, shapes, and textures. Then, according to a preset interaction mode, multiple prediction layer features are interacted, and the information of missed and falsely detected areas can be integrated into the prediction features, thereby improving the segmentation results.

[0049] Among them, no interaction means: in some cases, you can choose not to perform any interaction, but directly use the features extracted from the predicted image. Interacting with missed detection areas alone means: interacting the lesion prediction features with the missed detection area prediction features to emphasize the tumor parts that may appear in the missed detection areas. Interacting with false detection areas alone means: interacting the features of lesion prediction features with the false detection area prediction features to correct those parts that are mistakenly identified as tumors in the false detection areas. Simultaneous interaction with missed detection areas and false detection areas means: considering the information of missed detection and false detection areas at the same time, interacting the features of the lesion prediction image with both. This method can comprehensively utilize the information of the two areas to improve the segmentation results more comprehensively.

[0050] like Figure 3 As shown, Figure 3 It is a structural diagram of a topological information interaction module in an embodiment of the present application. In an implementation of the present application, the topological information interaction module includes an encoder network, a decoder network and a topological information interaction unit. The encoder network uses a predefined backbone network to perform convolution operations on the lesion prediction features, and the corresponding missed detection area prediction features and the features corresponding to the false detection area prediction features, respectively, to obtain multiple prediction layer features. The decoder network gradually restores the image resolution and fuses multi-scale features through deconvolution, convolution and jump connection operations to output features for predicting lesion areas, missed detection areas and false detection areas. The topological information interaction unit is used to input the features of the predicted lesion areas, missed detection areas and false detection areas, and interact according to a preset interaction method to obtain multiple interactive prediction features, and the preset interaction method includes one or more of no interaction, interaction with missed detection areas and false detection areas separately, and interaction with missed detection areas and false detection areas at the same time.

[0051] Among them, the encoder network uses a predefined backbone network (such as ResNet, VGG, etc.) to extract features from the lesion prediction image, the missed detection area label image, and the false detection area label image. These images are passed through multiple convolutional layers, pooling layers, etc. in the encoder network to gradually reduce the image resolution and extract features of different scales. These features will serve as the basis for subsequent processing. The decoder network is responsible for gradually restoring the image resolution and fusing multi-scale features to output features for predicting lesion areas, missed detection areas, and false detection areas. Through operations such as deconvolution, convolution, and jump connections, the decoder network can gradually upsample the features extracted by the encoder network to the resolution of the original image and fuse feature information of different scales in the process. This design can retain the detailed information in the image and improve the accuracy of the segmentation results. The topological information interaction unit is the core part of the module, which is responsible for interacting the features of the predicted lesion area, missed detection area, and false detection area. According to the preset interaction method, these features can be combined and fused in different ways.

[0052] In order to help those skilled in the art to understand the present application more intuitively, a specific example will be used here to illustrate.

[0053] like Figure 3 As shown, in one implementation of the present application, the encoder network includes 5 encoder modules, and the decoder network includes 4 decoder modules. Exemplarily, the encoding network uses ResNet50 as the backbone network, and the 5 encoder modules output 5 features of different resolutions respectively ( Figure 3 ).

[0054] For the first decoder module: refer to the operation of the UNet network and convert the features output by the fifth encoder module ( Figure 3 The 5th blue feature in the 1st row of the image is taken as input, and the image resolution is restored through a 3×3 deconvolution with a stride of 2. At the same time, the features of the deconvolution output are combined with the features output by the fourth encoder module through a skip connection. Figure 3 The 4th blue feature in the 1st row of the network is cascaded, and then two 3×3 convolutions (in series) are used to optimize the network, outputting the features of the decoder network ( Figure 3 The first red feature in the first row of ). The number of channels of the output feature of the 3×3 deconvolution with a stride of 2 is equal to the number of channels of the output feature of the fourth encoding module. The number of output channels of the first 3×3 convolution is equal to half of the number of input feature channels, and the second 3×3 convolution does not change the number of feature channels.

[0055] For the second decoder module: First, the features output by the first decoder module ( Figure 3The first red feature in the first row of ) is used as input, and the image resolution is restored through a 3×3 deconvolution with a stride of 2. Then, the network is optimized through a 3×3 convolution, and the intermediate layer features of the second decoder module are output ( Figure 3 The number of channels of the output feature of the 3×3 deconvolution with a stride of 2 is equal to the number of channels of the output feature of the third encoding module. The 3×3 convolution does not change the number of feature channels.

[0056] Secondly, the intermediate layer features are combined with the output features of the third encoder module ( Figure 3 The third blue feature in the first row of the cascade is then output through two 3×3 convolutions (in parallel) to predict the features of the missed detection area and the false detection area ( Figure 3 The number of output channels of the two 3×3 convolutions (in parallel) is equal to half the number of input feature channels.

[0057] Again, these two features are used to predict missed detection areas and false detection areas ( Figure 3 The green features of the second decoder module in ) and the intermediate layer features of the module ( Figure 3 The first row of the second red feature in the image is connected and concatenated with the feature output of the third encoder module through a skip connection. Then, a 3×3 convolution is performed to reduce the number of output feature channels to one-fourth of the output feature channels, and the feature ( Figure 3 Then, the features used to predict the lesion area ( Figure 3 The yellow features of the second decoder module in the image are added to the intermediate layer features to alleviate the problem of gradient vanishing and gradient exploding, and are used as the output of the second decoder module. At the same time, the features used to predict the lesion area ( Figure 3 The yellow features of the second decoder module in the image are upsampled by 2 times and used to optimize the third decoder module. Finally, the features of the predicted missed and falsely detected areas (the green features of the second decoder module) are compared with the features of the predicted lesion areas ( Figure 3 The yellow features of the second decoder module in are sent to the topology information interaction module for topology information interaction and loss calculation.

[0058] For the third decoder module: Similar to the second decoder module, the intermediate layer features of the third decoder module are output through a 3×3 deconvolution with a stride of 2 and a 3×3 convolution ( Figure 3 The number of channels of the output feature of the 3×3 deconvolution with a stride of 2 is equal to the number of channels of the output feature of the 2nd encoding module. The 3×3 convolution does not change the number of feature channels.

[0059] Secondly, the intermediate layer features are combined with the output features of the second encoder module ( Figure 3 The first row and the second blue feature in the cascade are then cascaded, and then two 3×3 convolutions (in parallel) are used to output the features for predicting missed detection areas and false detection areas ( Figure 3 Green features of the 3rd decoder module in ). The number of output channels of the two 3×3 convolutions (in parallel) is equal to half the number of input feature channels.

[0060] Again, these two features are used to predict missed detection areas and false detection areas ( Figure 3 The green features of the third decoder module in ) and the intermediate layer features of the module ( Figure 3 The red features of the third decoder module in the image are connected and concatenated with the features output by the second encoder module and the features of the lesion area prediction layer after upsampling in the previous module through a skip connection.

[0061] Next, through a 3×3 convolution, the number of output feature channels is reduced to one-fifth of the output feature channels, and the features used to predict the lesion area are output ( Figure 3 Then, the features used to predict the lesion area ( Figure 3 The yellow features of the third decoder module in the image are added to the intermediate layer features of the third decoder module and used as the output of the third decoder module. At the same time, the features used to predict the lesion area ( Figure 3 The yellow features of the 3rd decoder module in ) are upsampled by a factor of 2 and used to optimize the 4th decoder module.

[0062] Finally, the features of missed and falsely detected regions are predicted ( Figure 3 The green features of the third decoder module in ) and the features of the predicted lesion area ( Figure 3 The yellow features of the third decoder module in are sent to the topology information interaction unit for topology information interaction and loss calculation.

[0063] For the 4th decoder module: Consistent with the 3rd decoder module, the intermediate layer features of the 4th decoder module are output through a 3×3 deconvolution with a stride of 2 and a 3×3 convolution ( Figure 3 The 4th red feature in the 1st row of ). The number of channels of the output feature of the 3×3 deconvolution with a stride of 2 is equal to the number of channels of the output feature of the 1st encoding module. The 3×3 convolution does not change the number of feature channels.

[0064] Secondly, the intermediate layer features are combined with the output features of the second encoder module ( Figure 3 The first blue feature of the first row in the cascade is then output through two 3×3 convolutions (in parallel) to predict the features of missed detection areas and false detection areas ( Figure 3 Green features of the 4th decoder module in ). The number of output channels of the two 3×3 convolutions (in parallel) is equal to half the number of input feature channels.

[0065] Again, these two features are used to predict missed detection areas and false detection areas ( Figure 3 The green features of the third decoder module in ) and the intermediate layer features of the module ( Figure 3 The red features of the 4th decoder module in the image are connected and concatenated with the features output by the 1st encoder module and the features of the lesion area prediction layer after upsampling in the previous module through a skip connection.

[0066] Next, through a 3×3 convolution, the number of output feature channels is reduced to one-fifth of the output feature channels, and the features used to predict the lesion area are output ( Figure 3 Yellow features of the first decoder module in ).

[0067] Finally, the features of missed and falsely detected regions are predicted ( Figure 3 The green features of the 4th decoder module in ) and the features of the predicted lesion area ( Figure 3 The yellow features of the 4th decoder module in are sent to the topology information interaction module for topology information interaction and loss calculation, and the final prediction result of the lesion area is output.

[0068] like Figure 4 As shown, Figure 4 The topological information interaction unit in one embodiment of the present application is shown in FIG. 1 . For the t-th decoder module, a 3×3 convolution is first added after each of the three input feature layers to output three 4-channel prediction layer features, which are recorded as and They represent the lesion region prediction layer features of the t-th decoder module, the missed detection region prediction layer features of the t-th decoder module, and the misdetected region prediction layer features of the t-th decoder module. Then, the Softmax cross entropy loss function is used to convert and The loss is calculated with the corresponding Ground-Truth (label) and recorded as and Then, through the argmax operation, we can get three 4-category prediction images, namely the lesion area image, the missed detection area label image and the false detection area label image. Among them, category 0 represents the background area, category 1 represents the lesion area, category 2 represents the missed detection area, and category 3 represents the false detection area. Then, it is converted into a binary image, that is, a value greater than 0 is equal to 1, 0 represents the background category, and 1 represents the corresponding category value, recorded as and

[0069] Next, and Perform topological interaction to obtain different feature layers. Figure 4 As shown, a total of 6 interaction results are output.

[0070] The first interaction result: and and Zero interaction ( Figure 4 The lesion area does not interact with the missed detection area and the false detection area, and the output feature

[0071] The second interactive result: and Interact ( Figure 4 The detection results of the false detection area are covered. First, calculate and The common area of ​​the area greater than 0 in and The intersection of make The characteristics middle The parameters in the area are converted to Parameters, get new features

[0072] The third interactive result: and Interact ( Figure 4 The parameters of the lesion area are not changed, and the parameters of the remaining missed areas are covered. First, calculate and The common area of ​​the area greater than 0 in and The intersection of Then use Subtract the area in and The common area of ​​the area greater than 0 in is the area that needs to be changed, which is recorded as make The characteristics middle The parameters of the region are replaced by Parameters can output new features

[0073] The fourth interactive result: and and Interact ( Figure 4In the upper middle of the figure, the parameters of the lesion area are not changed, and the parameters of the missed detection area and the false detection area are adjusted. The calculation can be continued based on the results of the previous step. First, calculate and The common area of ​​the area greater than 0 in and The intersection of make The characteristics middle The parameters in the area are converted to Parameters, get new features

[0074] The fifth interactive result: and and Interact ( Figure 4 Adjust the parameters of missed detection area and false detection area, and continue to calculate based on the previous results. middle The parameters in the area are converted to Parameters, get new features

[0075] The sixth interactive result: and and Interact ( Figure 4 ), adjust the parameters of missed detection area and false detection area, and continue to calculate based on the previous results. The characteristics middle The parameters in the area are converted to Parameters, get new features

[0076] Because the model has different abilities to predict lesion areas, missed detection areas, and false detection areas, the accuracy of the features obtained through topological interaction in predicting lesions also varies. and First, the three input feature layers are cascaded, and then three fully connected layers are passed through. The output channels of the three fully connected layers are 4096, 512, and 6 respectively. Then, a Softmax layer is added after the output 6-channel array to reduce the interval of the array value to between 0 and 1, and a 6-dimensional weight value is obtained. Finally, the 6-dimensional weight value is weighted and added to the 6 topological interaction features to obtain the fusion prediction feature.

[0077] Through the argmax operation, the fusion prediction features Convert it into a 4-category prediction image, then convert the value equal to 2 in the image to 1, and the value equal to 3 to 0, and you can get the prediction image of the lesion area Then will Calculate the L1 loss with the annotated image of the lesion area to get the loss of the fused prediction feature

[0078] Considering the fusion prediction features There may be discontinuities in the predicted values. The local smoothing loss is obtained by multiplying the lesion area annotation image by a 3×3 convolution [[0,-1,0],[-1,4,-1],[0,-1,0]], and then calculating the L1 loss.

[0079] Among them, the loss of the t-th decoder can be recorded as:

[0080] Where 2≤t≤4.

[0081] The total loss can be recorded as: L = L 4 +0.3×L 3 +0.2×L 2 .

[0082] It should be noted that in the test phase of the model, no topological interaction is required between the second and third decoders, which can reduce the number of model parameters. In the fourth decoder, the output As the final lesion area prediction result of the model.

[0083] Example 2

[0084] See also Figure 5 , the embodiment of the present application provides a medical image segmentation system, which includes:

[0085] The data acquisition module 101 is used to acquire a breast ultrasound image training set; each breast ultrasound image in the breast ultrasound image training set includes an associated lesion annotation image and a lesion prediction image, and the lesion prediction image is a prediction image obtained by sending the breast ultrasound image into a pre-trained image segmentation model for image segmentation. The first processing module 102 is used to perform set operation processing based on the associated lesion annotation image and the lesion prediction image to obtain the corresponding missed detection area label image and the false detection area label image. The second processing module 103 is used to obtain the lesion prediction feature and the corresponding missed detection area prediction feature and the false detection area prediction feature based on the lesion prediction image and the corresponding missed detection area label image and the false detection area label image; the third processing module 104 is used to perform topological interaction processing based on the lesion prediction feature and the corresponding missed detection area prediction feature and the false detection area prediction feature to obtain multiple prediction features. The fourth processing module 105 is used to perform fusion processing on the multiple prediction features to obtain fusion prediction features.

[0086] For the specific implementation process of the above system, please refer to the medical image segmentation method provided in Example 1, which will not be repeated here.

[0087] Example 3

[0088] See also Figure 6 , the embodiment of the present application provides an electronic device, which includes at least one processor 201 and at least one memory 202; wherein the processor 201 and the memory 202 are directly connected to each other, or communicate with each other through a communication interface 203, or are electrically connected through one or more communication buses or signal lines to achieve data transmission or interaction; the memory 202 stores program instructions that can be executed by the processor 201, and the processor 201 calls the program instructions to execute a medical image segmentation method. For example, the following is implemented:

[0089] A breast ultrasound image training set is obtained; each breast ultrasound image in the breast ultrasound image training set includes an associated lesion annotation image and a lesion prediction image, and the lesion prediction image is a prediction image obtained by sending the breast ultrasound image to a pre-trained image segmentation model for image segmentation. Set operations are performed on the associated lesion annotation image and the lesion prediction image to obtain corresponding missed detection area label images and false detection area label images. Lesion prediction features, and corresponding missed detection area prediction features and false detection area prediction features are obtained based on the lesion prediction features, and corresponding missed detection area prediction features and false detection area prediction features. Topological interaction processing is performed on the lesion prediction features, and corresponding missed detection area prediction features and false detection area prediction features to obtain multiple prediction features. The multiple prediction features are fused to obtain fused prediction features.

[0090] Among them, the memory 202 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0091] The processor 201 may be an integrated circuit chip with signal processing capability. The processor 201 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0092] Understandably, Figure 6 The structure shown is for illustration only. The electronic device may also include Figure 6 More or fewer components as shown, or with Figure 6 Different configurations are shown. Figure 6 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0093] Example 4

[0094] The present application provides a computer-readable storage medium having a computer program stored thereon, which implements a medical image segmentation method when executed by the processor 201. For example, the following method is implemented:

[0095] A breast ultrasound image training set is obtained; each breast ultrasound image in the breast ultrasound image training set includes an associated lesion annotation image and a lesion prediction image, and the lesion prediction image is a prediction image obtained by sending the breast ultrasound image to a pre-trained image segmentation model for image segmentation. Set operations are performed on the associated lesion annotation image and the lesion prediction image to obtain corresponding missed detection area label images and false detection area label images. Lesion prediction features, and corresponding missed detection area prediction features and false detection area prediction features are obtained based on the lesion prediction features, and corresponding missed detection area prediction features and false detection area prediction features. Topological interaction processing is performed on the lesion prediction features, and corresponding missed detection area prediction features and false detection area prediction features to obtain multiple prediction features. The multiple prediction features are fused to obtain fused prediction features.

[0096] If the above functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0097] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A medical image segmentation method, characterized in that: The following steps are involved: Acquire a breast ultrasound image training set; each breast ultrasound image in the breast ultrasound image training set includes an associated lesion annotation image and a lesion prediction image, wherein the lesion prediction image is a prediction image obtained by sending the breast ultrasound image to a pre-trained image segmentation model for image segmentation; Performing set operations on the associated lesion annotation images and lesion prediction images to obtain corresponding missed detection area label images and misdetected area label images; Obtaining lesion prediction features, and corresponding missed-detection region prediction features, and corresponding false-detection region prediction features according to the lesion prediction image, and the corresponding missed-detection region label image, and the false-detection region label image; Performing topological interaction processing according to the lesion prediction features, and the corresponding missed detection region prediction features and the false detection region prediction features to obtain a plurality of prediction features, specifically comprising: sending the lesion prediction features, and the corresponding missed detection region prediction features and the false detection region prediction features to a topological information interaction module to obtain a plurality of prediction features; wherein the topological information interaction module is constructed according to the following steps: performing convolution operations on the lesion prediction features, and the features corresponding to the missed detection region prediction features and the false detection region prediction features, respectively, to obtain a plurality of prediction layer features; interacting the plurality of prediction layer features according to a preset interaction mode to obtain a plurality of interactive prediction features, wherein the preset interaction mode comprises one or more modes of no interaction, interaction with the missed detection region and the false detection region separately, and interaction with the missed detection region and the false detection region simultaneously; The multiple prediction features are fused to obtain a fused prediction feature.

2. A medical image segmentation method as claimed in claim 1, characterized in that: The step of performing set operation processing on the associated lesion annotation image and the lesion prediction image to obtain corresponding missed detection area label image and misdetected area label image includes: The common area of ​​the lesion annotation image and the lesion prediction image is subtracted from the lesion annotation image to obtain a label image of the missed detection area; The common area between the lesion annotation image and the lesion prediction image is subtracted from the lesion prediction image to obtain the misdetected area label image.

3. A medical image segmentation method as claimed in claim 1, characterized in that: The topology information interaction module includes an encoder network, a decoder network and a topology information interaction unit; The encoder network uses a predefined backbone network to perform convolution operations on the lesion prediction features, the corresponding missed detection region prediction features, and the features corresponding to the misdetected region prediction features, respectively, to obtain multiple prediction layer features; The decoder network gradually restores the image resolution and fuses multi-scale features through deconvolution, convolution and skip connection operations to output features for predicting lesion areas, missed detection areas and false detection areas; The topological information interaction unit is used to input the features of the predicted lesion area, missed detection area and false detection area, and interact according to a preset interaction method to obtain multiple interactive prediction features. The preset interaction method includes one or more of no interaction, interaction with the missed detection area and the false detection area separately, and interaction with the missed detection area and the false detection area at the same time.

4. A medical image segmentation method as claimed in claim 1, characterized in that: The method further comprises: The fused prediction features are converted into a fused prediction image, and the loss between the fused prediction image and the lesion annotation image is calculated; the loss includes a base loss and a local smoothing loss.

5. A medical image segmentation system, characterized in that: include: A data acquisition module is used to acquire a breast ultrasound image training set; each breast ultrasound image in the breast ultrasound image training set includes an associated lesion annotation image and a lesion prediction image, and the lesion prediction image is a prediction image obtained by sending the breast ultrasound image to a pre-trained image segmentation model for image segmentation; A first processing module is used to perform set operation processing according to the associated lesion annotation image and the lesion prediction image to obtain corresponding missed detection area label image and false detection area label image; The second processing module obtains the lesion prediction features and the corresponding missed detection area prediction features and the false detection area prediction features according to the lesion prediction image and the corresponding missed detection area label image and the false detection area label image; The third processing module is used to perform topological interaction processing according to the lesion prediction features, and the corresponding missed detection area prediction features and the false detection area prediction features to obtain multiple prediction features, specifically including: sending the lesion prediction features, and the corresponding missed detection area prediction features and the false detection area prediction features to the topological information interaction module to obtain multiple prediction features; wherein the topological information interaction module is constructed according to the following steps: performing convolution operations on the lesion prediction features, and the features corresponding to the missed detection area prediction features and the false detection area prediction features, respectively, to obtain multiple prediction layer features; interacting the multiple prediction layer features according to a preset interaction mode to obtain multiple interactive prediction features, wherein the preset interaction mode includes one or more modes of no interaction, interaction with the missed detection area and the false detection area separately, and interaction with the missed detection area and the false detection area at the same time; The fourth processing module is used to perform fusion processing on the multiple prediction features to obtain a fused prediction feature.

6. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images