Breast tumor ultrasonic image segmentation method and device based on multi-branch jump connection search

Abstract

The invention discloses a breast tumor ultrasonic image segmentation method and device based on multi-branch jump connection search. The method comprises three steps of optimizing a search space, searching for optimal jump connection and retraining and testing. Firstly, in order to further extract and fuse features and provide more useful information for a decoder, a multi-branch jump connection module (MBS) is designed. Secondly, searching the optimal MBS by adopting a search strategy based on learnable operation weights; and finally, constructing a coding-decoding neural network by adopting the searched optimal MBS, and performing evaluation test on the breast ultrasonic data set. According to the method, manual parameter adjustment is not needed, and meanwhile higher segmentation accuracy is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network image segmentation, and more specifically, to a method and device for breast tumor ultrasound image segmentation based on multi-branch skip connection search. Background Art

[0002] Breast cancer is a disease with high morbidity and mortality. Early detection of breast lesions in the clinical stage plays a vital role in clinical decision-making, treatment and rehabilitation plans. Ultrasound imaging has been widely used in the diagnosis of breast cancer due to its unique advantages such as safety, economy and efficiency. Accurate segmentation of ultrasound images of breast lesions is helpful for the diagnosis and treatment of breast cancer. However, due to the low-quality ultrasound imaging caused by speckle noise and strong shadows, as well as the irregularity of breast lesions caused by differences in patient tumor morphology, the segmentation of breast ultrasound lesions remains a challenging task.

[0003] Based on the powerful representation ability of convolutional neural networks (CNN), it shows great application prospects in breast cancer ultrasound image segmentation. For example, Yap et al. used fully convolutional neural networks (FCN) to segment breast cancer ultrasound images, but because FCN used a simple upsampling operation, the final segmentation results were not ideal. To solve this problem, symmetric encoder-decoder architectures such as U-Net were proposed to improve image segmentation accuracy. By adding dilated convolutions to the encoder-decoder structure to control the receptive field, Byra et al. achieved good results in the segmentation of breast cancer ultrasound images. Chen et al. proposed a nested U-Net model for the segmentation of breast tumor images. By using U-Nets with different depths and shared weights, a robust representation of breast tumors was achieved. Lyu et al. proposed a breast ultrasound image segmentation model based on an encoder-decoder architecture. The model uses depthwise separable convolutions in the encoder part to obtain multi-scale breast feature maps, and combines the attention mechanism to achieve effective segmentation of the lesion area. Although the above methods improve the accuracy of automatic segmentation of breast cancer ultrasound images, CNN, which expands the receptive field by stacking the number of convolutional layers and using larger convolutional kernels, still finds it difficult to effectively capture the long-term dependencies in breast ultrasound images.

[0004] To address this problem, the neural network can be further designed in combination with the Transformer that captures long-term dependencies. For example, Song et al. designed a feature aggregation network to fuse ultrasound breast tumor features extracted from Transformer, graph convolutional neural network and CNN, thereby improving the accuracy of breast cancer segmentation. Ahmed Iqbal et al. proposed integrating SwinTransformer into the traditional U-Net model to improve global modeling capabilities, and combined spatial interaction blocks, feature compression blocks and relationship aggregation blocks to further improve the model segmentation performance. Qin et al. embedded multi-scale fusion blocks and Transformer modules in the encoder, providing multi-scale and global attention information, thereby enhancing long-term dependency features. Although the above methods have achieved high accuracy, both the backbone network and the overall framework are manually designed. This manually designed model often requires a lot of parameter tuning, relies on manual experience, and has poor adaptability. Summary of the invention

[0005] One of the purposes of the present invention is to provide a method for breast tumor ultrasound image segmentation based on multi-branch skip connection search, so as to solve the technical problems that the manually designed image segmentation model in the prior art requires a large amount of parameter tuning, relies on manual experience, and has poor adaptability; the second purpose of the present invention is to provide a breast tumor ultrasound image segmentation device based on multi-branch skip connection search.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] A first aspect of the present invention provides a method for breast tumor ultrasound image segmentation based on multi-branch skip connection search, comprising the following steps:

[0008] Obtain ultrasound images of breast tumors;

[0009] Constructing a candidate neural network, wherein the candidate neural network includes a multi-branch skip connection module, wherein the multi-branch skip connection module obtains feature maps of different scales by channel through multiple branches, and transmits the feature maps of different scales to a decoder of the candidate neural network by using skip connections;

[0010] Searching for an optimal multi-branch skip connection module based on the breast tumor ultrasound image and the candidate neural network;

[0011] Using the optimal multi-branch skip connection module to construct an image segmentation network, and using the breast tumor ultrasound image to train the image segmentation network to obtain a trained image segmentation network;

[0012] The trained image segmentation network is used to perform image segmentation on the breast tumor ultrasound image to be segmented.

[0013] In the above technical means, a multi-branch skip connection module is proposed to further extract and fuse features, provide more effective information for network learning, and a search strategy based on learnable operation weights is used to search for the best multi-branch skip connection module. Finally, the searched best multi-branch skip connection module is used to construct a neural network and perform segmentation on a breast tumor ultrasound image dataset without manual parameter tuning, solving the technical problem that the manually designed image segmentation model in the technology requires a lot of parameter tuning, relies on manual experience, and has poor adaptability.

[0014] Furthermore, the candidate neural network is an encoder-decoder structure, the encoder includes a number of encoding blocks, each encoding block extracts features of different scales, the decoder includes decoding blocks corresponding to the several encoding blocks one by one, and each encoding block is connected to the corresponding decoding block through a multi-branch jump connection module.

[0015] Furthermore, the multi-branch jump connection module includes four branches B1, B2, B3 and B id ,in:

[0016] The input is divided into four parts according to the number of channels. The first three parts are input into branches B1, B2 and B3 respectively, and the last part is input into branch B. id In the above figure, branches B1, B2 and B3 respectively include operations of multiple different types of convolution kernels. After the inputs of branches B1, B2 and B3 are processed by operations of different types of convolution kernels, the output feature maps are fused by weighted summation to obtain the outputs of branches B1, B2 and B3. id It is the identity mapping branch;

[0017] The output of the multi-branch skip connection module is four branches B1, B2, B3 and B id The output of splicing.

[0018] Furthermore, the input is divided into four parts according to the number of channels, including dividing the input into four parts according to the ratio of the number of channels (1k):(1k):(1k):(1-3k), where k is the number of channels.

[0019] Furthermore, the output feature maps are fused by weighted summation, including:

[0020]

[0021] Where, X i For branch B i Input, op j () is the operation of the j-th type of convolution kernel, λ ij For branch Bi The weight of the j-th type of convolution kernel operation, X i ' is branch B i Output.

[0022] Furthermore, the operations of the multiple different types of convolution kernels include 3×3 depth-separable convolution kernel, 5×5 depth-separable convolution kernel, 1×11 depth-separable convolution kernel, 11×1 depth-separable convolution kernel, 9×1 depth-separable convolution kernel and 1×9 depth-separable convolution kernel.

[0023] Further, according to the breast tumor ultrasound image and the candidate neural network, the best multi-branch skip connection module is searched, including:

[0024] Define branch operation weights λ1, λ2 and λ3 for branches B1, B2 and B3 respectively;

[0025] The branch operation weights λ1, λ2 and λ3 are reparameterized using Gumbel-Softmax to obtain corresponding One-hot encodings;

[0026] One-hot encoding is used to select the best candidate operations and obtain the optimal multi-branch skip connection module.

[0027] Furthermore, when searching for the branch operation weights λ1, λ2 and λ3, different multi-branch jump connection modules in the candidate neural network share the same branch operation weight, or different multi-branch jump connection modules in the candidate neural network each independently set a branch operation weight for update learning.

[0028] Furthermore, the use of One-hot encoding to select the best candidate operation includes:

[0029] One-hot encoding is used to perform dot multiplication with the output of the candidate operation and the results are summed to generate the output of the branch.

[0030] A second aspect of the present invention provides a breast tumor ultrasound image segmentation device based on multi-branch skip connection search, characterized in that it includes:

[0031] An image acquisition module, wherein the image acquisition module is used to acquire an ultrasonic image of a breast tumor;

[0032] A network construction module, the network construction module is used to construct a candidate neural network, the candidate neural network includes a multi-branch jump connection module, the multi-branch jump connection module obtains feature maps of different scales by channel through multiple branches, and transmits the feature maps of different scales to the decoder of the candidate neural network by using jump connections;

[0033] A search module, the search module is used to search for an optimal multi-branch skip connection module based on the breast tumor ultrasound image and the candidate neural network;

[0034] A training module, wherein the training module is used to construct an image segmentation network using the optimal multi-branch skip connection module, and train the image segmentation network using the breast tumor ultrasound image to obtain a trained image segmentation network;

[0035] The segmentation module is used to perform image segmentation on the breast tumor ultrasound image to be segmented using a trained image segmentation network.

[0036] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0037] The present invention designs a multi-branch skip connection module, each branch uses different operations to extract features of different scales and types from some channels, so as to optimize the channel-level features from the encoder for decoding. Then, a learnable operation weight search strategy is proposed, which uses Gumbel-softmax to reparameterize the discrete operation weights, and uses both shared and non-shared methods to search for the best operation of the multi-branch skip connection module, without manual parameter adjustment, while achieving higher segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic flow chart of a method for segmenting breast tumor ultrasound images based on multi-branch skip connection search provided by an embodiment of the present invention;

[0039] Figure 2 A framework structure diagram of a breast tumor ultrasound image segmentation method based on multi-branch skip connection search provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the structure of a multi-branch jump connection module provided in an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of a search strategy based on learnable operation weights provided by an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of a candidate neural network structure provided by an embodiment of the present invention;

[0043] Figure 6 An example diagram of a benign breast ultrasound image sample provided by an embodiment of the present invention;

[0044] Figure 7 A schematic diagram of a structure of a module for searching for optimal multi-branch skip connection for different candidate neural networks provided by an embodiment of the present invention;

[0045] Figure 8 The embodiment of the present invention provides Figure 7 Schematic diagram of performance comparison of different candidate neural networks in searching for the best multi-branch skip connection module;

[0046] Fig. 9 A schematic diagram of the visualization results of BUSI dataset segmentation using different methods provided in an embodiment of the present invention;

[0047] Fig.10 A schematic diagram of a breast tumor ultrasound image segmentation device based on multi-branch skip connection search provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0049] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0050] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0051] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0052] Example

[0053] This embodiment provides a breast tumor ultrasound image segmentation method based on multi-branch skip connection search. Figure 1 As shown, the following steps are included:

[0054] Obtain ultrasound images of breast tumors;

[0055] Constructing a candidate neural network, wherein the candidate neural network includes a multi-branch skip connection module, wherein the multi-branch skip connection module obtains feature maps of different scales by channel through multiple branches, and transmits the feature maps of different scales to a decoder of the candidate neural network by using skip connections;

[0056] Searching for an optimal multi-branch skip connection module based on the breast tumor ultrasound image and the candidate neural network;

[0057] Using the optimal multi-branch skip connection module to construct an image segmentation network, and using the breast tumor ultrasound image to train the image segmentation network to obtain a trained image segmentation network;

[0058] The trained image segmentation network is used to perform image segmentation on the breast tumor ultrasound image to be segmented.

[0059] The method proposed in this embodiment mainly includes three steps: optimizing the search space, searching for the best skip connection, and retraining and testing (such as Figure 2 As shown in Figure 2, a multi-branch skip connection module (MBS) is designed to further extract and fuse features and provide more useful information for the decoder. Secondly, a search strategy based on learnable operation weights is used to search for the best MBS. Finally, the best MBS is used to construct an encoding-decoding neural network and an evaluation test is performed on the breast ultrasound dataset.

[0060] In a further embodiment, in order to adaptively select the encoding features to participate in decoding, a multi-branch skip connection MBS is designed. MBS obtains feature maps of different scales by channel through multiple branches, and transmits these multi-scale feature maps to the decoder using skip connections to enhance the effectiveness of the features. Figure 3 As shown, the multi-branch jump connection module includes four branches B1, B2, B3 and B id ,in:

[0061] The input is divided into four parts according to the number of channels:

[0062] X1,X2,X3,X id =Split(X)

[0063] Among them, X i For branch B i The input, op j () is the operation of the j-th type of convolution kernel, λ ij For branch B i The weight of the j-th type of convolution kernel operation, X i ' is branch B i The first three parts of the output are input into branches B1, B2 and B3 respectively, and the last part is input into branch B id In the figure, branches B1, B2 and B3 respectively include operations of multiple different types of convolution kernels. After the inputs of branches B1, B2 and B3 are processed by operations of different types of convolution kernels, the output feature maps are fused by weighted summation to obtain the outputs of branches B1, B2 and B3:

[0064]

[0065] Branch B id It is the identity mapping branch;

[0066] The output of the multi-branch skip connection module is four branches B1, B2, B3 and B id The concatenation of the output:

[0067] X out=Contact(X1',X'2,X3',X id )

[0068] In a further embodiment, dividing the input into four parts according to the number of channels includes dividing the input into four parts according to the ratio of the number of channels (1k):(1k):(1k):(1-3k), where k is the number of channels.

[0069] In a further embodiment, the operations of the multiple different types of convolution kernels include 3×3 depth-wise separable convolution kernel, 5×5 depth-wise separable convolution kernel, 1×11 depth-wise separable convolution kernel, 11×1 depth-wise separable convolution kernel, 9×1 depth-wise separable convolution kernel and 1×9 depth-wise separable convolution kernel.

[0070] In a specific embodiment, the candidate operations include traditional 3×3 and 5×5 depthwise separable convolution kernels, and further introduce more efficient 1×11, 11×1, 1×9 and 9×1 depthwise separable convolution kernels. The candidate operations and their corresponding operation codes are shown in Table 1.

[0071] Table 1 Candidate operations

[0072]

[0073]

[0074] In a further embodiment, based on the breast tumor ultrasound image and the candidate neural network, an optimal multi-branch skip connection module is searched, such as Figure 4 As shown, including:

[0075] Define branch operation weights λ1, λ2 and λ3 for branches B1, B2 and B3 respectively;

[0076] The branch operation weights λ1, λ2 and λ3 are reparameterized using Gumbel-Softmax to obtain corresponding One-hot encodings;

[0077] One-hot encoding is used to select the best candidate operations and obtain the optimal multi-branch skip connection module.

[0078] In a further embodiment, when searching for the branch operation weights λ1, λ2 and λ3, different multi-branch jump connection modules in the candidate neural network share the same branch operation weight or different multi-branch jump connection modules in the candidate neural network each independently set a branch operation weight for update learning.

[0079] The above two different search methods can increase the diversity of different operation combinations of candidate neural networks.

[0080] In a further embodiment, selecting the best candidate operation using One-hot encoding includes:

[0081] One-hot encoding is used to perform dot multiplication with the output of the candidate operation and the results are summed to generate the output of the branch.

[0082] In a further embodiment, the candidate neural network is an encoder-decoder structure, the encoder includes a plurality of encoding blocks, each encoding block extracts features of different scales, the decoder includes decoding blocks corresponding one-to-one to the plurality of encoding blocks, and each encoding block is connected to the corresponding decoding block through a multi-branch jump connection module.

[0083] In a specific embodiment, the candidate neural network adopts Swin-Unet and TransUnet as candidate encoding-decoding architectures, which are named Swin-MBS and Trans-MBS respectively. Figure 5 As shown in the figure, Swin-Unet uses a layered SwinTransformer with a shifted window as an encoder to extract contextual features, and designs a symmetric SwinTransformer decoder with a block expansion layer for performing upsampling operations to restore the spatial resolution of the feature map. Swin-Unet includes three encoding blocks and three corresponding decoding blocks. TransUnet adds a Transformer after the CNN encoder to improve global perception capabilities, and MBS is added between CNN-based encoding and decoding block pairs, and also includes three pairs of encoding and decoding block pairs.

[0084] In a specific embodiment, based on the searched best MBS, retraining and testing are performed under breast cancer ultrasound data. First, in the Swin-MBS neural network architecture, the best MBS is searched by sharing and non-sharing methods, and the constructed models are named Swin-MBS-S and Swin-MBS-NS. Secondly, in the Trans-MBS neural network architecture, the best MBS is searched by sharing and non-sharing methods, and the constructed models are named Trans-MBS-S and Trans-MBS-NS. Finally, these models are retrained and tested on the breast cancer ultrasound dataset.

[0085] This embodiment provides an experimental verification of a breast tumor ultrasound image segmentation method based on multi-branch jump connection search. The experimental environment is carried out on PyTorch, Intel Xeon CPU, and NVIDIA RTX 3090 GPU server with 24GB of memory. The experiment mainly includes two stages: search and test. In the search stage, two search methods, shared and non-shared, are used to search for the best MBS under the Swin-MBS and Trans-MBS neural network architectures. In the test stage, the best MBS searched is used to construct a neural network, and its segmentation accuracy and generalization ability on different breast cancer ultrasound data sets are compared.

[0086] The experiment uses two breast cancer ultrasound datasets, BUSI and OASBUD. The BUSI dataset contains 780 breast cancer ultrasound images and their corresponding ground truth GT, including 133 normal images, 210 benign tumor images, and 437 malignant tumor images. Since normal images do not contain breast cancer tumor GT, only benign and malignant tumor images are used in the experiment.

[0087] In the BUSI dataset, there are 17 samples (16 benign and 1 malignant) containing multiple GTs, but these GTs all belong to the same category. Therefore, these GTs are merged into a single GT. Figure 6 As shown in the figure, a benign breast ultrasound image sample contains two benign GTs, namely Figure 6 (b) and Figure 6 (c) shows that the combined result is Figure 6 (d) Finally, the dataset is divided into training and testing sets in a ratio of 85:15, and the dataset is used to search for the best MBS.

[0088] In order to study the generalization ability of the model, the OASBUD dataset was used to test the performance of the search model on different breast cancer ultrasound datasets. The dataset contains 100 breast cancer ultrasound radiofrequency images, of which 48 are benign and 52 are malignant, and each image contains only one lesion. The OASBUD dataset is also divided into training and test sets in a ratio of 85:15.

[0089] The specific parameters of the experiment are set as follows:

[0090] In the search phase, a total of 50 epochs were iterated, the batch size was 24, the initial learning rate was 0.0001, the learning rate was updated using polynomial decay, the optimization algorithm used was Adam, the weight was decreased to 0.0001, and k was set to 8.

[0091] In the test phase, the BUSI dataset was trained for 150 epochs, using SGD, with momentum set to 0.9, weight decay to 0.0001, batch size to 24, initial learning rate to 0.05, polynomial decay to update the learning rate, and k set to 8. On the OASBUD data, the batch size was 6, and the rest of the parameters were consistent with the BUSI dataset.

[0092] In the experiment, the specific steps of searching for the best MBS are as follows:

[0093] In the search phase, Swin-MBS and Trans-MBS are used as candidate neural network architectures, and the best MBS is searched on the BUSI dataset using a shared and non-shared search method. Figure 7 (a) is composed of two 1×9 and one 1×11 depthwise separable convolution kernels, focusing on capturing feature information in a specific direction. The best MBS of Swin-MBS-NS is shown in Figure 7 (b) shows that each MBS is configured with different convolution kernels to explore various operation combinations. The best MBS of Trans-MBS-S is shown in Figure 7 (c) shows that it is composed of 1×11, 3×3 and 11×1 depthwise separable convolution kernels, which combine convolution kernels of various shapes to adapt to feature processing of different scales and directions. The best MBS of Trans-MBS-NS is shown in Figure 7 As shown in (d), the feature integration capability is enhanced by diversified convolution kernel sizes and branch designs. The Swin-MBS and Trans-MBS models use two different search methods to search for the best MBS, and each search method takes about 26 minutes on the Swin-MBS model and about 27 minutes on the Trans-MBS model. Figure 8 The segmentation performance of the models under the four different search methods shown in the figure gradually changes with epochs. In contrast, the Dice coefficients of the two search methods based on Trans-MBS have a higher upward trend than the Swin-MBS model. Therefore, Trans-MBS will be used as a candidate neural network architecture and further tested on the breast cancer ultrasound dataset.

[0094] Table 2 shows the performance comparison results of Trans-MBS-S and Trans-MBS-NS with five SOTA methods, including (SK)U-Net, AAU-net, AMS-PAN, Swin-Unet and TransUnet, on the BUSI test set. Trans-MBS-S has the highest values ​​in the main indicators such as Dice, IoU, and Accuracy, with Dice and IoU being 85.94% and 77.71% respectively. In contrast, Trans-MBS-NS uses different convolution kernel designs for each MBS module, resulting in the failure to fully retain the feature information of some scales, making its overall segmentation performance slightly lower than Trans-MBS-S. TransUnet ranks third in performance indicators. It uses basically the same encoding and decoding structure as Trans-MBS-S and Trans-MBS-NS, but it uses ordinary skip connections. Swin-Unet uses an encoding and decoding architecture based on the Swin Transformer, and also uses ordinary skip connections between encoding and decoding blocks. Its Dice value is slightly lower than TransUnet. AMS-PAN, which combines attention mechanism and multi-scale fusion, uses global attention upsampling in the decoding stage, and its Dice value is slightly lower than that of Swin-Unet. Although AAU-net and (SK)U-Net both introduce attention mechanism to extract features, their segmentation results for images with blurred mass edges or low contrast are poor.

[0095] Table 2 Comparison of segmentation accuracy of different methods on the BUSI dataset

[0096]

[0097] Fig. 9 Examples of visual segmentation results of different methods on the BUSI dataset are shown. The first column is the original image, the second column is the GT, and the following columns are the segmentation results of different methods. Trans-MBS-NS has low accuracy when processing high-noise areas, resulting in certain segmentation errors in strong shadow areas (fourth column). Although the segmentation results of TransUnet have smooth edges in most cases, its segmentation accuracy is insufficient in images with blurred boundaries (fifth column). Due to the low contrast and high noise of ultrasound images, Swin-Unet has obvious burrs when segmenting tiny edges, resulting in insufficient smoothness in edge processing (sixth column). The segmentation results of AAU-net and (SK)U-Net have large errors with GT. Trans-MBS-S uses the MBS module to adaptively pass the encoded features to the decoder, achieving better segmentation results when processing the above images (third column).

[0098] Table 3 compares the number of parameters and computational complexity of Trans-MBS-S, Trans-MBS-NS, (SK)U-Net, AAU-net, Swin-Unet, and TransUnet. Although (SK)U-Net and Swin-Unet are lightweight in terms of parameter and computational complexity, with the smallest number of parameters and the lowest MACs, respectively, their segmentation accuracy is lower than that of Trans-MBS-S. In contrast, although the model structure of Trans-MBS-NS is similar to that of Trans-MBS-S, its computational complexity and number of parameters increase due to the use of multiple larger convolution kernels in the MBS module. Further comparing Trans-MBS-S with TransUnet, Trans-MBS-S only increases 0.003M parameters and 0.002 billion MACs by introducing the MBS module, but Dice increases by 2.02%. This result further proves that the MBS module can effectively control the consumption of computing resources while improving model performance, achieving a good balance between performance and efficiency.

[0099] Table 3 Comparison of parameters and calculation amount of different methods

[0100]

[0101]

[0102] In order to verify the generalization ability of the best model searched, segmentation tests were continued on the OASBUD dataset. The experimental results were compared with four SOTA methods, including (SK)U-Net, AMS-PAN, GED-Net and TransUnet. As shown in Table 5, the Trans-MBS-S model performed well in all major evaluation indicators. In contrast, the Trans-MBS-NS model was slightly lower than Trans-MBS-S in some indicators. The Dice value of TransUNet ranked third, and its neural network architecture was similar to that of Trans-MBS-NS, but the structure of each MBS module in Trans-MBS-NS was different. AMS-PAN, which ranked fourth, combined multi-scale features with an attention mechanism, and its Dice was slightly lower than that of TransUnet. (SK)U-Net had a weaker segmentation accuracy due to poor multi-scale feature fusion. GED-Net had poor segmentation performance when processing local details or high-noise ultrasound RF images, and its Dice values ​​were all low.

[0103] Table 4 Comparison of segmentation accuracy of different methods on OASBUD dataset

[0104]

[0105] This embodiment also provides a breast tumor ultrasound image segmentation device based on multi-branch skip connection search, such as Fig.10 As shown, including:

[0106] An image acquisition module, wherein the image acquisition module is used to acquire an ultrasonic image of a breast tumor;

[0107] A network construction module, the network construction module is used to construct a candidate neural network, the candidate neural network includes a multi-branch jump connection module, the multi-branch jump connection module obtains feature maps of different scales by channel through multiple branches, and transmits the feature maps of different scales to the decoder of the candidate neural network by using jump connections;

[0108] A search module, the search module is used to search for an optimal multi-branch skip connection module based on the breast tumor ultrasound image and the candidate neural network;

[0109] A training module, wherein the training module is used to construct an image segmentation network using the optimal multi-branch skip connection module, and train the image segmentation network using the breast tumor ultrasound image to obtain a trained image segmentation network;

[0110] The segmentation module is used to perform image segmentation on the breast tumor ultrasound image to be segmented using a trained image segmentation network.

[0111] The same or similar reference numerals correspond to the same or similar components;

[0112] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0113] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A breast tumor ultrasound image segmentation method based on multi-branch skip connection search, characterized in that: The following steps are involved: Obtain ultrasound images of breast tumors; Constructing a candidate neural network, wherein the candidate neural network includes a multi-branch skip connection module, wherein the multi-branch skip connection module obtains feature maps of different scales by channel through multiple branches, and transmits the feature maps of different scales to a decoder of the candidate neural network by using skip connections; Searching for an optimal multi-branch skip connection module based on the breast tumor ultrasound image and the candidate neural network; Using the optimal multi-branch skip connection module to construct an image segmentation network, and using the breast tumor ultrasound image to train the image segmentation network to obtain a trained image segmentation network; The trained image segmentation network is used to perform image segmentation on the breast tumor ultrasound image to be segmented.

2. The method for breast tumor ultrasound image segmentation based on multi-branch skip connection search according to claim 1, characterized in that: The candidate neural network is an encoder-decoder structure, the encoder includes a plurality of encoding blocks, each encoding block extracts features of different scales, the decoder includes decoding blocks corresponding to the plurality of encoding blocks one by one, and each encoding block is connected to the corresponding decoding block through a multi-branch jump connection module.

3. The method for breast tumor ultrasound image segmentation based on multi-branch skip connection search according to claim 2, characterized in that: The multi-branch skip connection module includes four branches B1, B2, B3 and B id ,in: The input is divided into four parts according to the number of channels. The first three parts are input into branches B1, B2 and B3 respectively, and the last part is input into branch B. id In the above figure, branches B1, B2 and B3 respectively include operations of multiple different types of convolution kernels. After the inputs of branches B1, B2 and B3 are processed by operations of different types of convolution kernels, the output feature maps are fused by weighted summation to obtain the outputs of branches B1, B2 and B3. id It is the identity mapping branch; The output of the multi-branch skip connection module is four branches B1, B2, B3 and B id The output of splicing.

4. The method for breast tumor ultrasound image segmentation based on multi-branch skip connection search according to claim 3, characterized in that: The dividing the input into four parts according to the number of channels includes dividing the input into four parts according to the ratio of the number of channels: (1k):(1k):(1k):(1-3k), where k is the number of channels.

5. The method for breast tumor ultrasound image segmentation based on multi-branch skip connection search according to claim 4, characterized in that: The output feature maps are fused by weighted summation, including: Where, X i For branch B i The input, op j () is the operation of the j-th type of convolution kernel, λ ij For branch B i The weight of the j-th type of convolution kernel operation, X i ' is branch B i Output.

6. The method for breast tumor ultrasound image segmentation based on multi-branch skip connection search according to claim 5, characterized in that: The operations of the multiple different types of convolution kernels include 3×3 depth-separable convolution kernel, 5×5 depth-separable convolution kernel, 1×11 depth-separable convolution kernel, 11×1 depth-separable convolution kernel, 9×1 depth-separable convolution kernel and 1×9 depth-separable convolution kernel.

7. The method for breast tumor ultrasound image segmentation based on multi-branch skip connection search according to claim 6, characterized in that: According to the breast tumor ultrasound image and the candidate neural network, an optimal multi-branch skip connection module is searched and obtained, including: Define branch operation weights λ1, λ2 and λ3 for branches B1, B2 and B3 respectively; The branch operation weights λ1, λ2 and λ3 are reparameterized using Gumbel-Softmax to obtain corresponding One-hot encodings; One-hot encoding is used to select the best candidate operations and obtain the optimal multi-branch skip connection module.

8. The method for breast tumor ultrasound image segmentation based on multi-branch skip connection search according to claim 7, characterized in that: When searching for the branch operation weights λ1, λ2 and λ3, different multi-branch jump connection modules in the candidate neural network share the same branch operation weight, or different multi-branch jump connection modules in the candidate neural network each independently set a branch operation weight for update learning.

9. The method for breast tumor ultrasound image segmentation based on multi-branch skip connection search according to claim 8, characterized in that: The method of selecting the best candidate operation by using One-hot encoding includes: One-hot encoding is used to perform dot multiplication with the output of the candidate operation and the results are summed to generate the output of the branch.

10. A breast tumor ultrasound image segmentation device based on multi-branch skip connection search, characterized in that: include: An image acquisition module, wherein the image acquisition module is used to acquire an ultrasonic image of a breast tumor; A network construction module, the network construction module is used to construct a candidate neural network, the candidate neural network includes a multi-branch jump connection module, the multi-branch jump connection module obtains feature maps of different scales by channel through multiple branches, and transmits the feature maps of different scales to the decoder of the candidate neural network by using jump connections; A search module, the search module is used to search for an optimal multi-branch skip connection module based on the breast tumor ultrasound image and the candidate neural network; A training module, wherein the training module is used to construct an image segmentation network using the optimal multi-branch skip connection module, and train the image segmentation network using the breast tumor ultrasound image to obtain a trained image segmentation network; The segmentation module is used to perform image segmentation on the breast tumor ultrasound image to be segmented using a trained image segmentation network.

Images