Ultrasonic mammary gland lesion area automatic segmentation method based on multi-scale hybrid convolution

Abstract

The invention belongs to the technical field of medical image processing, and particularly relates to an ultrasonic mammary gland lesion area automatic segmentation method based on multi-scale hybrid convolution, which comprises the step of carrying out image segmentation by adopting U2-Net as a network framework, and is characterized in that: at a jump connection part in each residual block, a multi-scale hybrid convolution algorithm is adopted to carry out image segmentation; the feature maps are processed through a multi-scale attention gate and then jump connection with the corresponding feature maps is carried out. According to the method, a hybrid convolution block and a multi-scale attention gate are fused with a U2-Net network structure, and unimportant features are further inhibited and valuable features are enhanced through three different receptive field convolution blocks, namely point-by-point convolution, common convolution and expansion convolution, so that a breast cancer lesion area is quickly and effectively segmented; secondly, a hybrid convolution block is connected to the last convolution layer of the encoder of the residual block and is used for extracting global context information of the small U shape; the method can improve the recognition and segmentation efficiency of the lesion area of the ultrasonic image, reduces the workload of a doctor, and assists the clinical diagnosis.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to an automatic segmentation method for ultrasonic breast lesion areas based on multi-scale mixed convolution. Background Art

[0002] Medical image processing based on deep learning is an interdisciplinary field that integrates computer science, medical imaging and artificial intelligence technology. This field mainly uses deep learning algorithms to automatically analyze and process medical images, thereby helping doctors improve diagnostic accuracy, optimize treatment plans, and promote the progress of medical research. The application of deep learning in medical image processing is mainly reflected in image classification and recognition, target detection and positioning, and image enhancement and reconstruction.

[0003] Medical images intuitively display the physiological changes of the human body and are crucial for disease diagnosis and evaluation. In recent years, artificial intelligence (AI) has made significant progress in medical image segmentation, which can accurately depict the contours and locations of lesions, provide key diagnostic information for pathological analysis, and is an important part of computer-aided diagnosis (CAD) systems.

[0004] Traditional breast ultrasound segmentation algorithms are effective in processing benign tumors with clear boundaries, but are not very effective for irregular tumors and fuzzy boundaries. With the advancement of machine learning, AI has changed traditional solutions in fields such as biomedicine, including protein structure prediction, gene sequence mutation identification, bone disease diagnosis, and organ lesion segmentation. Convolutional neural networks (CNNs) further promote biological research. One-dimensional CNNs are suitable for text or biological sequence data, two-dimensional CNNs are suitable for CT and ultrasound images, and three-dimensional CNNs are designed for magnetic resonance imaging. CNNs are good at capturing local features of biological data, and their application will promote innovation in biological research.

[0005] In the diagnosis and treatment of breast cancer, automatic segmentation of breast ultrasound images helps doctors to efficiently assess the condition. Currently, more and more studies have been conducted on combining CNN to achieve breast ultrasound image segmentation.

[0006] However, in the existing technology, deep convolutional neural networks usually use standard downsampling operations such as max pooling, average pooling and stride convolution, which may lead to information loss, and the lost information (such as boundaries and textures) may be very important for semantic segmentation. In addition, although convolutional neural networks have achieved remarkable performance in various computer vision tasks, this largely relies on a large amount of computing and storage resources, which poses a severe challenge to their effective deployment in resource-constrained environments. Summary of the invention

[0007] For the task of semantic segmentation of medical images, in order to avoid the loss of spatial information caused by pooling features in local neighborhoods, while reducing redundant features to achieve better performance, reduce complexity and computational cost, the present invention proposes an automatic segmentation method for ultrasonic breast lesions based on multi-scale hybrid convolution, which uses U2-Net as the network framework for image segmentation, characterized in that, at the jump connection in each residual block, the feature map is processed by a multi-scale attention gate and then jump-connected with the corresponding feature map.

[0008] Furthermore, the multi-scale attention gate to feature map process includes:

[0009] The input feature map f is processed using the point convolution module, the standard convolution module, and the dilated convolution module respectively, and batch normalization is performed on the processed ones respectively;

[0010] The feature maps after batch normalization are concatenated together and processed through the Relu activation function layer to obtain the feature map f1;

[0011] The feature map f1 is processed by the point convolution module and the Sigmoid activation function layer in turn to obtain the feature map f2;

[0012] The feature map f is fused with the feature map f2 by multiplying them to obtain the feature map f3, and the feature map f3 is fused with the feature map f by adding them to obtain the feature map f4, and the feature map f4 is used as the feature map output by the multi-scale attention gate.

[0013] Preferably, the multi-scale attention gate to feature map process is expressed as:

[0014]

[0015] f s =f×σ2(PointwiseConv(f Concat ))+f

[0016] Among them, f Concat represents the feature map obtained after processing by the Relu activation function layer; σ1(·) represents the Relu activation function, Concat{·} represents the concatenation operation; BN[·] represents the batch normalization operation;

[0017] PointwiseConv(·) represents the pointwise convolution operation, and f represents the feature map of the input multi-scale attention gate;

[0018] OrdinaryConv(·) represents a standard convolution operation; DilationConv(·) represents an expanded convolution operation; f s represents the feature map of the multi-scale attention gate output; σ2(·) represents the Sigmoid activation function.

[0019] Furthermore, a hybrid convolution module is added before the last layer of the encoding part in the residual block.

[0020] Furthermore, the hybrid convolution module consists of multiple cascaded depthwise separable convolutional layers, and the processing process of the depthwise separable convolutional layer on the feature map includes:

[0021] The input feature map is processed in sequence through the deep convolution layer, GELU activation layer, and batch normalization layer, and the processed feature map is fused with the input feature map by addition;

[0022] The fused feature map is processed by the point convolution module, GELU activation layer, and batch normalization layer in turn and serves as the output of the hybrid convolution module.

[0023] Preferably, the processing process of the hybrid convolution module on the feature map is expressed as:

[0024] f′ l = BN(σ3{DepthwiseConv(f l-1 )})+f l-1

[0025] f l = BN(σ3{PointwiseConv(f′ l )})

[0026] Among them, f l-1 represents the output of the lth depthwise separable convolutional layer of the hybrid convolutional module, l is an integer greater than or equal to 0, f0 represents the feature map of the input hybrid convolutional module; BN[·] represents the batch normalization operation; σ3{·} represents the GELU activation function; DepthwiseConv(·) represents the depthwise convolution operation; PointwiseConv(·) represents the pointwise convolution operation.

[0027] The present invention firstly proposes to integrate the hybrid convolution block and the multi-scale attention gate with the U2-Net network structure, and further suppresses unimportant features and enhances valuable features through three convolution blocks with different receptive fields, namely point-by-point convolution, ordinary convolution and dilated convolution, so as to achieve fast and effective segmentation of breast cancer lesion areas; secondly, the present invention also connects the hybrid convolution block to the last convolution layer of the encoder of the residual block to extract the global context information of the small U-shape. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of the U2-Net network framework used in an automatic segmentation method of ultrasonic breast lesion area based on multi-scale mixed convolution of the present invention;

[0029] Figure 2It is a schematic diagram of the structure of the RSU7 module and the RSU4F module in the present invention;

[0030] Figure 3 Schematic diagram of the multi-scale attention gate structure in the present invention;

[0031] Figure 4 Schematic diagram of the hybrid convolution module in the present invention;

[0032] Figure 5 These are simulation diagrams of the present invention and the existing segmentation method, where (a) is the original image of the ultrasonic sample, (b) is the label of the original image of the ultrasonic sample, (c) is a schematic diagram of the segmentation result based on CMU2-Net of the present invention, (d) is a schematic diagram of the segmentation result based on U-Net, and (e) is a schematic diagram of the segmentation result based on U-Net++. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The present invention proposes an automatic segmentation method for ultrasonic breast lesions based on multi-scale mixed convolution, which adopts U2-Net as the network framework for image segmentation. The method is characterized in that, at the jump connection in each residual block, the feature map is processed by a multi-scale attention gate and then jump-connected with the corresponding feature map.

[0035] In this implementation, U2-Net is a large U-shaped structure composed of 11 small U-shaped structures. U2-Net includes three parts, namely a six-level encoder, a five-level encoder, and a feature map fusion module. In the encoding stage, the resolution of each encoder is different, and is usually configured according to the spatial resolution of the input feature map. For larger feature maps, the encoder is required to extract more features. In the later stage of the encoding stage, due to the reduction in the resolution of the feature map, further reducing the resolution of the feature map will lead to the loss of useful information, so the dilated convolution is used to replace the pooling and upsampling operations; in the decoding stage, the structure of the decoder is similar to that of the encoder. The input of each decoder stage is the concatenation of the upsampled feature map of the previous stage and the feature map of the symmetrical encoder stage; the feature fusion module upsamples the side outputs of the last encoding layer and the five decoding stages to the size of the input image, and fuses them in series. See the overall framework diagram of the system. Figure 1 ,All channel numbers and sizes in the figure are the feature map channels and sizes of the input module.

[0036] In this embodiment, if Figure 1 , U2-Net includes the first to sixth encoders and the first to fifth decoders in cascade, wherein the first encoder and the fifth decoder adopt the network structure of RSU7, the second encoder and the fourth decoder adopt the network structure of RSU6, the third encoder and the third decoder adopt the network structure of RSU5, the fourth encoder and the second decoder adopt the network structure of RSU4, and the fifth encoder, the sixth encoder and the first decoder adopt the network structure of RSU4F, wherein:

[0037] The ultrasound image is first input into the RSU7 network. The number of channels and size of the original ultrasound image is 3×H×W. After RSU7, the number of channels becomes 64. After interpolation and downsampling, the size of the feature map is halved. At this time, the feature map is 64×1 / 2H×1 / 2W.

[0038] After the RSU6 module, the number of image channels becomes 128, and then after the downsampling operation, the size of the feature map is halved again. At this time, the feature map is 128×1 / 4H×1 / 4W;

[0039] After passing through RSU5, RSU4, two RSU4F modules and downsampling, the number of channels and size of the feature map become 512×1 / 32H×1 / 32W;

[0040] After upsampling, the input size to the third RSU4F module is 512×1 / 16H×1 / 16W. Then the feature map is decoded by using RSU4, RSU5, RSU6, and RSU7 modules plus sampling operations. The final output is a 64×H×W feature map. After convolution and upsampling in the fusion stage, the image has the same size as the input image but the number of channels is 1, that is, the segmented image.

[0041] For each residual block (i.e., RSU4F, RSU4, RSU5, RSU6, RSU7), a multi-scale attention gate is added to its jump connection to extract features at different scales to capture the diverse information in the image or input data, and by introducing the attention mechanism, the network is guided to focus on more important areas in the feature maps of each scale, thereby improving the accuracy of the task. At the same time, the module effectively integrates features of different scales to ensure the integrity and diversity of information, so that the network can capture information from the global to the local, enhance the robustness of the network, and improve the network's perception of features at different levels. Figure 2 , this embodiment provides a method for adding a multi-scale attention gate, that is, adding the multi-scale attention gate at the jump connection, and the first and last layers of the jump connection do not set the multi-scale attention gate.

[0042] In this embodiment, RSU7 is taken as an example to illustrate the structure of the residual block. Figure 2 , RSU7's encoding uses 7 Rebnconv combination modules to extract features from images. These 7 Rebnconv blocks are cascaded. Each Rebnconv block includes a convolution layer, a batch normalization layer, and a Relu activation layer. The last five Rebnconv modules in the 7 Rebnconv combination modules add downsampling, and the extracted features are processed by an expansion convolution as the output of the encoding part;

[0043] The output of the encoding part passes through 6 cascaded Rebnconv combination modules in sequence, and the last five Rebnconv modules in the 6 Rebnconv combination modules are added to the upsampling module. This embodiment uses 6 Rebnconv convolution modules to convolve the encoding stage features and the feature map after upsampling fusion, restore the image size, enhance the resolution of the feature map, and combine the deep semantic information and the shallow detail information to enhance the segmentation accuracy.

[0044] In this embodiment, RSU7 is a multi-stage convolutional neural network that combines convolution, pooling, dilated convolution (Rebnconv) and residual connection. When dirate is 1, the convolution in Rebnconv is a normal convolution. When dirate in Rebnconv is greater than 1, the convolution in the Rebnconv module is a dilated convolution. After the RSU7 module is initially processed by rebconvin, it gradually extracts features through multiple convolution and pooling layers (rebnconv1 to rebnconv5 and their corresponding 5 maximum pooling layers) to reduce the spatial resolution of the feature map. As an optional implementation, after rebnconv6 further processes the features, a hybrid convolution module is connected to perform hybrid processing on the intermediate feature map, and then dilated convolution is performed with rebnconv7 to enhance the receptive field. After that, the network enters the decoding stage, and the size of the feature map is restored layer by layer through upsampling operations. The upsampling of each layer not only restores the spatial resolution of the feature map, but also retains high-resolution detail information by skipping the feature map in the encoding stage. During the skip connection process, the feature map is further enhanced by the multi-scale attention module. After a series of upsampling and convolution processing, a feature map of the same size as the input image is obtained. The entire module uses multi-scale information fusion and feature recovery to extract and restore details and semantic information in the image during ultrasound image segmentation.

[0045] After completing the construction of the above RSU7 module, it constitutes the first encoder of the entire large Unet encoding stage. There are a total of 6 encoders in the encoding stage, among which,

[0046] The second, third, and fourth encoders are similar to the first one, except that one rebnconv module is reduced in the encoding and decoding stages respectively, and the last rebnconv module uses dilated convolution. Therefore, the RSU6 encoding stage consists of one rebnconvin and six rebnconv modules and four maximum pooling layers. At the same time, the multi-scale attention module of the jump connection is also reduced by one.

[0047] The decoding stage consists of 5 rebnconv modules and 4 upsampling layers. The RSU5 encoding stage consists of one rebnconvin and 5 rebnconv modules and 3 maximum pooling layers, and the decoding stage consists of 4 rebnconv modules and 3 upsampling layers.

[0048] The encoding stage of RSU4 consists of one rebnconvin and four rebnconv modules and two maximum pooling layers, and the decoding stage consists of three rebnconv modules and two upsampling layers.

[0049] After the feature maps pass through the modules of RSU7, RSU6, RSU5, and RSU4, the resolution of the feature maps is relatively low. Further downsampling of these feature maps will lead to the loss of useful context. Therefore, in the last two encoders of the encoding stage, namely RSU4F (F represents the dilated version of RSU), dilated convolution is used to replace the pooling and upsampling operations, which means that all intermediate feature maps of RSU4F have the same resolution as its input feature maps.

[0050] The decoding stage structure of the network is similar to the encoding stage structure, so the entire structure of the network is RSU7, RSU6, RSU5, RSU4, RSU4F, RSU4F downward, and RSU4F, RSU4, RSU5, RSU6, RSU7 upward. The six downward stages are the encoding stage, and the five upward stages are the decoding stage. In the process of module transmission, the decoding stage is the maximum pooling downsampling, and the feature resolution of the image is reduced successively. The decoding stage uses bilinear interpolation to upsample. Each decoder stage uses the upsampled feature map from the previous stage and the concatenation of the feature map from the symmetric encoder stage as input. The last part is the saliency feature fusion stage. Different feature maps are output on different stages of the decoder. They are processed by different convolutional layers to obtain their own output feature maps. Then, the output feature maps are upsampled to the same spatial size as the original input image by bilinear interpolation. This ensures that the spatial size of all output feature maps is consistent. Finally, all upsampled feature maps are spliced ​​in the channel dimension, and then the merged output is obtained through the final convolutional layer.

[0051] In this embodiment, a multi-scale attention gate is added to the jump link of each RSU module. The module structure is as follows: Figure 3As shown in the figure, in order to adaptively select features according to different resolutions, three types of convolutions with different receptive fields are selected to extract features, namely point-wise convolution, ordinary convolution and dilated convolution. Point-wise convolution is a special convolution operation in which the convolution kernel size is 1×1. It linearly transforms the channel of each spatial position without changing the spatial dimension of the feature map. Point-wise convolution is usually used to adjust the number of channels without changing the size of the feature map. Its receptive field is equal to a single pixel, so it can capture local features. Ordinary convolution is the most common convolution operation. It uses a fixed-size convolution kernel to slide on the input feature map to extract local features. The receptive field of ordinary convolution increases as the network deepens, and it can capture a wider range of spatial information. Dilated convolution is a special convolution operation that inserts holes (spaces) in the convolution kernel to increase the receptive field without increasing the number of parameters. Dilated convolution can capture a wider range of contextual information while maintaining computational efficiency, and is effective in capturing long-distance dependencies in images. Each convolution requires a batch normalization layer. Three different convolutions generate feature maps of the same size. The output feature maps are connected before Relu activation, and valuable features are voted through another point-by-point convolution. The multi-scale attention gate feature map process in this embodiment is expressed as:

[0052]

[0053] f s =f×σ2(PointwiseConv(f Concat ))+f

[0054] Among them, f Concat represents the feature map obtained after processing by the Relu activation function layer; σ1(·) represents the Relu activation function, Concat{·} represents the concatenation operation; BN[·] represents the batch normalization operation; PointwiseConv(·) represents the point convolution operation, and f represents the feature map of the input multi-scale attention gate; OrdinaryConv(·) represents the standard convolution operation; DilationConv(·) represents the dilated convolution operation; f s represents the feature map of the multi-scale attention gate output; σ2(·) represents the Sigmoid activation function.

[0055] This embodiment proposes to add a mixed convolution (ConvMixer) module before the last layer of the RSU module encoding, input the feature map into the ConvMixer module, and obtain mixed spatial and position information, thereby enriching the information captured by the network. Figure 4, ConvMixerBlock is a deep module, which consists of multiple deep separable convolutional layers. Each convolutional layer is connected by residual blocks to retain input information. The module performs deep convolution operations on the input, where each input channel is processed using an independent convolution kernel, and then the GELU activation function and batch normalization are applied for normalization. The features after deep convolution processing are added to the original input through residual block connections to form outputs, which are then mapped to channels through point convolution layers and processed by GELU activation functions and batch normalization operations. The above process will be repeated with the defined depth size, gradually extracting and processing features. ConvMixer can perform deep feature learning while maintaining input information. The processing process of the mixed convolution module on the feature map is expressed as:

[0056] f′ l = BN(σ3{DepthwiseConv(f l-1 )})+f l-1

[0057] f l = BN(σ3{PointwiseConv(f′ l )})

[0058] Among them, f l-1 represents the output of the lth depthwise separable convolutional layer of the hybrid convolutional module, l is an integer greater than or equal to 0, f0 represents the feature map of the input hybrid convolutional module; σ3{·} represents the GELU activation function; DepthwiseConv(·) is a depthwise convolution operation.

[0059] In order to verify the effectiveness of the present invention, the performance of U2-Net, U-Net, U-Net++, CMU-Net, and CMU2-Net (the present invention) when actually performing image segmentation tasks is compared in this embodiment.

[0060] As shown in Table 1, CMU2-Net achieves better performance than the other four methods. CMU2-Net combines the multi-scale feature extraction of CMU-Net and the local and global information fusion capabilities of U2-Net, and can provide higher accuracy and efficiency when processing complex images. Through multi-level feature extraction, it can not only capture the global background information of the image, but also accurately refine local details, and perform well in edge detection and image segmentation tasks. CMU2-Net has strong generalization capabilities and can adapt to images of different scales and complex backgrounds. At the same time, it is also optimized in computational efficiency and is suitable for running on resource-constrained devices.

[0061] Table 1

[0062]

[0063] This example also conducts ablation experiments to evaluate the contribution of each module in CMU2-Net. As shown in Table 2, the results show that the performance of the original U2-Net is improved after the introduction of multi-scale attention blocks and ConvMixer blocks respectively. Adding ConvMixer blocks and multi-scale attention gates at the same time further improves the performance of the model, indicating that the multi-scale attention gate can effectively enhance the influence of useful encoder features in the knowledge transfer process.

[0064] Table 2

[0065]

[0066] The simulation visualization image obtained in this embodiment is as follows: Figure 5 As shown, Figure 5 From left to right, each column is the original ultrasound image, label, CMU2-Net prediction result, U-Net prediction result, and U-Net++ prediction result, and each row is an image sample. It can be seen from the cut image that the effect image cut by the present invention is closest to the label and the effect is better than other methods.

[0067] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatic segmentation of ultrasound breast lesions based on multi-scale mixed convolution, using U2-Net as a network framework for image segmentation, characterized in that: At the skip connection in each residual block, the feature map is processed by a multi-scale attention gate and then skip-connected with the corresponding feature map.

2. According to the method for automatic segmentation of ultrasonic breast lesions based on multi-scale mixed convolution in claim 1, the multi-scale attention gate feature map process comprises: The input feature map f is processed using the point convolution module, the standard convolution module, and the dilated convolution module respectively, and batch normalization is performed on the processed ones respectively; The feature maps after batch normalization are concatenated together and processed through the Relu activation function layer to obtain the feature map f1; The feature map f1 is processed by the point convolution module and the Sigmoid activation function layer in turn to obtain the feature map f2; The feature map f is fused with the feature map f2 by multiplying them to obtain the feature map f3, and the feature map f3 is fused with the feature map f by adding them to obtain the feature map f4, and the feature map f4 is used as the feature map output by the multi-scale attention gate.

3. The method for automatic segmentation of ultrasonic breast lesions based on multi-scale mixed convolution according to claim 2, characterized in that: The multi-scale attention gate feature map process is expressed as: f s =f×σ2(PointwiseConv(f Concat ))+f Among them, f Concat represents the feature map obtained after processing by the Relu activation function layer; σ1(·) represents the Relu activation function, Concat{·} represents the concatenation operation; BN[·] represents the batch normalization operation; PointwiseConv(·) represents the point convolution operation, and f represents the feature map of the input multi-scale attention gate; OrdinaryConv(·) represents the standard convolution operation; DilationConv(·) represents the dilated convolution operation; f s represents the feature map of the multi-scale attention gate output; σ2(·) represents the Sigmoid activation function.

4. The method for automatic segmentation of ultrasonic breast lesions based on multi-scale mixed convolution according to claim 1 or 2, characterized in that: A hybrid convolution module is added before the last layer of the encoding part in the residual block.

5. The method for automatic segmentation of ultrasonic breast lesions based on multi-scale mixed convolution according to claim 4, characterized in that: The hybrid convolution module consists of multiple cascaded depth-wise separable convolutional layers. The depth-wise separable convolutional layer processes the feature map as follows: The input feature map is processed in sequence through the deep convolution layer, GELU activation layer, and batch normalization layer, and the processed feature map is fused with the input feature map by addition; The fused feature map is processed by the point convolution module, GELU activation layer, and batch normalization layer in turn and serves as the output of the hybrid convolution module.

6. The method for automatic segmentation of ultrasonic breast lesions based on multi-scale mixed convolution according to claim 5, characterized in that: The processing process of the hybrid convolution module on the feature map is expressed as: f l '=BN(σ3{DepthwiseConv(f l-1 )})+f l-1 fl=BN(σ3{PointwiseConv(fl')}) Among them, f l-1 represents the output of the lth depthwise separable convolutional layer of the hybrid convolutional module, l is an integer greater than or equal to 0, f0 represents the feature map of the input hybrid convolutional module; BN[·] represents the batch normalization operation; σ3{·} represents the GELU activation function; DepthwiseConv(·) represents the depthwise convolution operation; PointwiseConv(·) represents the pointwise convolution operation.

Images