Improved tissue ultrasound image segmentation lightweight system and method

Abstract

The invention discloses an improved tissue ultrasonic image segmentation system, and the system comprises an improved image segmentation network model FSwinNet which achieves the efficient segmentation of an image in a resource-constrained environment. The image segmentation network model comprises one or more encoder modules, one or more decoder modules and one or more bottleneck blocks; the Swin Transformer of the encoder module uses an SFFN (Sandwich Forward Neural Network) sand clock feedforward neural network module, and the decoder module uses an MBDW (Multimedia Broadcast Downlink Weight) lightweight operator. The invention further discloses a method for carrying out tissue ultrasound image lightweight segmentation by using the image segmentation system, and the method has a wide application prospect.

Description

Technical Field

[0001] The invention belongs to the technical field of medical image processing, and relates to an improved lightweight system and method for tissue ultrasonic image segmentation. Background Art

[0002] Ultrasound imaging in ultrasound medicine is listed as one of the four major modern medical imaging diagnostic technologies along with X-ray, CT, and MRI. It is a low-cost, non-invasive examination method that does not use any radiation and is safe for both patients and operators, especially for pregnant women and children. Traditional desktop ultrasound equipment is usually equipped with powerful computing hardware that can support complex image processing and analysis algorithms. Existing deep learning models can be easily integrated into these devices to achieve automatic detection, segmentation, and disease diagnosis. However, traditional desktop ultrasound equipment is large and inconvenient to carry, which limits its application in mobile medical environments; its high cost limits its popularization in resource-constrained environments. Therefore, miniaturized ultrasound equipment integrating edge computing and artificial intelligence will be more conducive to the application and popularization of low-cost ultrasound non-invasive examination technology.

[0003] In miniaturized ultrasound equipment, there is no powerful computing resources, and it is challenging to segment ultrasound images with many inspection locations and high anatomical feature similarity (such as carotid artery). On the one hand, most DL algorithms are based on U-Net and have certain limitations when extracting features through convolution for modeling. For example, there is no clear boundary between the intima and adventitia of the carotid artery, and the plaque tissue is greatly affected by the image gain. It is very difficult to effectively extract image features, capture global and local dependencies, and determine the location of the intima and the plaque; at the same time, the accuracy and robustness of this multi-class segmentation will also be greatly limited. On the other hand, models with large parameters and large computational loads are limited by limited computing resources and network resources, and are no longer suitable for miniaturized ultrasound equipment. Summary of the invention

[0004] In order to address the shortcomings of the prior art, the present invention designs a lightweight tissue ultrasound segmentation network. By adding the SFFN hourglass feedforward neural network module and the MBDW lightweight operator, higher-precision multi-class segmentation can be achieved, expanding the usage scenarios on the resource-constrained end side.

[0005] The present invention proposes an improved tissue ultrasound image segmentation system, which includes an improved image segmentation network model FSwinNet, to achieve efficient image segmentation in a resource-constrained environment;

[0006] The image segmentation network model includes one or more encoder modules, one or more decoder modules and one or more bottleneck blocks; the SFFN hourglass feedforward neural network module is used in the Swin Transformer of the encoder module, the scale of the Swin Transformer block contained therein is adjusted according to the number of encoders and the resolution size, and the MBDW lightweight operator is used in the decoder module.

[0007] The input layer and output layer of the SFFN hourglass feedforward neural network module both use deep convolution to extract local features; in the bottleneck of the SFFN hourglass feedforward neural network module, channel expansion and compression are performed through point-by-point convolution to capture more feature representations through channel expansion, and then the number of parameters and the amount of calculation are reduced through channel compression; and / or,

[0008] Adding a residual connection between the input and output of the SFFN hourglass feedforward neural network module; and / or,

[0009] The SFFN hourglass feedforward neural network module is expressed by the following formula:

[0010] G'=φ r1,p (φ r3,d (T)),

[0011] SFFN=φ r3,d (φ r1,p (G′))+T,

[0012] Among them, T is the tensor after multi-head self-attention MSA in the Swin Transformer block, φ i,d is the i-th depth convolution, φ i,p is the i-th dot product, r1 indicates that the convolution kernel size is 1, and r3 indicates that the convolution kernel size is 3.

[0013] The MBDW lightweight operator uses convolution kernels of multiple scales and performs feature extraction in both directions. Its calculation formula is as follows:

[0014]

[0015] where α x,y represents the depthwise convolution operation of x×y, represents the dot product from input channel C1 to C2, X represents the input feature vector, BN represents the batch normalization operation, and GELU represents the GELU function activation.

[0016] The present invention also provides an improved tissue ultrasound image lightweight segmentation method, the segmentation method comprising the following steps:

[0017] Step 1: Acquire multi-angle tissue ultrasound images to be segmented, and annotate and preprocess the tissue ultrasound images;

[0018] Step 2: Build an improved FSwinNet model for image segmentation and perform training optimization;

[0019] Step 3: Input the multi-angle tissue ultrasound image to be segmented in step 1 into the constructed FSwinNet model to achieve tissue ultrasound image segmentation.

[0020] In step 1, the multi-angle tissue ultrasound image includes a left and / or right transverse and / or longitudinal tissue ultrasound image; different tissue structures and backgrounds in the tissue ultrasound image are annotated, and corresponding pixel values ​​are defined respectively; the annotated tissue ultrasound image data is divided into data sets, and data enhancement is performed;

[0021] The data enhancement operations include but are not limited to data expansion, adaptive histogram equalization, Gaussian blur, etc.;

[0022] The data expansion refers to randomly applying one or more of the following expansion methods, including rotation expansion, Gaussian blur expansion, contrast adjustment expansion, horizontal flip expansion, vertical flip expansion, and brightness adjustment expansion, to the original image to expand and balance the proportion of data samples of different tissue structures;

[0023] The adaptive histogram equalization refers to segmenting the original image and applying histogram equalization to the segmented image;

[0024] The Gaussian blur refers to smoothing the image and reducing noise through a Gaussian blur operation;

[0025] After the above data enhancement, the image resolution needs to be adjusted to be consistent and normalized before being input into the subsequent FSwinNet model.

[0026] In step 2, the FSwinNet model includes an encoder module, a bottleneck block, and a decoder module; the encoder module is based on the Swin Transformer module combined with the hourglass feedforward neural network module SFFN to capture local and global information of the image and alleviate information loss and gradient confusion; the bottleneck block uses dilated convolution to improve the receptive field and enhance the extraction and connection of deep features; the decoder module uses a multi-scale bidirectional deep separable convolution operator MBDW, combines feature extraction of different scales, and fuses deep coding characteristics through jump connections to obtain prediction results.

[0027] In the encoder module, the present invention realizes feature extraction and feature fusion through the window-based multi-head self-attention W-MSA and the moving window-based multi-head self-attention SW-MSA modules in the Swin Transformer module, combined with the SFFN hourglass feedforward neural network and residual connection; wherein, W-MSA is used for self-attention calculation of local areas, segmenting detailed features in the image, and through the operation of local windows, W-MSA can focus on the features of each small area (such as plaques and inner membranes) and identify subtle boundaries. SW-MSA is used for feature interaction between different image areas (such as plaques and vascular lumens), and the moving window allows the model to capture global information at multiple levels, which helps to distinguish adjacent anatomical structures, such as the boundary between plaques and vascular lumens. The window-based multi-head self-attention (W-MSA) and the moving window-based multi-head self-attention (SW-MSA) capture local and global information of the image, expand and compress the channel through point-by-point convolution in SFFN, enhance the feature representation ability of the model, the linear layer improves the stability of network training, and the residual connection enhances the propagation ability of the gradient;

[0028] In the depth convolution process of the SFFN, activation layers are added only after the first depth convolution layer and the last point-by-point convolution layer, and batch normalization is required after the activation layer and after the last linear layer;

[0029] The process of combining the Swin Transformer module with the hourglass feedforward neural network SFFN is as follows:

[0030]

[0031] Among them, z i represents the output of the i-th Swin Transformer block, LN represents the linear layer, and Swin-Attention represents the Swin Transformer self-attention calculation.

[0032] In the bottleneck block, assuming that the initial channel of the input model is C, point-by-point convolution is used to compress the channel at the beginning of the bottleneck, and the MBDW lightweight operator is used for dilated convolution. The output feature map is fused with the input, and then the feature map channel is compressed to C after batch normalization, and the output is output after the GELU activation function. The receptive field of the network is expanded through the processing of the bottleneck block, and the input and output resolution and channels of the feature map remain unchanged.

[0033] In the specific implementation process of the present invention, C here is the original feature map channel of the input model, and the bottleneck block is usually compressed to 1 / 4 or 1 / 2 of the number of channels, depending on the number of model layers and the input feature map, and is used to adjust and compress the number of channels of the input feature map to concentrate on processing fewer feature dimensions; by reducing the number of channels of the feature map to a certain proportion of the original input channel number (such as 1 / 4 or 1 / 2), the amount of calculation can be reduced and key information can be retained.

[0034] In the decoder module, bidirectional feature extraction is performed through convolution kernels of different scales. Specifically, based on the multi-scale bidirectional depth-separable convolution design, a process including jump fusion, upsampling, channel compression, feature extraction, and point-by-point convolution is implemented.

[0035] The jump fusion connects deep encoding features by jump connection, allowing the model to capture complex features while retaining important context information and outputting a segmentation result that retains details;

[0036] The upsampling can increase the spatial dimension of the feature map, use nearest neighbor interpolation for upsampling, and then use depth-separable 3×3 convolution for feature extraction;

[0037] The channel compression reduces the number of channels to half of the original number through 1×1 convolution, which reduces the amount of calculation and enhances the information exchange between channels;

[0038] The feature extraction is performed by using multi-scale bidirectional depth-separable convolution, using convolution operations with small kernel i×i and large kernel j×j, factoring into 1×i, i×1, 1×j, j×1 convolution kernel bidirectional feature extraction, using two-dimensional convolution kernels with one large dimension and one small dimension to extract global and local features, adding the results in pairs, then splicing the outputs of the two paths together, adding residual connections to promote gradient flow, and outputting feature maps containing multi-scale spatial information;

[0039] The point-by-point convolution exchanges information between channels by applying C 1×1 convolution kernels, thereby enhancing the amount of information of the input features.

[0040] The present invention proposes a multi-class weighted hybrid loss function based on a combination of weighted cross entropy and Dice loss. When the samples are unbalanced, the loss function contribution and the weights of different categories can be adjusted. It can effectively guide the model to learn weights when used to organize multi-class segmentation tasks.

[0041] The cross entropy loss is calculated as follows:

[0042]

[0043] Where k is the number of categories, ω k is the weight of the kth category, y ikIndicates whether sample i belongs to category k, p ik is the probability of predicting sample i to be of class k, and N is the number of samples;

[0044] The Dice loss is calculated by the following formula:

[0045]

[0046] where y i is the true label of sample i, p i is the predicted probability of the positive class of sample i, and N is the number of samples;

[0047] The multi-class weighted mixed loss function is shown as follows:

[0048] WDL=α·L WCE +(1-α)L Dice ,

[0049] Among them, α and (1-α) are the weights of cross entropy loss and Dice loss respectively.

[0050] The present invention also provides the above-mentioned tissue ultrasound image segmentation system, or the application of the above-mentioned segmentation method in medical image analysis, ultrasound image segmentation, remote health monitoring, etc.

[0051] The beneficial effects of the present invention include:

[0052] Most of the existing segmentation networks are improved based on U-Net, and have certain limitations in the accuracy and robustness of multi-category segmentation indicators. The present invention implements a lightweight automatic segmentation network FSwinNet, which can effectively extract image features, capture global and local dependencies, and achieve an average Dice of 83.42%.

[0053] Currently, there are few lightweight carotid artery segmentation models on the market. This paper proposes an efficient, lightweight, multi-scale, bidirectional deep convolutional separable operator. The network model implemented based on this operator achieves good segmentation performance on the same type of medical image segmentation models, and the computational complexity is only 1 / 55 of that of Unet++ with approximate accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 It is the overall architecture diagram of the FSwinNet model of the present invention.

[0056] Figure 2 It is the structural diagram of the SFFN module of the model of the present invention.

[0057] Figure 3 It is a block diagram of the Swin-T encoder based on the hourglass feedforward network of the model of the present invention.

[0058] Figure 4 It is a diagram of a multi-scale bidirectional depth-separable convolution module (MBDW) in a specific implementation manner of the present invention.

[0059] Figure 5 It is a decoder module diagram based on MBDW of the present invention.

[0060] Figure 6 It is a schematic diagram of the bottleneck block structure in a specific implementation manner of the present invention.

[0061] Figure 7 It is a visualization comparison effect diagram of the FSwinNet model and the mainstream medical segmentation model in a specific embodiment of the present invention.

[0062] Figure 8 This is an AI automatic measurement test image of the edge of a carotid artery cross-section image in an embodiment of the present invention. The left image and the right image are respectively the input original image and the labeled result image.

[0063] Fig. 9 This is a test image of AI automatic measurement of the edge of a longitudinal section image of a carotid artery in an embodiment of the present invention. The left image and the right image are respectively the input original image and the labeled result image. DETAILED DESCRIPTION

[0064] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0065] The present invention proposes an improved tissue ultrasound image segmentation system, a segmentation network model FSwinNet based on a lightweight U-shaped encoder-decoder architecture, which is used for multi-class pathological tissue segmentation of tissues. The segmentation network model includes three modules: a Swin Transformer encoder module based on an hourglass feedforward neural network SFFN, a decoder module based on a multi-scale bidirectional deep separable convolution operator MBDW, and a bottleneck block. The encoder module improves the MLP of the traditional Swin Transformer block to an SFFN hourglass feedforward neural network module, which can effectively alleviate information loss and gradient confusion, and improve the accuracy of segmentation by capturing global dependencies through a linearly complex moving window multi-head self-attention mechanism. The decoder module proposes a multi-scale bidirectional deep separable convolution operator MBDW for upsampling, and the combination of multi-scale convolution kernels simultaneously captures features of different scales, which can provide a larger receptive field and reduce computational complexity, which is more efficient than the traditional CNN method; in the decoder module, the deep coding features are fused by simultaneous jump connections, and the prediction results are gradually upsampled and refined under limited computational complexity. Using dilated convolution in the bottleneck block to improve the receptive field can effectively connect the features of the deep network to the decoder for upsampling. This network model achieves good segmentation performance on the same type of medical image segmentation model. At the same time, it realizes a lightweight structural design based on moving windows and depth-separable convolution, reduces computing resources and model parameters, and realizes tissue image segmentation in resource-constrained environments, such as mobile devices, embedded systems, and edge computing devices. It helps to make the model more suitable for deployment on resource-constrained devices while ensuring segmentation performance.

[0066] In a specific embodiment, the FSwinNet model architecture diagram proposed in the present invention is as follows: Figure 1 As shown, specifically, the FSwinNet model is mainly composed of an encoder module, a decoder module and a bottleneck block. The basic feature extraction unit in the encoder module is a Swin Transformer block. An improved feedforward neural network is proposed based on the Swin Transformer, called the Sandglass Feed-Forward Neural Network (SFFN), which is used as an encoder module to extract global features and local features. After the image is input into the network, it will first be divided into non-overlapping patches of fixed size. Each patch will be converted into a one-dimensional sequence vector of a preset dimension through a linear layer. The encoder uses an improved Swin Transformer module to extract features. Multi-head attention allows the model to dynamically adjust weights according to input data during training, pay attention to relevant context to process relevant data, and make full use of the multi-head self-attention mechanism of the moving window to capture global and local information to reduce the computational complexity of the network.

[0067] In a specific embodiment, the bottleneck block is located at the connection between the encoder module and the decoder module, and after performing a depth-separable dilated convolution operation using a convolution kernel with a kernel size of 3, it further captures deep features at high levels and improves the receptive field without increasing the amount of computation.

[0068] The decoder module part uses a novel multi-scale bidirectional deep separable convolution operator MBDW operator proposed by the present invention. Specifically, the input feature map is first upsampled and then 3×3 deep separable convolution is performed, followed by point-by-point convolution, and then the multi-scale bidirectional deep separable convolution operator is used to extract the feature value, and its output is added with residual connection and further point-by-point convolution output feature is performed. MBDW mainly uses i×i and j×j convolution kernels, and uses 1×i, i×1, 1×j, j×1 convolution kernels for bidirectional feature extraction in parallel, and the results are added and summed. In the decoder part, jump connection deep coding features are used to supplement the spatial information loss caused by downsampling. By establishing jump connections between the encoder module and the decoder module, FSwinNet can retain more details during the segmentation process, thereby achieving higher accuracy and robustness in tissue ultrasound image segmentation. The last step consists of a linear projection layer, and the output is a pixel-level segmentation prediction. The present invention combines the cross entropy loss function and the Dice loss function to realize a multi-class weighted hybrid loss function, which is suitable for verifying the class imbalance of the data set. It emphasizes the accurate segmentation of difficult categories through Dice, adjusts the loss function contribution through coefficients, assigns different weights to the losses of different categories, and pays attention to the categories that are more difficult to predict during the training process, thereby achieving a more stable training process and better convergence results.

[0069] Example

[0070] In this embodiment, a method for segmenting clinical indicators of carotid artery ultrasound images based on a lightweight network is designed and implemented. The specific process of the core business is as follows:

[0071] First, the community doctor uses acquisition equipment, such as a handheld ultrasound device, to collect the patient's carotid artery ultrasound image. In order to obtain comprehensive image information of the patient's carotid artery, the community doctor needs to collect four types of image information, namely, the patient's left carotid artery cross-sectional image, left carotid artery longitudinal image, right carotid artery cross-sectional image, and right carotid artery longitudinal image. Then upload the information to the system's ultrasound image storage module through the interface, thus completing the carotid artery ultrasound image acquisition process.

[0072] After collecting the ultrasound image, professional ultrasound medical experts and medical staff use LabelMe software to finely screen and professionally annotate the image, forming a set containing 2533 groups of data items, each of which consists of the original image and the LabelMe form annotation file. For better training, the data set will be preprocessed. First, the annotation file is converted into a mask mask, and the pixel values ​​(0, 1, 2, 3) are defined for the background, carotid plaque, carotid intima-media, and vascular lumen according to the annotated categories. These files are stored in the image and mask directories respectively. After forming the annotated picture, the entire data set is divided into a training set, a validation set, and a test set in a ratio of 7:1:2, and three csv files are formed according to the divided sets: train.csv, val.csv, and test.csv. These files record the file paths of the original image and its annotated image, which is convenient for reading during training and testing. Considering that the data sample size is small and the sample categories are seriously unbalanced, the present invention uses a mixed loss function of weighted WCE loss and Dice loss, and also performs a series of enhancements on the data. Data enhancement methods include data expansion, adaptive histogram equalization, Gaussian blur, etc. First, the data is expanded. The expansion methods include rotation change expansion, Gaussian blur expansion, contrast adjustment expansion, horizontal flip expansion, vertical flip expansion, and brightness adjustment expansion. These methods are used randomly on the original image alone or in combination to train a more robust segmentation model. Considering the sample imbalance problem and the proportion of each category, the expansion ratio of each category is inconsistent. Try to make the sample size of the three categories of carotid plaque, carotid intima-media, and vascular lumen roughly the same after expansion, and the proportion of each category is not much different.

[0073] Before being fed into the network, the original image, including the one after data augmentation, will be processed in two ways. The first one uses the adaptive histogram equalization method, which divides the image into small blocks and then applies histogram equalization on these local areas. Then the second one is to use Gaussian blur operation to smooth the image and reduce noise. Finally, the image resolution is adjusted to the same size and normalized before being fed into the network.

[0074] The FSwinNet model architecture proposed in the present invention is shown in the figure Figure 1As shown, specifically, the FSwinNet model is mainly composed of an encoder module, a decoder module and a bottleneck block. The encoder module uses an improved SwinTransformer module to extract features. The multi-head attention allows the model to dynamically adjust weights according to input data during training, pay attention to relevant contexts to process relevant data, and make full use of the multi-head self-attention mechanism of the moving window to capture global and local information to reduce the computational complexity of the network. The highest layer of the network uses a bottleneck structure module to connect the encoder module and the decoder module. The decoder module performs upsampling based on a multi-scale bidirectional depth-separable convolution operator, and uses the jump connection of the U-shaped network to fuse the encoded features to gradually complete the output image details.

[0075] The model encoder in the present invention is a Swin Transformer module combined with an hourglass feedforward neural network SFFN. Although the multi-head attention mechanism in the Vision Transformer has a powerful learning ability, its attention complexity is quadratic to the image. In order to achieve a balance between speed and performance, the present invention uses the window-based multi-head self-attention (W-MSA) and moving window-based multi-head self-attention (SW-MSA) in the Swin Transformer to capture the local and global information of the image.

[0076] The Swin Transformer block in the prior art includes a normalization layer and a feedforward neural network after attention input. The feedforward neural network can promote the features of the network MSA layer output, which consists of two linear transformation layers and a nonlinear activation function. This process can be expressed as follows:

[0077] FFN(x)=max(0,W1x+b1)W2+b2 (1)

[0078] Where W1, W2 are weights, b1, b2 are biases, and max is the ReLU activation function.

[0079] Based on Swin Transformer, this paper proposes an improved feedforward neural network, called Sandglass Feed-Forward Neural Network (SFFN), which is used in the encoder module to extract global features and local features. The traditional feedforward neural network FFN is composed of two linear layers connected by the GELU activation function. The first linear layer of FFN will expand the channel of the input data by 4 times, and then restore the original channel through the second linear layer. Figure 2 The hourglass feedforward neural network SFFN of the encoder module in the present invention uses deep convolution operations in its input layer and output layer. Both deep convolutions are performed in high-dimensional space. The deep convolution ( Figure 2 The first and fourth layers of the Dwise3*3) are used to extract local information, which can extract rich information with almost no additional computational cost. Only activation layers are added after the first deep convolution layer and the last point-by-point convolution layer. Batch normalization is required after the activation layer and the last linear layer. The bottleneck of SFFN is compressed after channel expansion through point-by-point convolution. Compared with the existing hourglass block, the network can capture more feature representations by expanding the channel first, thereby increasing the learning ability of the model. Compression after channel expansion can reduce the number of parameters and the amount of calculation to avoid excessive increase in the computational burden, making the model more lightweight and suitable for running on devices with limited resources. The bottleneck is kept in the middle of the residual path to save parameters and computational costs. In addition, residual connections are added to the input and output parts of the SFFN structure, such as Figure 2 As shown in the figure, the gradient propagation of the network layer is further strengthened. This design uses a small number of parameters to perform identity mapping and spatial transformation in a higher dimension, thereby effectively alleviating information loss and gradient confusion, and can help the network achieve better performance in segmentation tasks. The SFFN proposed in this invention is an innovation based on the original FFN, and its calculation formula is as follows:

[0080] G′=φ r1,p (φ r3,d (T)) (2)

[0081] SFFN=φ r3,d (φ r1,p (G′))+T (3)

[0082] Where T is the tensor after MSA in the Swin Transformer block, φ i,d is the depthwise convolution, φ i,p is the dot product, r1 means the convolution kernel size is 1, and r3 means the convolution kernel size is 3.

[0083] The improved Swin Transformer encoding module structure of the present invention is as follows: Figure 3 As shown, the red box is the improved part. The process can be described as formula 4 and formula 5, where z i represents the output of the i-th Swin Transformer block, LN represents the linear layer, and Swin-Attention represents the Swin Transformer self-attention calculation:

[0084]

[0085] The decoder module in the FSwinNet model of the present invention is designed using the MBDW operator. To ensure that the decoder module is lightweight and accurate in reconstructing and outputting the features extracted from the encoder module, the present invention proposes an efficient and lightweight depth-separable operator, Multi-Scale Bidirectional Depthwise Separable Convolutions (MBDW), whose module structure is as follows: Figure 4 As shown. In the present invention, the operator module is fixed, a smaller convolution kernel and a larger convolution kernel; the number of decoder modules and encoder modules in the model may not be fixed, x encoder modules and x+1 decoding modules are all possible, x is generally greater than 3, and the specific parameters involved in the present invention are parameters with relatively good exploration calculation amount and accuracy. First, assume that the dimension of the feature map is H×W×C. After the input, it enters the parallel multi-branch deep convolution stage. MBDW mainly uses 3×3 and 7×7 convolution kernels for feature extraction. The 3×3 convolution provides a smaller receptive field to capture local features and details of the image. The use of 7×7 convolution can provide a larger receptive field. The combination of the two can capture features of different scales at the same time. In order to further compress the amount of calculation, the present invention factorizes the three-dimensional 3×3 convolution into the sum of two-dimensional convolution operations of 1×3 and 3×1, and the 7×7 convolution is factorized into the sum of 1×7 and 7×1 convolution operations. After four parallel 2D convolution operations, feature maps with C channels are obtained. After adding two by two, they are concatenated into a feature map with 2C channels. The spatial feature maps are fused through C 1×1 convolutions. Finally, the initial feature output is connected using residuals, and the output feature map is H×W×C. The calculation formula of this operator is shown in Formula 6 and Formula 7:

[0086]

[0087] where α i,j represents the i×j depth convolution operation, Represents from input channel C1 to C2, dot product X represents input feature vector, BN represents batch normalization operation, and GELU represents GELU function activation.

[0088] Based on the above multi-scale bidirectional depth-separable convolution operator MBDW, a decoder module is designed. The decoder module is named MBDW Decoder, and its structural design follows the depth-separable convolution architecture. Figure 5 As shown in Figure 2, under this design, the decoder module implementation process includes five key steps: skip fusion, upsampling, channel compression, feature extraction, and point-by-point convolution.

[0089] Skip fusion: Skip connection deep encoding features, combined with skip connection deep features, improves the accuracy of the model because it allows the model to capture complex features while retaining important contextual information and output segmentation results that retain details.

[0090] Upsampling: After fusion, the feature map is first upsampled, which is used to increase its spatial dimension, upsampled using nearest neighbor interpolation, and then feature extracted using a depth-wise separable 3×3 convolution.

[0091] Channel compression: Reduce the number of channels to half through 1×1 convolution, which reduces the amount of calculation while enhancing the information exchange between channels.

[0092] Feature extraction: Use multi-scale bidirectional depth-wise separable convolution, and use 1×i, j×1, 1×i, j×1 convolution kernels for bidirectional feature extraction. The results are added pairwise, and then the outputs of the two paths are concatenated together. In addition, residual connections are added to promote gradient flow, and feature maps containing multi-scale spatial information are output.

[0093] Point-by-point convolution: By applying C 1×1 convolution kernels, information between channels is exchanged, thereby enhancing the amount of information in the input features.

[0094] The above decoder structure is fused through multi-branch, factorized and summed depth-separable convolutions, maintaining efficient feature extraction while using fewer parameters, not only excelling in reconstructing details, but also maintaining a balance between parameter efficiency and processing power. This design enables the decoder to effectively reconstruct high-quality output in a variety of scenarios, especially in resource-limited environments.

[0095] The structure of the bottleneck block in the FSwinNet model is as follows Figure 6 As shown. In this embodiment, C is the original feature map channel of the input model. First, the point-by-point convolution is used to compress the channel to C / 4 at the beginning of the bottleneck to reduce the number of learning parameters. The MBDW convolution operator is used for dilated convolution with expansion rates of 1, 2, and 3 respectively. The output feature map and input are fused together, and then the feature map channel is compressed to C after batch normalization, and the output is output after using the GELU activation function. The receptive field of the network is expanded by the processing of the bottleneck block, and the input and output resolution and channels of the feature map remain unchanged. The application of deep dilated convolution and point-by-point convolution for channel compression improves the performance of the network in processing complex images, while the module remains lightweight and efficient.

[0096] The implementation of the method of the present invention is mainly aimed at the multi-dimensional data structure of feature map. The dot product and expansion rate operations are performed on different dimensions of the tensor (feature map). The compression channel and expansion rate described below can be adjusted according to the image resolution.

[0097] The present invention proposes a multi-class weighted mixed loss function based on a combination of weighted cross entropy (WCE) and Dice loss, which can effectively guide the model to learn weights for the multi-class segmentation task of carotid artery.

[0098] Data sets often have unbalanced categories. Using the standard cross entropy loss will cause the categories with a large number of samples to dominate the training process, resulting in poor training results for categories with a small number of samples. A weighted cross entropy loss function can be used, as shown in Formula 8:

[0099]

[0100] where k is the number of categories, ω k is the weight of the kth category, y ik Indicates whether sample i belongs to category k, p ik is the probability of predicting sample i to be of class k, and N is the number of samples. WCE reduces the problem of imbalanced training attention caused by class imbalance by weighting the losses of different classes. The weight coefficient is usually set based on the proportion of each class in the training set, giving higher weights to classes with low proportions.

[0101] Dice loss is a loss function based on regional analysis. It is essentially derived from the Dice coefficient, which measures the degree of overlap between the segmentation result and the true value. The Dice coefficient ranges from 0 to 1. When it is 1, it means that the predicted value and the true value overlap completely. The formula is shown in Formula 9:

[0102]

[0103] where y i is the true label of sample i, p i is the predicted probability of the positive class of sample i, and N is the number of samples. This experimental dataset has class imbalance. Dice loss is very suitable for imbalanced samples, but the use of Dice can easily affect the back propagation during training and cause training difficulties. Although the CE loss function can ensure the accuracy of the segmentation results, it may focus too much on the background area and ignore other areas.

[0104] The present invention combines the above two loss functions to implement a multi-class weighted hybrid loss function, where α is the loss function contribution weight, calculated as shown in Formula 10:

[0105] WDL=α·L WCE +(1-α)L Dice (10)

[0106] The FSwinNet network of the present invention is compared with other popular medical image segmentation networks in a carotid ultrasound image segmentation experiment. The compared models include SwinUnet based on Swin-T, and PSPNet, AttentionUnet, Unet, FPNNet, and Unet++ based on CNN. The input image is 224×224, and the hyperparameters of the experiment are set as follows: the epoch is 100, the batch size is 8, the optimizer is AdamW, the optimizer learning rate lr is 0.0001, and the weight decay is set to 0.001; the weight α of the WDL loss function is set to 0.5 and the category weights ω1, ω2, ω3, and ω4 are set to 1, 2, 2, and 3, respectively; the number of Swin Transformer blocks based on SFFN in the encoder is 2, 2, 2, and 2 by layer, and the multi-head attention is set to (2, 4, 8, 16).

[0107] Table 1 Comparison of carotid artery segmentation performance between FSwinNet and common medical models

[0108]

[0109] Table 2 Comparison of computational complexity performance of FSwinNet and common medical models

[0110]

[0111] Table 1 shows the segmentation results of each network model on the carotid ultrasound image dataset. The segmentation performance in the table is evaluated by Dice and IoU indicators for three types of pathological tissues of the carotid artery: plaque, intima-media membrane, and lumen, and finally the average Dice and IoU are calculated; Table 2 shows the performance parameters of the model in terms of computational complexity, parameter quantity, and maximum running memory. Among them, the bold value represents the best result, and the underline represents the second best result. It can be concluded from the table that the FSwinNet network of the present invention has achieved the second best performance in the overall performance of the same type of medical segmentation, and in terms of computational complexity and parameter quantity, the FSwinNet network proposed in the present invention is the lowest, with a computational amount of only 0.79Gmac, while the Unet++ network with similar performance has a computational amount of more than 55 times, and the parameter quantity is also reduced by 15 times. The above analysis shows that the FSwinNet network proposed in the present invention is extremely competitive in both segmentation performance and lightweight.

[0112] To further demonstrate the actual segmentation effect and details of the present invention, the present invention visualizes PSPNet, UNet, AttentionUNet, SwinUnet, FPNNet, Unet++ and the FSwinNet network proposed in the present invention. Figure 7The visualization results of each network model on the carotid artery test set are shown. The first row is the original image, which contains the cross-section and longitudinal section of the lumen, intima-media, and plaque. The last row is the gold standard. The remaining rows are the prediction values ​​of each model, with the background as the black part, the lumen as the blue part, the intima-media as the green part, and the plaque as the red part. It can be seen from the figure that FSwinNet's segmentation is closest to the gold standard. In the prediction of the fourth column of images, most models did not accurately identify the morphology of the intima-media and plaque of the carotid artery, while FSwinNet can better segment the intima-media and plaque.

[0113] Figure 8 and Fig. 9 The single inference results of the improved lightweight tissue ultrasonic segmentation system of the present invention on the transverse and longitudinal ultrasonic images are shown respectively. It can be observed that the carotid lumen and carotid plaque are accurately marked for the transverse image, and the carotid intima-media is marked for the longitudinal image, and the corresponding image segmentation can be performed correctly.

[0114] In the specific implementation process, the FSwinNet lightweight model proposed in the present invention has good generalization ability. Through the transfer learning method, the pre-trained weights of the model can be applied to ultrasound image segmentation tasks of other parts of the human body. For example, using the pre-trained carotid artery model as the initial weight, a small amount of fine-tuning of ultrasound image data of other parts such as the liver, heart, breast, etc. can achieve a higher segmentation accuracy. Transfer learning and hyperparameter adjustment help to reduce the impact of feature differences in different anatomical parts on the segmentation effect of the model, so that the method of the present invention can be widely used in segmentation tasks of various medical images. In addition, through adaptive data enhancement techniques (such as rotation, flipping, Gaussian blur, etc.), the model can further enhance its robustness and generalization performance when dealing with image changes between different parts or different individuals, thereby ensuring the accuracy and stability of segmentation. "

[0115] In addition to the above applications, the image segmentation model in the present invention is also suitable for image segmentation tasks in image data sets of various sizes, such as pathological tissue segmentation of ultrasound images, CT images, and magnetic resonance images in the medical field. It has significant effects on tissue segmentation that requires global semantic judgment, and is also suitable for scenarios such as autonomous driving, security monitoring, and industrial inspection.

[0116] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. An improved tissue ultrasound image segmentation system, characterized in that: The tissue ultrasound image segmentation system includes an improved image segmentation network model FSwinNet, which includes an SFFN hourglass feedforward neural network module and an MBDW lightweight operator to achieve efficient image segmentation in a resource-constrained environment; The image segmentation network model includes one or more encoder modules, one or more decoder modules and one or more bottleneck blocks; the Swin Transformer of the encoder module uses an SFFN hourglass feedforward neural network module, and the decoder module uses an MBDW lightweight operator.

2. The tissue ultrasound image segmentation system according to claim 1, characterized in that: The input layer and output layer of the SFFN hourglass feedforward neural network module both use deep convolution to extract local features; in the bottleneck of the SFFN hourglass feedforward neural network module, channel expansion and compression are performed through point-by-point convolution to capture more feature representations through channel expansion, and then the number of parameters and the amount of calculation are reduced through channel compression; and / or, Adding a residual connection between the input and output of the SFFN hourglass feedforward neural network module; and / or, The SFFN hourglass feedforward neural network module is expressed by the following formula: G=φ r1,p (f r3,d (T)), SFFN=φ r3,d (f r1,p (G′))+T, Among them, T is the tensor after multi-head self-attention MSA in the Swin Transformer block, φ i,d is the i-th depth convolution, φ i,p is the i-th dot product, r1 indicates that the convolution kernel size is 1, and r3 indicates that the convolution kernel size is 3.

3. The tissue ultrasound image segmentation system according to claim 1, characterized in that: The MBDW lightweight operator uses convolution kernels of multiple scales and performs feature extraction in both directions. The calculation formula of the MBDW lightweight operator is as follows: where α x,y represents the depthwise convolution operation of x×y, represents the dot product from input channel C1 to C2, X represents the input feature vector, BN represents the batch normalization operation, and GELU represents the GELU function activation.

4. An improved lightweight segmentation method for tissue ultrasound images, characterized in that: The segmentation method comprises the following steps: Step 1: Acquire multi-angle tissue ultrasound images to be segmented, and annotate and preprocess the tissue ultrasound images; Step 2: Build an improved FSwinNet model for image segmentation and perform training optimization; Step 3: Input the multi-angle tissue ultrasound image to be segmented in step 1 into the constructed FSwinNet model to achieve tissue ultrasound image segmentation.

5. The segmentation method according to claim 4, characterized in that: In step 1, the multi-angle tissue ultrasound image includes a left and / or right transverse and / or longitudinal tissue ultrasound image; different tissue structures and backgrounds in the tissue ultrasound image are annotated, and corresponding pixel values ​​are defined respectively; The labeled tissue ultrasound image data is divided into data sets and data enhancement is performed.

6. The segmentation method according to claim 4, characterized in that: In step 2, the FSwinNet model includes an encoder module, a bottleneck block, and a decoder module; the encoder module is based on the SwinTransformer module combined with the hourglass feedforward neural network module SFFN to capture local and global information of the image and alleviate information loss and gradient confusion; the bottleneck block uses a dilated convolution to enhance the receptive field and enhance the extraction and connection of deep features; the decoder module uses a multi-scale bidirectional deep separable convolution operator MBDW, combines feature extraction of different scales, and fuses deep coding characteristics through jump connections to obtain prediction results.

7. The segmentation method according to claim 6, characterized in that: In the encoder module, feature extraction and feature fusion are realized through the window-based multi-head self-attention W-MSA and moving window-based multi-head self-attention SW-MSA modules in the SwinTransformer module, combined with the SFFN hourglass feedforward neural network and residual connection; In the depth convolution process of the SFFN, activation layers are added only after the first depth convolution layer and the last point-by-point convolution layer, and batch normalization is required after the activation layer and after the last linear layer; The process of combining the Swin Transformer module with the hourglass feedforward neural network SFFN is as follows: Among them, z i represents the output of the i-th Swin Transformer block, LN represents the linear layer, and Swin-Attention represents the Swin Transformer self-attention calculation; and / or, In the bottleneck block, point-by-point convolution is used to compress the channel at the beginning of the bottleneck, and the MBDW lightweight operator is used for dilated convolution. The output feature map is fused with the input, and then the feature map channel is compressed to the initial channel C of the input model after batch normalization, and the output is output after the GELU activation function. The receptive field of the network is expanded through the processing of the bottleneck block, and the input and output resolution and channels of the feature map remain unchanged. and / or, In the decoder module, based on the multi-scale bidirectional deep separable convolution design, the process including jump fusion, upsampling, channel compression, feature extraction, and point-by-point convolution is implemented; The jump fusion connects deep encoding features by jump connection, allowing the model to capture complex features while retaining important context information and outputting a segmentation result that retains details; The upsampling can increase the spatial dimension of the feature map, use nearest neighbor interpolation for upsampling, and then use depth-separable 3×3 convolution for feature extraction; The channel compression reduces the number of channels to half of the original number through 1×1 convolution, which reduces the amount of calculation and enhances the information exchange between channels; The feature extraction is performed by using multi-scale bidirectional depthwise separable convolution, using 1×i, i×1, 1×j, j×1 convolution kernels for bidirectional feature extraction, adding the results two by two, then concatenating the outputs of the two paths, adding residual connections to promote gradient flow, and outputting feature maps containing multi-scale spatial information; The point-by-point convolution exchanges information between channels by applying C 1×1 convolution kernels, thereby enhancing the amount of information of the input features.

8. The segmentation method according to claim 4, characterized in that: In step 2, a multi-class weighted hybrid loss function based on a combination of weighted cross entropy loss and Dice loss is used to organize the learning of model weights on multi-class segmentation tasks by adjusting the loss function contribution and the weights of different loss functions when the samples are unbalanced; The cross entropy loss is calculated as follows: Where k is the number of categories, ω k is the weight of the kth category, y ik Indicates whether sample i belongs to category k, p ik is the probability of predicting sample i to be of class k, and N is the number of samples; The Dice loss is calculated by the following formula: where y i is the true label of sample i, p i is the predicted probability of the positive class of sample i, and N is the number of samples; The multi-class weighted mixed loss function is shown as follows: WDL=α·L WCE +(1-α)L Dice , Among them, α and (1-α) are the weights of cross entropy loss and Dice loss respectively.

9. Application of the tissue ultrasound image segmentation system as described in any one of claims 1 to 3, or the segmentation method as described in any one of claims 4 to 8 in medical image analysis, ultrasound image segmentation, and remote health monitoring.

Images