An improved tissue ultrasound image segmentation light-weight system and method

Abstract

The application discloses an improved tissue ultrasound image segmentation system, which comprises an improved image segmentation network model FSwinNet, realizes efficient image segmentation in a resource-limited environment, and comprises one or more encoder modules, one or more decoder modules and one or more bottleneck blocks; an SFFN hourglass feedforward neural network module is used in a Swin Transformer of the encoder module, and an MBDW lightweight operator is used in the decoder module. The application further discloses a method for carrying out lightweight segmentation of tissue ultrasound images by using the image segmentation system, and has a wide application prospect.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and relates to an improved lightweight system and method for tissue ultrasound image segmentation. Background Technology

[0002] Ultrasound imaging, along with X-ray, CT, and MRI, is considered one of the four major modern medical imaging diagnostic technologies. It is a low-cost, non-invasive examination method that uses no radiation, making it safe for both patients and operators, and especially suitable for pregnant women and children. Traditional desktop ultrasound devices are typically equipped with powerful computing hardware capable of supporting complex image processing and analysis algorithms. Existing deep learning models can be easily integrated into these devices to achieve automated detection, segmentation, and disease diagnosis. However, traditional desktop ultrasound devices are large and inconvenient to carry, limiting their application in mobile healthcare environments; their high cost also limits their widespread adoption in resource-constrained environments. Therefore, miniaturized ultrasound devices integrating edge computing and artificial intelligence will be more conducive to the application and popularization of low-cost, non-invasive ultrasound examination technology.

[0003] In miniaturized ultrasound equipment, the lack of powerful computing resources makes segmentation of ultrasound images with numerous examination locations and high similarity in anatomical features (such as the carotid artery) challenging. This is partly because most deep learning algorithms are improvements on U-Net, which have limitations in convolutional feature extraction and modeling. For example, there is no clear boundary between the intima-media and adventitia of the carotid artery, and plaque tissue is greatly affected by image gain, making it very difficult to effectively extract image features, capture global and local dependencies, and determine the location of the intima-media and plaques. Furthermore, the accuracy and robustness of such multi-class segmentation are also significantly limited. On the other hand, models with large parameters and high computational costs are no longer suitable for miniaturized ultrasound equipment due to limited computing and network resources. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention designs a lightweight tissue ultrasound segmentation network. By incorporating an SFFN hourglass feedforward neural network module and an MBDW lightweight operator, it can achieve higher-precision multi-class segmentation and expand its application scenarios in resource-constrained end-user environments.

[0005] This invention proposes an improved tissue ultrasound image segmentation system, which includes an improved image segmentation network model FSwinNet to achieve efficient image segmentation in resource-constrained environments.

[0006] The image segmentation network model includes one or more encoder modules, one or more decoder modules, and one or more bottleneck blocks; the encoder module uses the SFFN hourglass feedforward neural network module in its Swing Transformer, and the size of the Swing Transformer blocks it contains is adjusted according to the number of encoders and the resolution size; the decoder module uses the MBDW lightweight operator.

[0007] The input and output layers of the SFFN hourglass feedforward neural network module both use depthwise convolution for local feature extraction. At the bottleneck of the SFFN hourglass feedforward neural network module, channel dilation is performed through pointwise convolution followed by compression. Channel dilation captures more feature representations, while channel compression reduces the number of parameters and computational cost; and / or,

[0008] Add residual connections between the input and output of the SFFN hourglass feedforward neural network module; and / or,

[0009] The SFFN hourglass feedforward neural network module is represented by the following formula:

[0010] G'=φ r1,p (φ r3,d (T)),

[0011] SFFN=φ r3,d (φ r1,p (G′))+T,

[0012] Where T is the tensor within the Swin Transformer block after multi-head self-attention MSA, and φ i,d For the i-th depth convolution, φ i,p Let r1 be the i-th dot product, and r3 be the kernel size of 1.

[0013] The MBDW lightweight operator uses convolutional kernels of multiple scales and performs bidirectional feature extraction. Its calculation formula is as follows:

[0014]

[0015] Where α x,y This represents a depthwise convolution operation of x×y. denoted as the dot product from input channel C1 to C2, X represents the input feature vector, BN represents the batch normalization operation, and GELU represents the GELU function activation.

[0016] This invention also provides an improved lightweight segmentation method for tissue ultrasound images, the segmentation method comprising the following steps:

[0017] Step 1: Acquire ultrasound images of the tissue to be segmented from multiple angles, and annotate and preprocess the ultrasound images of the tissue;

[0018] Step 2: Construct an improved FSwinNet model for image segmentation and train and optimize it;

[0019] Step 3: Input the multi-angle ultrasound images of the tissue to be segmented in Step 1 into the constructed FSwinNet model to achieve tissue ultrasound image segmentation.

[0020] In step one, the multi-angle tissue ultrasound images include transverse and / or longitudinal tissue ultrasound images of the left and / or right sides; different tissue structures and backgrounds in the tissue ultrasound images are labeled, and corresponding pixel values ​​are defined respectively; the labeled tissue ultrasound image data are divided into datasets and data augmentation is performed;

[0021] The data augmentation operations include, but are not limited to, data augmentation, adaptive histogram equalization, and Gaussian blurring.

[0022] The data augmentation refers to randomly applying one or more of the following methods to the original image: rotational augmentation, Gaussian blur augmentation, contrast adjustment augmentation, horizontal flip augmentation, vertical flip augmentation, and brightness adjustment augmentation, to augment and balance the proportion of data samples with different tissue structures.

[0023] The adaptive histogram equalization refers to segmenting the original image and applying histogram equalization to the segmented images.

[0024] Gaussian blur refers to smoothing an image and reducing noise through Gaussian blurring operations.

[0025] After the above data augmentation, the image resolution needs to be adjusted to be consistent, normalized, and then input into the subsequent FSwinNet model.

[0026] In step two, the FSwinNet model includes an encoder module, a bottleneck block, and a decoder module. The encoder module is based on the Swin Transformer module, which combines the hourglass feedforward neural network module SFFN, to capture local and global information of the image and alleviate information loss and gradient confusion. The bottleneck block uses dilated convolution to increase the receptive field and enhance the extraction and connection of deep features. The decoder module adopts the multi-scale bidirectional depth-separable convolution operator MBDW, which combines feature extraction at different scales and fuses deep coding characteristics through skip connections to obtain prediction results.

[0027] In the encoder module, this invention utilizes the window-based multi-head self-attention (W-MSA) and moving-window-based multi-head self-attention (SW-MSA) modules within the Swin Transformer module, combined with the SFFN hourglass feedforward neural network and residual connections, to achieve feature extraction and feature fusion. W-MSA is used for self-attention calculation in local regions, segmenting detailed features in the image. Through local window operations, W-MSA can focus on the features of each small region (such as plaques and the intima-media), identifying subtle boundaries. SW-MSA is used for feature interaction between different image regions (such as plaques and vascular lumens). The moving window allows the model to capture global information at multiple levels, helping to distinguish adjacent anatomical structures, such as the boundary between plaques and vascular lumens. The window-based multi-head self-attention (W-MSA) and the moving-window-based multi-head self-attention (SW-MSA) capture local and global information of the image. Pointwise convolution in the SFFN performs channel expansion and compression, enhancing the model's feature representation capability. Linear layers improve the stability of network training, and residual connections enhance gradient propagation.

[0028] During the depthwise convolution process of the SFFN, activation layers are added only after the first depthwise convolutional layer and the last pointwise convolutional layer. Batch normalization is required after the activation layers and after the last linear layer.

[0029] The process of combining the Swin Transformer module of the hourglass feedforward neural network (SFFN) is shown in the following equation:

[0030]

[0031] Among them, z i This represents the output of the i-th Swing Transformer block, LN represents a linear layer, and Swing-Attention represents the Swing Transformer self-attention computation.

[0032] In the bottleneck block, assuming the initial input model has C channels, pointwise convolution is used at the beginning of the bottleneck to compress the channels. Dilated convolution is performed using the MBDW lightweight operator. The output feature map is fused with the input, and after batch normalization, the feature map channels are compressed to C. The output is then activated by the GELU function. The processing of the bottleneck block expands the receptive field of the network, while the input and output resolution and channels of the feature map remain unchanged.

[0033] In the specific implementation of this invention, C here refers to the original feature map channel of the input model. The bottleneck block is usually compressed to 1 / 4 or 1 / 2 of the number of channels, depending on the number of model layers and the input feature map. It is used to adjust and compress the number of channels of the input feature map in order to focus on processing fewer feature dimensions. By reducing the number of channels of the feature map to a certain proportion of the original number of input channels (such as 1 / 4 or 1 / 2), the amount of computation is reduced and key information is retained.

[0034] In the decoder module, bidirectional feature extraction is performed using convolutional kernels of different scales; specifically, based on a multi-scale bidirectional depth-separable convolutional design, the process includes skip fusion, upsampling, channel compression, feature extraction, and pointwise convolution.

[0035] The skip fusion method uses skip connections to connect deep encoded features, allowing the model to capture complex features while retaining important contextual information and output segmentation results that preserve details.

[0036] The upsampling can increase the spatial dimension of the feature map. Upsampling is performed using nearest neighbor interpolation, and then feature extraction is performed using depth-separable 3×3 convolution.

[0037] The channel compression reduces the number of channels to half of the original number through 1×1 convolution, reducing the amount of computation while enhancing the information exchange between channels.

[0038] The feature extraction is achieved by using multi-scale bidirectional depthwise separable convolutions, employing convolution operations with small kernels (i×i) and large kernels (j×j), factoring into bidirectional feature extraction with convolution kernels of 1×i, i×1, 1×j, and j×1. Global and local features are extracted using two-dimensional convolution kernels with different dimensions, the results are summed pairwise, and the outputs of the two paths are concatenated. Residual connections are added to promote gradient flow, resulting in a feature map containing multi-scale spatial information.

[0039] The pointwise convolution, by applying C 1×1 convolution kernels, exchanges information between channels, thereby enhancing the amount of information in the input features.

[0040] This invention presents a multi-class weighted hybrid loss function based on a combination of weighted cross-entropy and Dice loss. When samples are imbalanced, it can effectively guide the model to learn weights by adjusting the contribution of the loss function and the weights of different classes. This function can be used to organize multi-class segmentation tasks.

[0041] The cross-entropy loss is calculated using the following formula:

[0042]

[0043] Where k is the number of categories, ω k It is the weight of the k-th category, y ikp indicates whether sample i belongs to category k. ik It is the probability that sample i belongs to class k, and N is the number of samples;

[0044] The Dice loss is calculated using the following formula:

[0045]

[0046] Where y i p is the true label of sample i. i is the positive class prediction probability of sample i, and N is the number of samples;

[0047] The multi-class weighted mixture loss function is shown in the following equation:

[0048] WDL=α·L WCE +(1-α)L Dice ,

[0049] Where α and (1-α) are the weights of the cross-entropy loss and the Dice loss, respectively.

[0050] The present invention also provides the application of the above-mentioned tissue ultrasound image segmentation system or the above-mentioned segmentation method in medical image analysis, ultrasound image segmentation, remote health monitoring, etc.

[0051] The beneficial effects of this invention include:

[0052] Most existing segmentation networks are based on improvements to U-Net, which have certain limitations in terms of accuracy and robustness across multiple segmentation metrics. This invention implements a lightweight automatic segmentation network, FSwinNet, which can effectively extract image features, capture global and local dependencies, and achieve an average segmentation Dice of 83.42%.

[0053] Currently, there are very few lightweight carotid artery segmentation models on the market. This invention proposes an efficient and lightweight multi-scale bidirectional depth convolutional separable operator. The network model based on this operator achieves excellent segmentation performance on similar medical image segmentation models, with a computational cost that is only 1 / 55 of the approximate accuracy of Unet++. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is the overall architecture diagram of the FSwinNet model of this invention.

[0056] Figure 2 This is a structural diagram of the SFFN module of the present invention.

[0057] Figure 3 This is a diagram of the Swin-T encoder module based on an hourglass feedforward network in the model of this invention.

[0058] Figure 4 This is a diagram of the multi-scale bidirectional depth-separable convolutional module (MBDW) in a specific embodiment of the present invention.

[0059] Figure 5 This is a diagram of the decoder module based on MBDW in this invention.

[0060] Figure 6 This is a schematic diagram of the bottleneck block structure in a specific embodiment of the present invention.

[0061] Figure 7 This is a visualization comparison of the FSwinNet model and mainstream medical segmentation models in a specific embodiment of the present invention.

[0062] Figure 8 This is a test image of AI-automated edge measurement of a carotid artery transverse image in an embodiment of the present invention. The left and right images are the original input image and the labeled result image, respectively.

[0063] Figure 9 This is a test image of the automatic AI measurement of the edge of the longitudinal section image of the carotid artery in an embodiment of the present invention. The left and right images are the original input image and the labeled result image, respectively. Detailed Implementation

[0064] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0065] This invention proposes an improved tissue ultrasound image segmentation system based on a lightweight U-shaped encoder-decoder architecture segmentation network model, FSwinNet, for segmenting multiple types of pathological tissues. This segmentation network model comprises three modules: a Swin Transformer encoder module based on an hourglass feedforward neural network (SFFN), a decoder module based on a multi-scale bidirectional depth-separable convolutional operator (MBDW), and a bottleneck block. The encoder module improves upon the traditional Swin Transformer block's MLP by replacing it with an SFFN hourglass feedforward neural network module, effectively mitigating information loss and gradient confusion. It also captures global dependencies through a linearly complex moving window multi-head self-attention mechanism, improving segmentation accuracy. The decoder module proposes a multi-scale bidirectional depth-separable convolutional operator (MBDW) for upsampling. The multi-scale convolutional kernels, combined with the MBDW, simultaneously capture features at different scales, providing a larger receptive field and reducing computational complexity, making it more efficient than traditional CNN methods. The decoder module also incorporates skip connections to fuse deep encoded features, gradually upsampling and refining the prediction results with limited computational resources. Using dilated convolutions in the bottleneck block to increase the receptive field effectively upsamples the features of deep networks connected to the decoder. This network model achieves excellent segmentation performance on similar medical image segmentation models. Furthermore, its lightweight architecture, based on moving windows and depthwise separable convolutions, reduces computational resources and the number of model parameters, making tissue image segmentation suitable for resource-constrained environments such as mobile devices, embedded systems, and edge computing devices. This helps to make the model more suitable for deployment on resource-limited devices while maintaining segmentation performance.

[0066] In one specific embodiment, the architecture diagram of the FSwinNet model proposed in this invention is as follows: Figure 1 As shown, specifically, the FSwinNet model mainly consists of an encoder module, a decoder module, and a bottleneck block. The basic feature extraction unit in the encoder module is the Swing Transformer block. Based on the Swing Transformer, an improved feedforward neural network, called the Sandglass Feed-Forward Neural Network (SFFN), is proposed and used by the encoder module to extract global and local features. After the image is input into the network, it is first divided into non-overlapping patches of fixed size. Each patch is converted into a one-dimensional sequence vector of a preset dimension through a linear layer. The encoder uses the improved Swing Transformer module to extract features. Multi-head attention allows the model to dynamically adjust weights according to the input data during training, paying attention to relevant context to process relevant data. In addition, the multi-head self-attention mechanism of the moving window is fully utilized to capture global and local information to reduce the computational complexity of the network.

[0067] In one specific implementation, the bottleneck block at the connection between the encoder module and the decoder module performs a depth-separable dilated convolution operation using a convolution kernel with a kernel size of 3, thereby capturing deep features at higher levels and increasing the receptive field without increasing computational cost.

[0068] The decoder module utilizes a novel multi-scale bidirectional depth-separable convolution operator (MBDW) proposed in this invention. Specifically, the input feature map is first upsampled, then subjected to 3×3 depth-separable convolution, followed by pointwise convolution. Feature values ​​are then extracted using the multi-scale bidirectional depth-separable convolution operator. The output is joined with residual connections and further subjected to pointwise convolution to output features. MBDW primarily uses i×i and j×j convolution kernels, and in parallel uses 1×i, i×1, 1×j, and j×1 convolution kernels for bidirectional feature extraction. The results are then summed pairwise. Skip connections are used in the decoder to compensate for the spatial information loss caused by downsampling. Through the skip connections established between the encoder and decoder modules, FSwinNet can retain more details during segmentation, thereby achieving higher accuracy and robustness in tissue ultrasound image segmentation. The final step consists of linear projection layers, outputting pixel-level segmentation predictions. This invention combines the cross-entropy loss function and the Dice loss function to implement a multi-class weighted hybrid loss function, which is suitable for verifying datasets with class imbalance. By using Dice, it emphasizes the accurate segmentation of difficult classes, and by adjusting the contribution of the loss function through coefficients, it assigns different weights to the loss of different classes, focusing on classes that are more difficult to predict during training, thereby achieving a more stable training process and better convergence results.

[0069] Example

[0070] This embodiment designs and implements a clinical index segmentation method for carotid artery ultrasound images based on lightweight networks. The core business process is as follows:

[0071] First, community doctors collect carotid artery ultrasound images of patients using acquisition devices, such as handheld ultrasound devices. In order to obtain comprehensive image information of the patient's carotid artery, community doctors need to collect four types of image information: transverse image of the left carotid artery, longitudinal image of the left carotid artery, transverse image of the right carotid artery, and longitudinal image of the right carotid artery. Then, they upload the images to the system's ultrasound image storage module through an interface, thus completing the carotid artery ultrasound image acquisition process.

[0072] After acquiring ultrasound images, professional ultrasound medical experts and healthcare professionals used LabelMe software to fine-tune and professionally annotate the images, forming a dataset containing 2533 data items. Each set consists of the original image and a LabelMe-formatted annotation file. For better training, the dataset undergoes preprocessing. First, the annotation files are converted into masks, and pixel values ​​(0, 1, 2, 3) are defined according to the annotation categories: background, carotid plaque, carotid intima-media, and vessel lumen. These files are stored in the image and mask directories, respectively. After generating the annotated images, the entire dataset is divided into a 7:1:2 ratio into training, validation, and test sets. Three CSV files are generated according to the partition: train.csv, val.csv, and test.csv. These files record the file paths of the original images and their annotated images for easy access during training and testing. Considering the small sample size and severe class imbalance, this invention, in addition to using a weighted WCE loss and Dice loss hybrid loss function, also performs a series of data augmentations. Data augmentation techniques include data augmentation, adaptive histogram equalization, and Gaussian blur. First, the data is augmented using techniques such as rotation augmentation, Gaussian blur augmentation, contrast adjustment augmentation, horizontal flip augmentation, vertical flip augmentation, and brightness adjustment augmentation. These techniques are applied randomly, individually or in combination, to the original image to train a more robust segmentation model. Considering the imbalanced sample problem, the augmentation ratio for each category is not uniform, aiming to ensure that the sample sizes for carotid plaque, carotid intima-media, and vascular lumen are roughly the same after augmentation, with minimal difference in the proportions of each category.

[0073] Before being input into the network, the original image, including the one after data augmentation, undergoes two image processing methods. First, adaptive histogram equalization is used, dividing the image into small patches and applying histogram equalization to these local regions. Second, Gaussian blur is applied to smooth the image and reduce noise. Finally, the image resolution is adjusted to be consistent, normalized, and then input into the network.

[0074] The architecture diagram of the FSwinNet model proposed in this invention is as follows: Figure 1As shown, specifically, the FSwinNet model mainly consists of an encoder module, a decoder module, and a bottleneck block. The encoder module uses an improved SwinTransformer module to extract features. Multi-head attention allows the model to dynamically adjust weights based on input data during training, paying attention to relevant context to process relevant data. In addition, it fully utilizes the multi-head self-attention mechanism of moving windows to capture global and local information to reduce network computational complexity. The highest layer of the network uses a bottleneck structure module to connect the encoder and decoder modules. The decoder module performs upsampling based on a multi-scale bidirectional depthwise separable convolution operator, while using skip connections of a U-shaped network to fuse encoded features and gradually complete the details of the output image.

[0075] In this invention, the model encoder is a Swing Transformer module that combines an hourglass feedforward neural network (SFFN). Although the multi-head attention mechanism in Vision Transformer has a strong learning ability, its attention complexity is quadratic of the image. To achieve a balance between speed and performance, this invention utilizes window-based multi-head self-attention (W-MSA) and moving window-based multi-head self-attention (SW-MSA) in Swing Transformer to capture local and global information of the image.

[0076] In existing technologies, the Swing Transformer block, after attention, includes a normalization layer and a feedforward neural network as input. The feedforward neural network enhances the features output by the network's MSA layer, which consists of two linear transformation layers and a non-linear activation function. This process can be represented as follows:

[0077] FFN(x)=max(0,W1x+b1)W2+b2 (1)

[0078] Where W1 and W2 are weights, b1 and b2 are biases, and max is the ReLU activation function.

[0079] This invention proposes an improved feedforward neural network based on the Swin Transformer, called the Sandglass Feed-Forward Neural Network (SFFN), for use in the encoder module to extract global and local features. Traditional feedforward neural networks (FFNs) consist of two linear layers connected by the GELU activation function. The first linear layer of the FFN expands the input data channels by a factor of four, and then the second linear layer restores the original channels. For example... Figure 2 The hourglass feedforward neural network (SFFN) of the encoder module in this invention uses depthwise convolution operations in its input and output layers. Both depthwise convolutions are performed in a high-dimensional space. Figure 2 The first and fourth layers (Dwise 3*3) are used to extract local information, which can extract rich information with almost no additional computational cost. Activation layers are added only after the first depthwise convolutional layer and the last pointwise convolutional layer. Batch normalization is required after the activation layers and the last linear layer. The bottleneck of SFFN is addressed by channel dilation followed by compression in pointwise convolution. Compared to existing hourglass blocks, expanding the channels first allows the network to capture more feature representations, increasing the model's learning ability. Compression after channel dilation reduces the number of parameters and computational cost, avoiding excessive computational burden and making the model more lightweight, suitable for running on resource-constrained devices. The bottleneck is kept in the middle of the residual path to save parameters and computational cost. In addition, residual connections are added at the input and output of the SFFN structure, such as... Figure 2 As shown, this design further enhances gradient propagation in the network layers. By using a small number of parameters to perform identity mapping and spatial transformation in a higher dimension, it effectively mitigates information loss and gradient confusion, helping the network achieve better performance in segmentation tasks. The SFFN proposed in this invention is an innovation based on the original FFN, and its calculation formula is as follows:

[0080] G′=φ r1,p (φ r3,d (T)) (2)

[0081] SFFN=φ r3,d (φ r1,p (G′))+T (3)

[0082] Where T is the tensor within the Swing Transformer block after MSA, and φ i,d For depthwise convolution, φ i,p For the dot product, r1 indicates a kernel size of 1, and r3 indicates a kernel size of 3.

[0083] The improved Swing Transformer encoding module structure of this invention is as follows: Figure 3 As shown, the areas marked in red are the areas for improvement. The process can be described as shown in Formulas 4 and 5, where z... i This represents the output of the i-th Swing Transformer block, LN represents a linear layer, and Swing-Attention represents the Swing Transformer self-attention computation:

[0084]

[0085] The decoder module in the FSwinNet model of this invention uses the MBDW operator. To ensure that the decoder module can reconstruct and output the features extracted from the encoder module in a lightweight and accurate manner, this invention proposes an efficient and lightweight depth-separable operator—Multi-Scale Bidirectional Depthwise Separable Convolutions (MBDW), whose module structure is as follows: Figure 4 As shown. In this invention, the operator modules are fixed, consisting of a smaller convolutional kernel and a larger convolutional kernel; the number of decoder and encoder modules in the model is not fixed, x encoder modules or x+1 decoder modules are both possible, with x generally greater than 3. The specific parameters involved in this invention are those that explore relatively good computational complexity and accuracy. First, assuming the feature map dimensions are H×W×C, after input, it enters a parallel multi-branch depthwise convolution stage. MBDW mainly uses 3×3 and 7×7 convolutional kernels for feature extraction. The 3×3 convolution provides a smaller receptive field, capturing local image features and details, while the 7×7 convolution provides a larger receptive field. The combination of both captures features at different scales simultaneously. To further reduce computational complexity, this invention factorizes the three-dimensional 3×3 convolution into the sum of two two-dimensional convolution operations: 1×3 and 3×1. The 7×7 convolution is factored into the sum of convolution operations of 1×7 and 7×1. Four parallel 2D convolutional operations each yield a feature map with C channels. These are then pairwise summed and concatenated to form a feature map with 2C channels. The spatial feature maps are then fused using C 1×1 convolutions. Finally, residual connections are used to connect the initial feature outputs, resulting in an output feature map of H×W×C. The calculation formulas for this operator are shown in Equations 6 and 7.

[0086]

[0087] Where α i,j This represents a depthwise convolution operation of i×j. This represents the input channels C1 to C2, the dot product X represents the input feature vector, BN represents the batch normalization operation, and GELU represents the GELU function activation.

[0088] Based on the aforementioned multi-scale bidirectional depthwise separable convolution operator MBDW, a decoder module was designed. The decoder module, named MBDW Decoder, follows a depthwise separable convolution architecture. Figure 5 As shown in the diagram, the decoder module implementation process comprises five key steps: skip fusion, upsampling, channel compression, feature extraction, and pointwise convolution.

[0089] Skip fusion: Skip connections to deep encoded features improve the accuracy of the model because it allows the model to capture complex features while preserving important contextual information and output segmentation results that retain details.

[0090] Upsampling: The fused feature map is first upsampled to increase its spatial dimension. Upsampling is performed using nearest neighbor interpolation, and then feature extraction is performed using depthwise separable 3×3 convolutions.

[0091] Channel compression: The number of channels is reduced to half of the original number by 1×1 convolution, which reduces the amount of computation and enhances the information exchange between channels.

[0092] Feature extraction: Multi-scale bidirectional depthwise separable convolution is used, with bidirectional feature extraction using 1×i, j×1 and 1×i, j×1 convolution kernels. The results are summed pairwise, and then the outputs of the two paths are concatenated. In addition, residual connections are added to promote gradient flow, and the output is a feature map containing multi-scale spatial information.

[0093] Pointwise convolution: By applying C 1×1 convolution kernels, information is exchanged between channels, thus enhancing the amount of information in the input features.

[0094] The decoder structure described above achieves fusion through multi-branch, factorization-summated depthwise separable convolutions, maintaining efficient feature extraction while using fewer parameters. It not only excels in reconstructing details but also strikes a balance between parameter efficiency and processing power. This design enables the decoder to effectively reconstruct high-quality output in various scenarios, especially in resource-constrained environments.

[0095] The structure of the bottleneck block in the FSwinNet model is as follows: Figure 6 As shown. In this embodiment, C represents the original feature map channels of the input model. First, pointwise convolution is used at the bottleneck to compress the channels to C / 4 to reduce the number of learning parameters. Dilated convolution is performed using the MBDW convolution operator with dilation rates of 1, 2, and 3. The output feature map is then fused with the input, batch normalized, and the feature map channels are compressed to C. The GELU activation function is then applied to the output. By processing the bottleneck block, the receptive field of the network is expanded, while the input and output resolution and channels of the feature map remain unchanged. The application of depthwise dilated convolution and pointwise convolution for channel compression improves the network's performance in processing complex images, while maintaining lightweight and high efficiency.

[0096] The implementation of the method of this invention is mainly aimed at multi-dimensional data structures such as feature maps. The dot product and dilation rate operations are performed on different dimensions of the tensor (feature map). The compression channels and dilation rates described below can be adjusted according to the image resolution.

[0097] This invention proposes a multi-class weighted hybrid loss function based on a combination of weighted cross entropy (WCE) and Dice loss, which can effectively guide the model to learn weights in the carotid artery multi-class segmentation task.

[0098] Datasets often suffer from class imbalance. Using standard cross-entropy loss can cause the class with more samples to dominate during training, resulting in poor training performance for the class with fewer samples. A weighted cross-entropy loss function can be used, as shown in Equation 8:

[0099]

[0100] Where k is the number of categories, ω k It is the weight of the k-th category, y ik p indicates whether sample i belongs to category k. ik This predicts the probability that sample i belongs to class k, where N is the number of samples. WCE mitigates the problem of unbalanced training attention caused by class imbalance by weighting the losses of different classes. The weight coefficients are usually set based on the proportion of each class in the training set, assigning higher weights to classes with lower proportions.

[0101] Dice loss is a region-based loss function, essentially derived from the Dice coefficient. It measures the degree of overlap between the segmentation result and the true value. The Dice coefficient ranges from 0 to 1, with a value of 1 indicating complete overlap between the predicted and true values. The formula is shown in Equation 9:

[0102]

[0103] Where y i p is the true label of sample i. i Let N be the positive class prediction probability of sample i, and N be the number of samples. This experimental dataset exhibits class imbalance. The Dice loss function is well-suited for imbalanced samples; however, its use can easily affect backpropagation during training, leading to training difficulties. While the CE loss function ensures the accuracy of the segmentation results, it may overemphasize the background region and ignore other regions.

[0104] This invention combines the two loss functions mentioned above to implement a multi-class weighted hybrid loss function, where α is the contribution weight of the loss function, calculated as shown in Equation 10:

[0105] WDL=α·L WCE +(1-α)L Dice (10)

[0106] The FSwinNet network of this invention was compared with other popular medical image segmentation networks in a carotid ultrasound image segmentation experiment. The models compared included SwinUnet based on Swin-T, and PSPNet, AttentionUnet, Unet, FPNNet, and Unet++ based on CNN. The input image was 224×224. The hyperparameters of the experiment were set as follows: epoch was 100, batch size was 8, the optimizer was AdamW, the optimizer learning rate lr was 0.0001, and the weight decay was set to 0.001; the weight α of the WDL loss function was set to 0.5, and the class weights ω1, ω2, ω3, and ω4 were set to 1, 2, 2, and 3, respectively; the number of SFFN-based Swin Transformer blocks in the encoder was 2, 2, 2, 2, and the multi-head attention was set to (2, 4, 8, 16).

[0107] Table 1. Comparison of Carotid Artery Segmentation Performance between FSwinNet and Commonly Used Medical Models

[0108]

[0109] Table 2 Comparison of computational complexity performance between FSwinNet and commonly used medical models

[0110]

[0111] Table 1 shows the segmentation results of each network model on the carotid ultrasound image dataset. The segmentation performance in the table is evaluated using Dice and IoU indices for three types of pathological tissue in the carotid artery: plaque, intima-media, and lumen. The average Dice and IoU are calculated. Table 2 shows the model's performance parameters in terms of computational complexity, number of parameters, and maximum runtime memory. Bold values ​​represent the best results, and underlined values ​​represent the second-best results. The tables show that the FSwinNet network of this invention achieves the second-best overall segmentation performance in similar medical segmentation scenarios. However, in terms of computational complexity and number of parameters, the FSwinNet network proposed in this invention has the lowest computational complexity, at only 0.79 Gmac. In contrast, the Unet++ network, which has similar performance, has more than 55 times the computational complexity and 15 times fewer parameters. The above analysis demonstrates that the FSwinNet network proposed in this invention is highly competitive in both segmentation performance and lightweight design.

[0112] To further demonstrate the actual segmentation effect and details of this invention, the present invention visualizes PSPNet, UNet, AttentionUNet, SwinUnet, FPNNet, Unet++, and the FSwinNet network proposed in this invention. Figure 7The visualization results of each network model on the carotid artery test set are shown. The first row contains the original images, including cross-sectional and longitudinal views of the lumen, intima-media, and plaque. The last row is the gold standard, and the remaining rows are the predicted values ​​of each model. The background is black, the lumen is blue, the intima-media is green, and the plaque is red. The figure shows that FSwinNet's segmentation is closest to the gold standard. In the prediction of the fourth column of images, most models failed to accurately identify the morphology of the carotid intima-media and plaque, while FSwinNet segmented the intima-media and plaque quite well.

[0113] Figure 8 and Figure 9 The figures show the inference results of a single transverse and longitudinal ultrasound image analysis using the improved lightweight tissue ultrasound segmentation system of this invention. It can be observed that the carotid lumen and plaque are accurately labeled in the transverse image, and the carotid intima-media is labeled in the longitudinal image, demonstrating correct image segmentation.

[0114] In practical implementation, the FSwinNet lightweight model proposed in this invention exhibits excellent generalization ability. Through transfer learning, the model's pre-trained weights can be applied to ultrasound image segmentation tasks for other human body parts. For example, using a pre-trained carotid artery model as initial weights, with minor fine-tuning for ultrasound image data from other parts such as the liver, heart, and breast, high segmentation accuracy can be achieved. Transfer learning and hyperparameter tuning help reduce the impact of differences in features across different anatomical locations on the model's segmentation performance, enabling the method of this invention to be widely applied to various medical image segmentation tasks. Furthermore, through adaptive data augmentation techniques (such as rotation, flipping, Gaussian blur, etc.), the model can further enhance its robustness and generalization performance when handling image variations between different parts or individuals, thereby ensuring the accuracy and stability of segmentation.

[0115] In addition to the above applications, the image segmentation model in this invention is also applicable to image segmentation tasks for image datasets of various sizes, such as pathological tissue segmentation of ultrasound images, CT images, and MRI images in the medical field. It has significant effects on tissue segmentation that requires global semantic judgment and is also applicable to scenarios such as autonomous driving, security monitoring, and industrial inspection.

[0116] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. An improved tissue ultrasound image segmentation system, characterized in that, The tissue ultrasound image segmentation system includes an improved image segmentation network model FSwinNet, which incorporates an hourglass feedforward neural network module SFFN and a depth-separable convolution operator MBDW to achieve efficient image segmentation in resource-constrained environments. The image segmentation network model FSwinNet includes an encoder module, a bottleneck block, and a decoder module. The encoder module is based on the Swin Transformer module, which combines the hourglass feedforward neural network module SFFN, to capture local and global information of the image and alleviate information loss and gradient confusion. The bottleneck block uses dilated convolution to increase the receptive field and enhance the extraction and connection of deep features. The decoder module adopts the multi-scale bidirectional depth-separable convolution operator MBDW, which combines feature extraction at different scales and fuses deep coding characteristics through skip connections to obtain prediction results. In the encoder module, feature extraction and feature fusion are achieved by using the window-based multi-head self-attention (W-MSA) and moving window-based multi-head self-attention (SW-MSA) modules in the Swin Transformer module, combined with the hourglass feedforward neural network module (SFFN) and residual connections. The input and output layers of the hourglass feedforward neural network module SFFN both use depthwise convolution to extract local features. In the bottleneck of the hourglass feedforward neural network module SFFN, channel dilation is performed by pointwise convolution and then compression is performed. Channel dilation captures more feature representations, and channel compression reduces the number of parameters and computation. Add residual connections between the input and output of the hourglass feedforward neural network module SFFN; During the deep convolution process of the hourglass feedforward neural network module SFFN, activation layers are added only after the first deep convolutional layer and the last pointwise convolutional layer. Batch normalization is required after the activation layer and after the last linear layer. The process of combining the Swin Transformer module with the hourglass feedforward neural network module SFFN is shown in the following equation: ， ， in, This represents the output of the i-th Swing Transformer module, LN represents a linear layer, and Swing-Attention represents the Swing Transformer self-attention computation; In the bottleneck block, pointwise convolution is used to compress the channels at the beginning of the bottleneck. Difference-separable convolution operator MBDW is used for dilated convolution. The output feature map is fused with the input. After batch normalization, the feature map channels are compressed to the initial input channel C of the model. The GELU activation function is then used for the output. The processing of the bottleneck block expands the receptive field of the network, while the input and output resolution and channels of the feature map remain unchanged. In the decoder module, a multi-scale bidirectional depth-separable convolutional design is used to implement processes including skip fusion, upsampling, channel compression, feature extraction, and pointwise convolution. The skip fusion method uses skip connections to connect deep encoded features, allowing the model to capture complex features while retaining important contextual information and output segmentation results that preserve details. The upsampling can increase the spatial dimension of the feature map. Upsampling is performed using nearest neighbor interpolation, and then feature extraction is performed using depth-separable 3×3 convolution. The channel compression reduces the number of channels to half of the original number through 1×1 convolution, reducing the amount of computation while enhancing the information exchange between channels. The feature extraction is performed by using multi-scale bidirectional depthwise separable convolutions with 1×i, i×1, 1×j, j×1 convolution kernels for bidirectional feature extraction. The results are summed pairwise, and then the outputs of the two paths are concatenated together. Residual connections are added to promote gradient flow, and the output is a feature map containing multi-scale spatial information. The pointwise convolution enhances the information content of the input features by applying C 1×1 convolution kernels to exchange information between channels. The hourglass feedforward neural network module SFFN is represented by the following formula: ， ， Where T is the tensor within the Swin Transformer module after undergoing multi-head self-attention MSA. For the i-th depth convolution, For the i-th dot product, r1 indicates that the kernel size is 1, and r3 indicates that the kernel size is 3; The depthwise separable convolution operator MBDW uses convolution kernels of multiple scales and performs bidirectional feature extraction; the calculation formula for the depthwise separable convolution operator MBDW is as follows: ， ， in This represents a depthwise convolution operation of x×y. Indicates from the input channel arrive Dot product, where X represents the input feature vector, BN represents the batch normalization operation, and GELU represents the GELU function activation; A multi-class weighted hybrid loss function based on a combination of weighted cross-entropy loss and Dice loss is used to organize the learning of model weights on multi-class segmentation tasks by adjusting the contribution of the loss function and the weights of different loss functions when the samples are imbalanced. The cross-entropy loss is calculated using the following formula: ， Where k is the number of categories, It is the weight of the k-th category. Indicates whether sample i belongs to category k. It is the probability that sample i belongs to class k, and N is the number of samples; The Dice loss is calculated using the following formula: ， in, It is the true label of sample i. is the positive class prediction probability of sample i, and N is the number of samples; The multi-class weighted mixture loss function is shown in the following equation: ， in, and These are the weights for the cross-entropy loss and the Dice loss, respectively.

2. The application of the tissue ultrasound image segmentation system as described in claim 1 in medical image analysis, ultrasound image segmentation, and remote health monitoring.

3. An improved lightweight segmentation method for tissue ultrasound images, characterized in that, The segmentation method includes the following steps: Step 1: Acquire ultrasound images of the tissue to be segmented from multiple angles, and annotate and preprocess the ultrasound images of the tissue; Step 2: Construct the improved image segmentation network model FSwinNet and train and optimize it; In step two, the image segmentation network model FSwinNet includes an encoder module, a bottleneck block, and a decoder module. The encoder module is based on the Swin Transformer module combined with the hourglass feedforward neural network module SFFN, which captures local and global information of the image and alleviates information loss and gradient confusion. The bottleneck block uses dilated convolution to increase the receptive field and enhance the extraction and connection of deep features. The decoder module adopts the multi-scale bidirectional depth-separable convolution operator MBDW, which combines feature extraction at different scales and fuses deep coding characteristics through skip connections to obtain prediction results. In the encoder module, feature extraction and feature fusion are achieved by using the window-based multi-head self-attention (W-MSA) and moving window-based multi-head self-attention (SW-MSA) modules in the Swin Transformer module, combined with the hourglass feedforward neural network module (SFFN) and residual connections. The input and output layers of the hourglass feedforward neural network module SFFN both use depthwise convolution to extract local features. In the bottleneck of the hourglass feedforward neural network module SFFN, channel dilation is performed by pointwise convolution and then compression is performed. Channel dilation captures more feature representations, and channel compression reduces the number of parameters and computation. Add residual connections between the input and output of the hourglass feedforward neural network module SFFN; During the deep convolution process of the hourglass feedforward neural network module SFFN, activation layers are added only after the first deep convolutional layer and the last pointwise convolutional layer. Batch normalization is required after the activation layer and after the last linear layer. The process of combining the Swin Transformer module with the hourglass feedforward neural network module SFFN is shown in the following equation: ， ， in, This represents the output of the i-th Swing Transformer module, LN represents a linear layer, and Swing-Attention represents the Swing Transformer self-attention computation; In the bottleneck block, pointwise convolution is used to compress the channels at the beginning of the bottleneck. Difference-separable convolution operator MBDW is used for dilated convolution. The output feature map is fused with the input. After batch normalization, the feature map channels are compressed to the initial input channel C of the model. The GELU activation function is then used for the output. The processing of the bottleneck block expands the receptive field of the network, while the input and output resolution and channels of the feature map remain unchanged. In the decoder module, a multi-scale bidirectional depth-separable convolutional design is used to implement processes including skip fusion, upsampling, channel compression, feature extraction, and pointwise convolution. The skip fusion method uses skip connections to connect deep encoded features, allowing the model to capture complex features while retaining important contextual information and output segmentation results that preserve details. The upsampling can increase the spatial dimension of the feature map. Upsampling is performed using nearest neighbor interpolation, and then feature extraction is performed using depth-separable 3×3 convolution. The channel compression reduces the number of channels to half of the original number through 1×1 convolution, reducing the amount of computation while enhancing the information exchange between channels. The feature extraction is performed by using multi-scale bidirectional depthwise separable convolutions with 1×i, i×1, 1×j, j×1 convolution kernels for bidirectional feature extraction. The results are summed pairwise, and then the outputs of the two paths are concatenated together. Residual connections are added to promote gradient flow, and the output is a feature map containing multi-scale spatial information. The pointwise convolution enhances the information content of the input features by applying C 1×1 convolution kernels to exchange information between channels. The hourglass feedforward neural network module SFFN is represented by the following formula: ， ， Where T is the tensor within the Swin Transformer module after undergoing multi-head self-attention MSA. For the i-th depth convolution, For the i-th dot product, r1 indicates that the kernel size is 1, and r3 indicates that the kernel size is 3; The depthwise separable convolution operator MBDW uses convolution kernels of multiple scales and performs bidirectional feature extraction; the calculation formula for the depthwise separable convolution operator MBDW is as follows: ， ， in This represents a depthwise convolution operation of x×y. Indicates from the input channel arrive Dot product, where X represents the input feature vector, BN represents the batch normalization operation, and GELU represents the GELU function activation; In step two, a multi-class weighted hybrid loss function based on the combination of weighted cross-entropy loss and Dice loss is used to organize the learning of model weights on multi-class segmentation tasks by adjusting the contribution of the loss function and the weights of different loss functions when the samples are imbalanced. The cross-entropy loss is calculated using the following formula: ， Where k is the number of categories, It is the weight of the k-th category. Indicates whether sample i belongs to category k. It is the probability that sample i belongs to class k, and N is the number of samples; The Dice loss is calculated using the following formula: ， in, It is the true label of sample i. is the positive class prediction probability of sample i, and N is the number of samples; The multi-class weighted mixture loss function is shown in the following equation: ， in, and These are the weights for the cross-entropy loss and the Dice loss, respectively. Step 3: Input the multi-angle ultrasound images of the tissue to be segmented in Step 1 into the constructed image segmentation network model FSwinNet to achieve tissue ultrasound image segmentation.

4. The segmentation method as described in claim 3, characterized in that, In step one, the multi-angle tissue ultrasound images include transverse and / or longitudinal tissue ultrasound images of the left and / or right sides; different tissue structures and backgrounds in the tissue ultrasound images are labeled, and corresponding pixel values ​​are defined respectively; The labeled tissue ultrasound image data were divided into datasets and then augmented.

5. The application of the segmentation method as described in claim 3 or 4 in medical image analysis, ultrasound image segmentation, and remote health monitoring.

Images