Ultrasound image segmentation system and method based on frequency perception and feature interaction

Abstract

The application discloses an ultrasonic image segmentation system and method based on frequency perception and feature interaction, proposes a multi-scale frequency perception self-attention module, which can effectively capture multi-scale feature information in the self-attention layer, so as to fully utilize the advantages of CNN and the Transformer network to model local and multi-scale features and long-range dependency. In order to enhance the representation of edge and other detail features, a frequency perception network is proposed to learn the interaction between spatial positions in the frequency domain of an image. In addition, a selective feature interaction module is proposed to promote the collaborative interaction of the semantic and boundary branches in the decoder network, and a more accurate segmentation result is obtained under a multi-scale boundary supervision mechanism. The application effectively improves the segmentation performance of medical ultrasonic images.

Description

Technical Field

[0001] This invention relates to the field of image processing, and specifically to an ultrasound image segmentation system and method based on frequency perception and feature interaction. Background Technology

[0002] Medical ultrasound imaging, due to its non-invasive, radiation-free, and real-time characteristics, has become an important technology in the clinical setting, helping physicians accurately assess disease characteristics and the size and distribution of lesions within the body. Furthermore, it plays a crucial role in guiding treatment decisions and monitoring the effectiveness of interventions. In current clinical practice, the review of ultrasound images largely relies on the knowledge and experience of radiologists. However, this manual process is very time-consuming and error-prone. To improve the efficiency of disease screening and diagnosis, reduce the workload of radiologists, and lower the misdiagnosis rate, there is an urgent need for automated segmentation of lesions or organs in ultrasound images. However, this task presents significant challenges. First, the size, shape, and texture patterns of lesions or organs vary greatly in ultrasound images from different cases. Second, ultrasound images exhibit inherent challenges, such as low contrast, high speckle noise, artifacts, and blurred boundaries. These difficulties further complicate the segmentation process, making automated segmentation of ultrasound images a challenging task.

[0003] While methods based on convolutional neural networks (CNNs) and Transformers have achieved some results in this field—for example, Chen et al. proposed a hybrid adaptive attention module to replace traditional convolution operations, thereby guiding the network to adaptively select more robust representations in both channel and spatial dimensions to cope with more complex ultrasound image segmentation (G. Chen, L. Li, Y. Dai, J. Zhang and MHYap, "AAU-Net: An Adaptive Attention U-Net for Breast Lesions Segmentation in Ultrasound Images," in IEEE Transactions on Medical Imaging, May 2023). Wu et al. proposed a cross-image context modeling scheme and cross-image dependency loss to capture more consistent feature representations and mitigate noise interference in ultrasound image segmentation tasks (H. Wu, X. Huang, X. Guo, Z. Wen and J. Qin, "Cross-Image Dependency Modeling for Breast Ultrasound Segmentation," in IEEE Transactions on Medical Imaging, June 2023). Gong et al. proposed a prior-guided attention network for thyroid region enhancement, used for accurate segmentation of ultrasound images (Haifan Gong, Jiaxin Chen, Guanqi Chen, Haofeng Li, Guanbin Li, Fei Chen, "Thyroid region prior guided attention for ultrasound segmentation of thyroid nodules," in Computers in Biology and Medicine, 2023). However, each method has its own advantages and limitations. CNNs are effective in modeling local and multi-scale features, but struggle to capture long-range dependencies; conversely, Transformers excel at modeling long-range dependencies but perform poorly in capturing local and multi-scale features. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-scale frequency-aware self-attention module that enhances the model's ability to perceive different frequency components and strengthens key detail features such as edges. Simultaneously, it can effectively capture multi-scale features within the self-attention layer, modeling both local and global features, and fully leveraging the advantages of CNNs and Transformers.

[0005] The technical solution for achieving the objective of this invention is as follows:

[0006] An ultrasound image segmentation system based on frequency awareness and feature interaction includes an encoder network and a decoder network. The encoder network comprises four stages: the first stage includes a Conv Stem layer and a frequency-aware Transformer block; the subsequent three stages each consist of a Patch Embedding layer and a frequency-aware Transformer block; and the four stages systematically generate pyramid feature maps in descending order of resolution. The decoder network divides the pyramid feature map into semantic and boundary branches. Through a selective feature interaction module, it gradually promotes feature interaction between the two branches from low resolution to high resolution. Simultaneously, a multi-scale boundary supervision mechanism is used to guide and supervise features at different scales to obtain a predicted map for semantic segmentation.

[0007] Furthermore, the frequency-aware Transformer block includes two layers of normalization components, a multi-scale frequency-aware self-attention module, and a feedforward network.

[0008] Furthermore, the multi-scale frequency-aware self-attention module includes a frequency-aware network, two deep convolutional layers, a linear transformation layer, and a multi-head self-attention module.

[0009] Furthermore, the frequency-aware network uses a 2D Fourier transform to obtain frequency domain features for the input feature X. The learnable parameters U are adaptively assigned as weights to each frequency component to obtain... The weighted frequency domain features are transformed into spatial domain features using an inverse Fourier transform. in It is the Fourier transform function. This is the inverse Fourier transform function.

[0010] Furthermore, the deep convolutional layer is used to provide features Dimensionality reduction is achieved by using two depthwise convolutions with different kernels and strides to obtain multi-scale features.

[0011] Furthermore, the linear transformation layer will input feature X or frequency-aware feature X. The result is transformed into three vectors: query, key, and value. The multi-head self-attention module performs multi-head self-attention calculation, and its output is: Where Q, K, and V are three matrices representing the query vector, key vector, and value vector, respectively; d represents the dimension of each attention head; and Softmax represents the normalized exponential function. Finally, the self-attention vectors Attn1 and Attn2 at different scales are concatenated.

[0012] Furthermore, the selective feature interaction module includes a fusion module, a spatial selection module, and a residual enhancement module. The fusion module will process the input semantic branch features F. s and boundary branch features F b The features are connected and enhanced through convolution. The spatial selection module uses a spatial selection mechanism to generate two attention maps from the output features of the fusion module, which are applied to the semantic and boundary branches respectively. The residual enhancement module fuses the semantic and boundary features output by the spatial selection module based on the residual enhancement strategy.

[0013] Furthermore, the output of the space selection module is: in, For the output of the fusion module, This indicates a space selection operation.

[0014] Furthermore, the output of the residual enhancement module is:

[0015]

[0016]

[0017] An ultrasound image segmentation method based on frequency awareness and feature interaction includes the following steps:

[0018] Pyramid features are obtained through four stages of the encoder network. The first stage includes a Conv Stem layer and a frequency-aware Transformer block, while the latter three stages each consist of a Patch Embedding and a frequency-aware Transformer block. The four stages systematically generate pyramid feature maps in descending order of resolution.

[0019] The pyramid feature map is divided into two branches, semantic and boundary, by a decoder network. Through a selective feature interaction module, feature interaction between the two branches is gradually promoted from low resolution to high resolution. At the same time, a multi-scale boundary supervision mechanism is adopted to guide and supervise features at different scales to obtain a predicted map for semantic segmentation.

[0020] Compared with the prior art, the significant effects of this invention are as follows: (1) This invention proposes a new frequency-aware Transformer model, which combines spatial feature information, multi-scale features and long-range dependencies of features in the frequency domain. These components work together to enhance the feature representation capability of the model; (2) The multi-scale frequency-aware self-attention module proposed in this invention effectively learns the interaction between spatial locations within an image in the frequency domain; (3) This invention proposes a collaborative interactive decoder network, which adopts a multi-scale supervision mechanism and a selective feature interaction module to achieve semantic and boundary feature interaction and mutual enhancement. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall framework of the model of the present invention.

[0022] Figure 2 This is a detailed structural diagram of the frequency-aware Transformer module.

[0023] Figure 3 This is a detailed structural diagram of the selective feature interaction module.

[0024] Figure 4 This is a visual comparison chart between the prediction results of the model of this invention and the prediction results of the H2Former model. Detailed Implementation

[0025] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1

[0027] Figure 1 This is the overall framework of the model in this invention, an encoder-decoder structure. The encoder network consists of four stages. The first stage includes a Conv Stem layer and a frequency-aware Transformer block, while the subsequent three stages each consist of a PatchEmbedding layer and a frequency-aware Transformer block. These four stages systematically generate pyramid feature maps in descending order of resolution. The decoder network then divides the pyramid feature maps into two branches: semantic and boundary. By using the proposed selective feature interaction module, feature interaction between the two branches can be progressively facilitated from low to high resolution. Simultaneously, a multi-scale boundary supervision mechanism provides guidance for the boundary feature maps of each stage, ultimately yielding a predicted map for semantic segmentation.

[0028] Multi-scale Frequency-aware Self-attention Module: This module is the core of the pyramid vision Transformer encoder network. The encoder network consists of four stages. The first stage includes a Conv Stem layer and a frequency-aware Transformer block. The subsequent three stages each consist of a Patch Embedding layer and a frequency-aware Transformer block. These four stages systematically generate pyramid feature maps in descending order of resolution. The architecture of the frequency-aware Transformer block includes two-layer normalization components, a multi-scale frequency-aware self-attention module, and a feedforward network. Figure 2 The detailed structure of the frequency-aware Transformer module is shown below, with its core being the proposed multi-scale frequency-aware self-attention module. This self-attention mechanism aims to aggregate multi-scale tokens, thereby deriving keys and values ​​at various scales. Before this aggregation, features are passed through a frequency-aware network, which uses a Fast Fourier Transform to transform the features from the spatial domain to the frequency domain, obtaining weights associated with different frequency components, and then transforms them back to the spatial domain through an inverse transform. The structure of each part of the multi-scale frequency-aware self-attention module is as follows:

[0029] Frequency-aware network: For input features X, it utilizes 2D Fourier transform. Obtain frequency domain features Then, a learnable parameter U is set, and each frequency domain feature is weighted to obtain... The weighted frequency domain features are then subjected to inverse Fourier transform. Transformed into spatial domain features

[0030] Depthwise convolution: used to provide features Dimensionality reduction is achieved by using two depthwise convolutions with different kernels and strides, thus obtaining multi-scale features. Moreover, depthwise convolutions can effectively reduce model parameters compared to ordinary convolutions.

[0031] Linear transformation: transforming input features X or frequency-aware features... Transform it into a query, and use the three vectors of key and value for multi-head self-attention calculation.

[0032] Multi-head self-attention: Where Q, K, and V are three matrices representing the query vector, key vector, and value vector, respectively, d represents the dimension of each attention head, and Softmax represents the normalized exponential function.

[0033] Example 2

[0034] This embodiment provides a Selective Feature Interaction Module to facilitate feature interaction between semantic branches and boundary branches, evolving from small-scale to large-scale. By adopting this progressive refinement strategy, semantic and boundary cues can be better integrated to obtain more accurate segmentation results.

[0035] In the decoder network, convolution is first used to enhance and fuse features, and upsampling operations ensure the feature scale transitions from the current stage to the next. Semantic and boundary features serve as inputs to each selective feature interaction module. Simultaneously, a multi-scale boundary supervision mechanism guides and supervises features at different scales, thereby obtaining more accurate segmentation results. Figure 3 This is the detailed structure of the selective feature interaction module. After fusion and separation, it adaptively aggregates semantic and boundary branch information in the spatial dimension. The structure of each part of the selective feature interaction module is as follows:

[0036] Fusion: For the semantic branch features F of the input s and boundary branch features F b Connect them and enhance them through convolution.

[0037] Spatial selection: A spatial selection mechanism is used to generate two attention maps from the fused semantic and boundary features, which are applied to the semantic and boundary branches respectively. in Represents space selection operation

[0038] Residual Enhancement: The spatial selection weighting process can emphasize the importance of certain spatial locations based on their relevance to the segmentation task. Furthermore, to capture additional details that may be missed in the early stages of processing, we introduce a residual enhancement strategy to fuse previous features.

[0039] Figure 4 This is a visual comparison between the prediction results of the model of this invention and the prediction results of other advanced models. As can be seen from the figure, the segmentation results of the model of this invention are more accurate and can handle details such as edges better.

Claims

1. An ultrasound image segmentation system based on frequency sensing and feature interaction, characterized in that, It includes an encoder network and a decoder network. The encoder network consists of four stages. The first stage includes a Conv Stem layer and a frequency-aware Transformer block. The latter three stages are composed of a Patch Embedding layer and a frequency-aware Transformer block. The four stages systematically generate pyramid feature maps in descending order of resolution. The decoder network divides the pyramid feature map into two branches: semantic and boundary. Through a selective feature interaction module, it gradually promotes feature interaction between the two branches from low resolution to high resolution. At the same time, a multi-scale boundary supervision mechanism is used to guide and supervise features at different scales to obtain a predicted map for semantic segmentation. The frequency-aware Transformer block includes two layers of normalization components, a multi-scale frequency-aware self-attention module, and a feedforward network. The multi-scale frequency-aware self-attention module includes a frequency-aware network, two deep convolutional layers, a linear transform layer, and a multi-head self-attention module. The frequency sensing network for input features Frequency domain features are obtained using 2D Fourier transform. Learnable parameters As a weight, adaptively assigned to each frequency component, resulting in The weighted frequency domain features are transformed into spatial domain features through inverse Fourier transform. ,in It is the Fourier transform function. It is the inverse Fourier transform function; The linear transformation layer will input features or frequency-sensing features The result is transformed into three vectors: query, key, and value. The multi-head self-attention module performs multi-head self-attention calculation, and its output is: ,in , , The three matrices represent the query vector, key vector, and value vector, respectively. d represents the dimension of each attention head, and Softmax represents the normalized exponential function.

2. The ultrasound image segmentation system based on frequency perception and feature interaction according to claim 1, characterized in that, The deep convolutional layer is used to provide features. Dimensionality reduction is achieved by using two depthwise convolutions with different kernels and strides to obtain multi-scale features.

3. The ultrasound image segmentation system based on frequency perception and feature interaction according to claim 1, characterized in that, The selective feature interaction module includes a fusion module, a spatial selection module, and a residual enhancement module. The fusion module integrates the input semantic branch features. and boundary branch features The features are connected and enhanced through convolution. The spatial selection module uses a spatial selection mechanism to generate two attention maps from the output features of the fusion module, which are applied to the semantic and boundary branches respectively. The residual enhancement module fuses the semantic and boundary features output by the spatial selection module based on the residual enhancement strategy.

4. The ultrasound image segmentation system based on frequency perception and feature interaction according to claim 3, characterized in that, The output of the space selection module is: ,in, For the output of the fusion module, This indicates a space selection operation.

5. The ultrasound image segmentation system based on frequency perception and feature interaction according to claim 4, characterized in that, The output of the residual enhancement module is: 。 6. A method for an ultrasound image segmentation system based on frequency sensing and feature interaction as described in any one of claims 1 to 5, characterized in that, Including the following steps: Pyramid features are obtained through the four stages of the encoder network; The four stages consist of a ConvStem layer and a frequency-aware Transformer block in the first stage, and a Patch Embedding and a frequency-aware Transformer block in the latter three stages. The four stages systematically generate pyramid feature maps in descending order of resolution. The pyramid feature map is divided into two branches, semantic and boundary, by a decoder network. Through a selective feature interaction module, feature interaction between the two branches is gradually promoted from low resolution to high resolution. At the same time, a multi-scale boundary supervision mechanism is adopted to guide and supervise features at different scales to obtain a predicted map for semantic segmentation.

Images