Multi-scale perceptual decoupling ultrasonic image segmentation method

Abstract

The invention discloses a multi-scale perceptual decoupling ultrasonic image segmentation method, and relates to the technical field of image segmentation, and the method comprises the steps: inputting a collected ultrasonic image into a multi-scale perceptual encoder, utilizing a multi-scale perception encoder to generate multi-scale perception encoder feature embedding in the ultrasonic image; mapping the dilution prompt by using a prompt encoder to obtain image prompt feature embedding; and performing multi-level fusion on the multi-scale perception encoder feature embedding and the image prompt feature embedding, and performing feature enhancement on a multi-level fusion result to obtain an ultrasonic image segmentation result. According to the invention, the feature learning ability of the encoder to the target can be enhanced through the multi-scale perception encoder, and the shape boundary adaptive perception module is embedded in the encoder, so that the shape and edge information of the target can be captured more accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a multi-scale perceptual decoupling ultrasound image segmentation method. Background Art

[0002] Ultrasound image segmentation is an important image processing technology, especially in the field of medical imaging. This technology involves the accurate identification and separation of different tissues or structures in ultrasound images. Ultrasound image segmentation based on multi-scale perceptual decoupling is an advanced image processing technology that aims to improve the accuracy and efficiency of ultrasound image segmentation, especially when processing complex medical images. This method can better understand and distinguish various structures and details in the image by capturing and processing image information at different scales.

[0003] Multiscale perception refers to the simultaneous consideration of image features at different scales (size, resolution), so as to capture structural information from coarse to fine, which is particularly important for ultrasound images because the tissue structures in ultrasound images show different characteristics at different scales. The powerful segmentation capability of multiscale perception decoupling in natural images has attracted widespread attention, and some researchers have successfully transferred it to medical image segmentation. However, due to the high coupling between segmentation masks and cue information, the segmentation results are extremely susceptible to the influence of bad cues. At the same time, the fuzzy boundaries of ultrasound images and the shape differences of targets bring great challenges to the target segmentation of ultrasound images.

[0004] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0005] In view of the problems in the related art, the present invention proposes a multi-scale perceptual decoupling ultrasound image segmentation method to overcome the above-mentioned technical problems existing in the existing related art.

[0006] To this end, the specific technical solution adopted by the present invention is as follows:

[0007] A multi-scale perceptual decoupling ultrasound image segmentation method, the multi-scale perceptual decoupling ultrasound image segmentation method comprising:

[0008] Inputting the collected ultrasound image into a multi-scale perceptual encoder, and using the multi-scale perceptual encoder to generate a multi-scale perceptual encoder feature embedding in the ultrasound image;

[0009] Using the cue encoder to map the diluted cues, we get the image cue feature embedding;

[0010] The multi-scale perceptual encoder feature embedding and the image prompt feature embedding are fused at multiple levels, and the multi-level fusion results are feature enhanced to obtain the ultrasound image segmentation result.

[0011] Preferably, inputting the acquired ultrasound image into a multi-scale perceptual encoder, and using the multi-scale perceptual encoder to generate a multi-scale perceptual encoder feature embedding in the ultrasound image comprises:

[0012] The backbone network in the multi-scale perceptual encoder is used to perform preliminary encoding processing on the ultrasound image to obtain the global context features of the image;

[0013] The auxiliary network in the multi-scale perceptual encoder is used to extract detail features in the ultrasound image, and the detail features are sequentially spliced ​​and dimensionally transformed to obtain local detail features of the ultrasound image;

[0014] The local detail features are integrated into the backbone network for fine-tuning, and a resonance mechanism is introduced between the backbone network and the auxiliary network to achieve collaborative optimization between feature maps.

[0015] Preferably, the backbone network in the multi-scale perceptual encoder is used to perform preliminary encoding processing on the ultrasound image, and the global context features of the image obtained include:

[0016] Train the SAM encoder on a predefined dataset and use the SAM encoder weights to initialize the backbone network of the multi-scale perceptual encoder;

[0017] Use the predefined convolutional layer of the backbone network to perform preliminary encoding processing on the ultrasound image;

[0018] The initially encoded ultrasound image passes through several visual transformers including windowed attention and residual connections to obtain the global context features of the image.

[0019] Preferably, the auxiliary network in the multi-scale perceptual encoder is used to extract detail features in the ultrasound image, and the detail features are sequentially spliced ​​and dimensionally converted to obtain local detail features of the ultrasound image, including:

[0020] Down-sampling the ultrasound image using a predetermined convolution module to obtain a number of initial feature maps;

[0021] The initial feature map is used as the input of the shape boundary adaptive perception module, and the shape boundary adaptive perception module is used to capture the shape and boundary information in the initial feature map to obtain a detail feature map;

[0022] The detail feature map is sequentially subjected to channel splicing and dimension conversion to obtain the local detail features of the ultrasound image.

[0023] Preferably, the shape and boundary information in the initial feature map is captured by the shape boundary adaptive perception module to obtain the detail feature map, including:

[0024] Extract the shape information weight from the initial feature map, multiply the shape information weight with the initial feature map, and then perform channel concatenation to obtain a shape enhanced feature map;

[0025] The filter is used to sharpen the edge features of the ultrasonic image segmentation target to obtain an edge feature map, and the edge feature map is integrated and channel-joined with the initial feature map to obtain an edge enhancement feature map;

[0026] The shape enhancement feature map and the edge enhancement feature map are channel-wise concatenated, and the final enhanced feature map is generated by combining the broadcast mechanism.

[0027] Preferably, channel-joining the shape enhancement feature map and the edge enhancement feature map, and combining the broadcast mechanism to generate the final enhancement feature map includes:

[0028] The shape enhancement feature map and the edge enhancement feature map are channel-joined to obtain a fused feature map;

[0029] The global average pooling layer is used to perform feature abstraction and dimension lifting and lowering on the fused feature map in turn to obtain an enhanced attention map.

[0030] Calculate the channel dimension weight, multiply the enhanced attention map by the channel dimension weight, and obtain the feature map after channel dimension enhancement;

[0031] Get the average value of the position corresponding to the feature map after channel dimension enhancement in each channel dimension, and calculate the weight of the spatial dimension;

[0032] Combined with the broadcast mechanism, the spatial dimension weighted fusion feature maps are added together to obtain the final enhanced feature map.

[0033] Preferably, the local detail features are integrated into the backbone network for fine-tuning, and a resonance mechanism is introduced between the backbone network and the auxiliary network to achieve collaborative optimization between feature maps, including:

[0034] The global context features and local detail features of the image are multiplied by the weights of the backbone network and the auxiliary network respectively to achieve feature resonance between the backbone network and the auxiliary network.

[0035] The local detail features extracted by the auxiliary network are integrated into the multi-head attention module of the backbone network for fine-tuning to generate multi-scale perceptual encoder feature embedding.

[0036] Preferably, the local detail features extracted by the auxiliary network are integrated into the multi-head attention module of the backbone network for fine-tuning, and the multi-scale perceptual encoder feature embedding is generated, including:

[0037] The key vector and value vector are calculated by combining the local detail features, and the query vector is calculated based on the backbone features generated after window attention and normalization of the local detail features;

[0038] Based on the key vector, value vector and query vector, the fusion feature is calculated through the attention mechanism;

[0039] The backbone features and fusion features are residually connected and added to the multi-head attention module of the backbone network to generate the final fusion features;

[0040] The final fused features are added to the global context features to obtain the multi-scale perceptual encoder feature embedding.

[0041] Preferably, the multi-scale perceptual encoder feature embedding and the image prompt feature embedding are multi-level fused, and the multi-level fusion result is feature enhanced to obtain the ultrasound image segmentation result including:

[0042] Input the multi-scale perceptual encoder feature embedding and the image cue feature embedding into the cross attention to obtain the mask embedding;

[0043] The predefined convolutional module is used to perform feature dimension conversion and weight extraction on multi-scale perceptual encoder feature embedding, image cue feature embedding and mask embedding in turn;

[0044] The background attention map is calculated based on the weight extraction result, and the background attention map is element-wise multiplied with the multi-scale perceptual encoder feature embedding and then input into the spatial attention module for feature enhancement;

[0045] The local detail features and image hint features are embedded for channel splicing and input into the convolution module for feature fusion, and the ultrasound image is segmented in combination with the texture information of the ultrasound image;

[0046] The mask embedding and local detail features are channel-concatenated and input into the upsampling layer to restore the feature map size and obtain the ultrasound image segmentation result.

[0047] Preferably, the expression of the ultrasound image segmentation result is:

[0048] F o =Up(Conv(Sat((1-S(F k )⊙F ma ))+Conv(F k ΘF c )))

[0049] In the formula, F orepresents the ultrasound image segmentation result, Up represents the upsampling layer; Conv represents the convolution module; Sat represents the spatial attention module; S represents the Sigmoid function; ⊙ represents the dot product; Θ represents the channel concatenation; F k represents mask embedding; F ma represents the multi-scale perceptual encoder feature embedding; F c Represents local detail features.

[0050] The beneficial effects of the present invention are:

[0051] The present invention can enhance the feature learning ability of the encoder for the target through a multi-scale perceptual encoder, and embed a shape boundary adaptive perception module in the encoder, so as to more accurately capture the shape and edge information of the target; at the same time, in order to reduce the coupling between the segmentation result and the prompt information, the decoupling enhancement decoder performs cross-attention enhancement on the features and multi-level fusion features from the multi-scale perceptual encoder, thereby reducing the proportion of prompt information in the feature learning process and reducing the sensitivity of the network to bad prompt information; finally, the fused features are input into the spatial attention module for feature enhancement, thereby obtaining the final segmentation result, thereby improving the segmentation quality of the ultrasonic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a flowchart of a multi-scale perceptual decoupling ultrasound image segmentation method according to an embodiment of the present invention;

[0054] Figure 2 It is a MDA-SAM framework diagram in a multi-scale perceptual decoupling ultrasound image segmentation method according to an embodiment of the present invention;

[0055] Figure 3 It is a framework diagram of a dual perception module in a multi-scale perception decoupling ultrasound image segmentation method according to an embodiment of the present invention;

[0056] Figure 4 It is a schematic diagram of segmentation results in a multi-scale perceptual decoupling ultrasound image segmentation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments. They can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementation methods and advantages of the present invention.

[0058] According to an embodiment of the present invention, a multi-scale perceptual decoupling ultrasound image segmentation method is provided.

[0059] Figure 2 The Multi-scale Aware Decoupling SAM (MAD-SAM) framework for ultrasound image segmentation is presented. It is mainly composed of a Multi-scale Aware Encoder (MA-Encoder), a Prompt Encoder, and a Decoupling Enhanced Decoder (DE-Decoder). An ultrasound image is input with multi-scale awareness decoupling. First, the Multi-scale Aware Encoder is used to generate a multi-scale shape and boundary-aware enhanced image embedding; the Prompt Encoder is used to map the diluted prompts (such as points and boxes) to obtain the prompt embedding; the Decoupling Enhanced Decoder uses the multi-level fusion of the image and prompt embedding to accurately predict the segmentation result. In order to solve the problem of poor ultrasound image quality and large size variation of the segmentation target in the ultrasound image, the present invention can effectively alleviate the coupling between the segmentation result and the prompt information through the innovative Multi-scale Aware Decoupling SAM framework, thereby minimizing the sensitivity of SAM to the randomness of the prompt.

[0060] It should be noted that if Figure 2 As shown in the figure, MDA-SAM consists of three parts: MA-Encoder, PromptDecoder and DE-Decoder. The main branch of MA-Encoder uses pre-trained weights for feature extraction, while the auxiliary branch fine-tunes the main branch by extracting multi-perceptual detail features from ultrasound images. The Prompt encoder provides interactive segmentation simulation. DE-Decoder reduces the dependence of the result on prompts by multi-level fusion of image and prompt embeddings.

[0061] exist Figure 2In the figure, Patch Embedding represents patch embedding; Window Transformer represents window transformer; Layer Norm represents layer normalization; Multi-head Attention represents multi-head attention; Feedforward Network represents forward propagation; Prompt Encoder represents prompt encoder; Cross Attention represents cross attention; SEAPM represents shape boundary adaptive perception; Down represents downsampling; Interactive Fusion represents cross fusion; Channel Expander represents channel expansion; Feature Resonance represents feature resonance; Up represents upsampling; Main-Branch represents main branch; Auxiliary-Branch represents auxiliary branch; Point Prompts represents point prompts; Box Prompts represents box prompts; Element-wise Addention represents element-wise addition; Element-wise Product represents element-wise multiplication; Channel-wise Concatenation represents channel-wise concatenation; Spatial Attention means spatial attention; Query means query vector; Key means key vector; Value means value vector; DE-Decoder means decoupled enhanced decoder; ConvTranspose2d means two-dimensional deconvolution; Maxpool+Conv2d+LN+GELU: Maxpool means maximum pooling; Conv2d means two-dimensional convolution; LN (Layer Norm) means layer normalization; GELU (Gaussian Error Linear Unit) means Gaussian error linear unit.

[0062] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to the multi-scale perceptual decoupling ultrasound image segmentation method of an embodiment of the present invention, the multi-scale perceptual decoupling ultrasound image segmentation method includes:

[0063] S1. Input the collected ultrasound image into a multi-scale perceptual encoder, and use the multi-scale perceptual encoder to generate a multi-scale perceptual encoder feature embedding in the ultrasound image.

[0064] It should be noted that the superior segmentation ability of SAM is attributed to its large-capacity image encoder. When applying SAM to medical images, existing studies usually adopt two strategies: one is to directly utilize the pre-trained SAM encoder, and the other is to fine-tune it for a specific dataset. However, the low contrast and fuzzy boundaries of ultrasound images, as well as the significant shape and position changes of objects, increase the difficulty of accurately segmenting objects. Therefore, it is not enough to rely solely on a set of image encoders based on Vision Transformer (ViT) to segment ultrasound images.

[0065] Since Convolutional Neural Network (CNN) can help ViT better capture local spatial information. Inspired by this, the present invention proposes a CNN-based multi-scale perceptual encoder, which uses multi-scale features and shape and boundary perception modules to capture the features of ultrasound images. By embedding these features into the multi-head attention mechanism of the backbone encoder, the encoder is fine-tuned, the feature capture capability of the encoder is enhanced, and thus the accuracy of ultrasound image segmentation is improved. Specifically, the MA-Encoder proposed in the present invention has two feature extraction branches, the backbone branch and the auxiliary branch.

[0066] The collected ultrasound image is input into the multi-scale perceptual encoder, and the multi-scale perceptual encoder is used to generate a multi-scale perceptual encoder feature embedding in the ultrasound image, including:

[0067] The backbone network in the multi-scale perceptual encoder is used to perform preliminary encoding processing on the ultrasound image to obtain the global context features of the image.

[0068] Among them, the backbone network in the multi-scale perceptual encoder is used to perform preliminary encoding processing on the ultrasound image, and the global context features of the image are obtained, including:

[0069] Train the SAM encoder on a predefined dataset and use the SAM encoder weights to initialize the backbone network of the multi-scale perceptual encoder;

[0070] Use the predefined convolutional layer of the backbone network to perform preliminary encoding processing on the ultrasound image;

[0071] The initially encoded ultrasound image passes through several visual transformers including windowed attention and residual connections to obtain the global context features of the image.

[0072] It should be noted that the main branch (Main-Branch) uses the SAM image encoder with the same architecture as the visual converter, which is pre-trained on the SAM-1B dataset and obtains the weights of image encoders of three different scales. In this invention, the weights of ViT-B are selected to initialize the main branch of MA-Encoder. First, the input 256×256 ultrasound image F is processed by the main branch, and the image is preliminarily encoded using a convolutional layer with a convolution kernel size of 8×8 and a stride of 8; the image is passed through multiple ViT blocks containing window attention and residual connections to obtain the image embedding F of the main network v .

[0073] The auxiliary network in the multi-scale perceptual encoder is used to extract detail features in the ultrasound image, and the detail features are sequentially spliced ​​and dimensionally transformed to obtain local detail features of the ultrasound image.

[0074] Among them, the auxiliary network in the multi-scale perceptual encoder is used to extract the detail features in the ultrasound image, and the detail features are sequentially spliced ​​and dimensionally transformed to obtain the local detail features of the ultrasound image, including:

[0075] Down-sampling the ultrasound image using a predetermined convolution module to obtain a number of initial feature maps;

[0076] The initial feature map is used as the input of the shape boundary adaptive perception module, and the shape boundary adaptive perception module is used to capture the shape and boundary information in the initial feature map to obtain a detail feature map;

[0077] The detail feature map is sequentially subjected to channel splicing and dimension conversion to obtain the local detail features of the ultrasound image.

[0078] It should be noted that the auxiliary branch is intended to fine-tune the main branch and extract the detailed features of the ultrasound image through multi-scale convolution and shape boundary adaptive perception modules. The steps include:

[0079] The input image F is downsampled three times through convolution modules with kernel sizes of 3×3, 5×5, and 7×7 and a step size of 2 to obtain the feature map and This multi-scale convolution can gradually extract detail information and local features of ultrasound images.

[0080] Considering that the targets in ultrasound images have different sizes and fuzzy boundaries, the present invention designs a shape edge adaptive perception module (SEAPM). By using the shape perception module, the boundary perception module and the adaptive feature enhancement module to further extract features, it can effectively capture the shape information and boundary information of targets of different scales. Three detail feature maps are obtained by this method: and Its expression is:

[0081]

[0082] In the formula, Represents i detail feature maps; Seapm i (·) represents the i-th shape boundary adaptive perception module; Conv i () denotes convolutional modules with kernel sizes of 3×3, 5×5, and 7×7, respectively.

[0083] The maximum pooling layer, convolution layer, normalization layer and GELU activation function are used to map the detail feature map. and Downsample to make it equal to Consistent, these feature maps are spliced ​​in the channel dimension and dimensionally transformed to obtain the local detail features F extracted by the auxiliary branch c , which is expressed as:

[0084]

[0085] In the formula, F c represents local detail features; Down() represents the downsampling layer; Θ represents channel splicing.

[0086] An interactive fusion module is introduced to organically integrate the extracted content into the backbone encoder. In order to enhance the network's attention to the shape and boundary information of the target area, the present invention introduces a shape boundary adaptive perception module based on feature maps of different sizes. Figure 3 shown.

[0087] Figure 3 This is the framework diagram of the dual perception module. The dual perception module strengthens the network's attention to the shape and boundary details of the target area. When inputting feature maps of different sizes, the shape perception module will retain the shape details through adaptive weight adjustment. At the same time, the edge perception module can effectively capture and enhance edge information.

[0088] Among them, the shape boundary adaptive perception module is used to capture the shape and boundary information in the initial feature map, and the detailed feature map obtained includes:

[0089] Extract the shape information weight from the initial feature map, multiply the shape information weight with the initial feature map, and then perform channel concatenation to obtain a shape enhanced feature map;

[0090] The filter is used to sharpen the edge features of the ultrasonic image segmentation target to obtain an edge feature map, and the edge feature map is integrated and channel-joined with the initial feature map to obtain an edge enhancement feature map;

[0091] The shape enhancement feature map and the edge enhancement feature map are channel-wise concatenated, and the final enhanced feature map is generated by combining the broadcast mechanism.

[0092] Among them, the shape enhancement feature map and the edge enhancement feature map are channel-joined, and the final enhancement feature map is generated by combining the broadcast mechanism, including:

[0093] The shape enhancement feature map and the edge enhancement feature map are channel-joined to obtain a fused feature map;

[0094] The global average pooling layer is used to perform feature abstraction and dimension lifting and lowering on the fused feature map in turn to obtain an enhanced attention map.

[0095] Calculate the channel dimension weight, multiply the enhanced attention map by the channel dimension weight, and obtain the feature map after channel dimension enhancement;

[0096] Get the average value of the position corresponding to the feature map after channel dimension enhancement in each channel dimension, and calculate the weight of the spatial dimension;

[0097] Combined with the broadcast mechanism, the spatial dimension weighted fusion feature maps are added together to obtain the final enhanced feature map.

[0098] It should be noted that the broadcast mechanism is used to perform element-level operations between tensors of different shapes. The principle is to automatically expand the smaller tensor to match the larger tensor in a specific dimension, thereby achieving operations such as addition and multiplication between the two tensors.

[0099] It should be noted that the Shape Edge Adaptive Perception Module consists of three parts: shape perception, edge perception and adaptive feature enhancement.

[0100] Among them, the shape perception module uses a set of 3×3 convolution modules to halve the number of channels while keeping the size of the feature map unchanged, thereby reducing the number of parameters and the amount of calculation;

[0101] The feature map is downsampled using an adaptive max pooling layer with a stride of 4 to capture salient information in the feature map;

[0102] The feature map is restored to its original size through the bilinear interpolation function, and the weight of the shape information is extracted using the Sigmoid function Will With the original feature map Multiply to emphasize shape information;

[0103] The feature map is combined with the input feature map through a 3×3 convolution module Perform channel splicing to form the final output of the shape perception module Its expression is:

[0104]

[0105] In the formula, represents a product operation; Conv() represents a 3×3 convolution operation; Θ represents a channel splicing operation. This design not only retains the details of the shape information, but also improves the network's sensitivity to the target shape features through adaptive weight adjustment, which helps to improve the accuracy of ultrasound image segmentation.

[0106] It should be noted that the edge perception module is parallel to the shape perception module and is responsible for enhancing the edge information in ultrasound image target segmentation using traditional operators, which specifically includes:

[0107] Use a 3×3 convolution module to perform preliminary extraction of feature maps;

[0108] Use four 3×3 Sobel filters in specific directions to sharpen the edge features of the segmented target and integrate these edge feature maps into a unified feature map;

[0109] The result is passed to the 3×3 convolution module and Channel splicing is performed to form the final edge enhancement feature map, which is expressed as:

[0110]

[0111] In the formula, Φ(·) represents the convolution operation, S k denote the horizontal, vertical and two diagonal edge filters respectively, specifically [1, 2, 1], [0, 0, 0], [-1, -2, -1], [-1, 0, 1], [-2, 0, 2], [-1, 0, 1], [0, 1, 2], [-1, 0, 1], [-2, -1, 0] and [2, 1, 0], [1, 0, -1], [0, -1, -2]; Θ denotes the channel splicing operation.

[0112] By adding the convolution results in each direction pixel by pixel, a unified feature map is generated to achieve directional invariance. The edge perception module designed in this way can not only effectively capture and enhance edge information, but also organically integrate with the original features to improve the accuracy and robustness of ultrasound image segmentation.

[0113] It should be noted that the input of the adaptive feature enhancement module is the feature map F me and F ms The feature map F concatenated in the channel dimension se , which specifically include:

[0114] Through the global average pooling layer, F se Processing is performed to obtain a feature map of size B×C, where B is the batch size and C is the number of channels;

[0115] The feature map is reduced and increased in dimension through one-dimensional convolution to highlight the salient attention map, and the weight of the channel dimension is calculated through the Sigmoid function. The input feature is multiplied by the weight to obtain the feature map enhanced by the channel dimension.

[0116] The enhanced feature map is averaged to obtain the average value of the corresponding position in each channel dimension, and then the weight of the spatial dimension is calculated by the ReLU function;

[0117] Use the broadcast mechanism to add the weights of the spatial dimension to the input feature map to obtain the final enhanced feature map F m , whose expression is:

[0118] F m =Reshape(ReLU(Mean(Conv(S(GAP(F se )))×F se )))+F se

[0119] In the formula, Reshape() represents dimension transformation; ReLU() represents activation function; Mean() represents the mean operation of the corresponding position in each channel dimension; GAP() represents average pooling operation.

[0120] By adding an adaptive enhancement module after the parallel shape perception module and boundary perception module, the model can automatically select the most appropriate feature fusion method to enhance feature expression according to the currently processed data.

[0121] The local detail features are integrated into the backbone network for fine-tuning, and a resonance mechanism is introduced between the backbone network and the auxiliary network to achieve collaborative optimization between feature maps.

[0122] Among them, the local detail features are integrated into the backbone network for fine-tuning, and a resonance mechanism is introduced between the backbone network and the auxiliary network to achieve collaborative optimization between feature maps, including:

[0123] The global context features and local detail features of the image are multiplied by the weights of the backbone network and the auxiliary network respectively to achieve feature resonance between the backbone network and the auxiliary network.

[0124] It should be noted that the Feature Resonance Module aims to achieve collaborative optimization and enhancement of features by introducing a resonance mechanism between the backbone network and the auxiliary network, which specifically includes:

[0125] Extract feature maps F from the backbone network and auxiliary network respectively v and F c , and reduce the number of parameters through downsampling operations. The downsampling process includes a maximum pooling layer, a 3×3 convolution layer, layer normalization, and a GELU activation function, thereby effectively compressing the feature map;

[0126] The Sigmoid function is used to extract the weights of the main branch and the auxiliary branch respectively, and the two weights are added together. The main branch features and the auxiliary branch features are multiplied by the calculated weights respectively to achieve the resonance fusion of the features and obtain the final resonance result. The residual of the resonance result and the auxiliary branch features are added to retain the original feature information.

[0127] The processed results are passed to the decoder and encoder parts. Through this process, the feature resonance module can effectively coordinate the extraction and enhancement of features between the main branch and the auxiliary branch, further improving the feature extraction capability and overall performance of the network.

[0128] The local detail features extracted by the auxiliary network are integrated into the multi-head attention module of the backbone network for fine-tuning to generate multi-scale perceptual encoder feature embedding.

[0129] Among them, the local detail features extracted by the auxiliary network are integrated into the multi-head attention module of the backbone network for fine-tuning to generate multi-scale perceptual encoder feature embedding including:

[0130] The key vector and value vector are calculated by combining the local detail features, and the query vector is calculated based on the backbone features generated after window attention and normalization of the local detail features;

[0131] Based on the key vector, value vector and query vector, the fusion feature is calculated through the attention mechanism;

[0132] The backbone features and fusion features are residually connected and added to the multi-head attention module of the backbone network to generate the final fusion features;

[0133] The final fused features are added to the global context features to obtain the multi-scale perceptual encoder feature embedding.

[0134] It should be noted that the Interactive Fusion Module can organically combine data from different feature domains, so the present invention uses it to extract the local detail features F c Integrate into the backbone network for fine-tuning. Since multiple ViT modules are stacked in the backbone network, in order to reduce the burden of model training and keep the backbone network parameters as stable as possible, the present invention only performs feature fusion on one-third of the modules, which specifically includes:

[0135] F c Through the matrix E k and E v The key vector K and value vector V are calculated respectively, and the backbone feature F after window attention and normalization is l j Through the matrix E q Calculate the query vector Q;

[0136] The fusion features are calculated through the attention mechanism In order to increase the convergence ability of the model, F l j and fusion features Perform residual connection and pass it into the multi-head attention module of the backbone network. The expression is:

[0137]

[0138] Where S represents the Sigmoid function; Represents the dimension of force.

[0139] The final fusion feature The final output F of the main network v Add them together to get the output F of MA-Encoder ma .

[0140] Through the collaborative work of the main branch and the auxiliary branch, MA-Encoder can not only capture the multi-scale features and detail information in ultrasound images, but also use the shape boundary adaptive perception module to enhance the richness and accuracy of feature representation. The encoder designed in this way not only improves the overall performance of ultrasound image segmentation, but also provides more comprehensive and detailed feature input for the subsequent DE-Decoder, enabling the model to better cope with complex segmentation tasks in ultrasound images.

[0141] S2. Use the prompt encoder to map the diluted prompts to obtain image prompt feature embedding.

[0142] It should be noted that the image prompt feature embedding obtained by mapping the dilution prompt using the prompt encoder includes:

[0143] 1. Prompt the encoder input:

[0144] The hint encoder receives dilute hints, such as key points, bounding boxes, or masks. These hints are input to the encoder in the form of tensors, which contain the 2D coordinates of the hint boxes and key points.

[0145] 2. Mapping process:

[0146] Key point hint mapping:

[0147] ⑴Point prompt processing:

[0148] The cue encoder receives keypoint cues as input in the form of a 2D tensor representing the coordinates of points on the image. To ensure that the point cue accurately represents the center of the pixel, the coordinates of the point cue are first offset, that is, the coordinates of each point are offset by 0.5, so that the point coordinates are moved to the center of the pixel.

[0149] ⑵Position embedding mapping:

[0150] The encoder embeds the coordinates of the point cue using the position embedding layer. The embedding layer maps the point cue coordinates to high-dimensional features by generating a random spatial frequency matrix. Specifically, the position embedding layer first normalizes the input coordinates, that is, scales them to the range [0, 1]. Then, the coordinates are linearly transformed through a randomly generated Gaussian matrix, the cosine and sine values ​​of each coordinate point are calculated, and these values ​​are concatenated into the final embedding vector.

[0151] ⑶ Click on the prompt to embed adjustment:

[0152] For different types of point prompts, the system performs embedding adjustments based on the label value. Specifically, if the label value of the point prompt is 0 (negative sample), the encoder will add the embedding weight of the negative sample point to the embedding vector of the point prompt; if the label value of the point prompt is 1 (positive sample), the encoder will add the embedding weight of the positive sample point to the embedding vector of the point prompt.

[0153] (4) Click on the embedded output:

[0154] Through the above processing, the cue encoder generates a high-dimensional feature embedding P of the point cue p , with shape (B, N, e_dim), where B represents the batch size, N represents the number of point cues, and e_dim represents the dimension of the embedding vector.

[0155] Box prompt mapping:

[0156] ⑴ Processing of input bounding box prompt:

[0157] The hint encoder receives the input bounding box hint, represented as a coordinate tensor indicating the coordinates of the two diagonal points of the bounding box. To improve the accuracy of the bounding box hint, the encoder first offsets the coordinates of the bounding box by adding 0.5 to the coordinate value so that the coordinates of the two opposite corners of the bounding box (the upper left corner and the lower right corner) accurately represent the pixel center position.

[0158] ⑵ Extraction of diagonal point coordinates:

[0159] The encoder converts the coordinate tensor of the bounding box hint into a tensor of shape (B, 2, 2), where B represents the batch size, 2 represents the two corner points of the bounding box, and the subsequent two 2s represent the two-dimensional coordinates of the upper left corner and lower right corner of the bounding box.

[0160] ⑶Diagonal point embedding mapping:

[0161] The encoder also encodes the coordinates of the diagonal points of the bounding box using the position embedding layer mentioned above. Based on the size of the input image, the position embedding layer transforms the two-dimensional coordinates of the diagonal points through a linear transformation with a Gaussian matrix to generate a high-dimensional embedding vector related to the spatial position.

[0162] ⑷Diagonal point embedding adjustment:

[0163] The encoder adjusts its embedding vectors according to the two diagonal points of the bounding box. The embedding vector of the top left corner is weighted accordingly, and the embedding vector of the bottom right corner is also weighted accordingly.

[0164] ⑸ Output of bounding box hint embedding:

[0165] The system finally generates a high-dimensional feature embedding tensor P of the bounding box hint b , the shape is (B, 2, e_dim), where B represents the batch size, 2 represents the relative corner points of the bounding box, and e_dim represents the dimension of the embedding vector.

[0166] 3. Output of prompt encoder mapping:

[0167] By concatenating the embedding vector P of the key points p And the embedding vector P of the prompt box b Generates sparse embeddings of the model, with shape (B, N+2, e_dim), where B represents the batch size, N represents the number of prompts, and e_dim represents the embedding dimension.

[0168] S3. Multi-scale perceptual encoder feature embedding and image prompt feature embedding are multi-level fused, and feature enhancement is performed on the multi-level fusion result to obtain the ultrasound image segmentation result.

[0169] Among them, the multi-scale perceptual encoder feature embedding and the image prompt feature embedding are multi-level fused, and the multi-level fusion results are feature enhanced to obtain the ultrasound image segmentation results including:

[0170] Input the multi-scale perceptual encoder feature embedding and the image cue feature embedding into the cross attention to obtain the mask embedding;

[0171] The predefined convolutional module is used to perform feature dimension conversion and weight extraction on multi-scale perceptual encoder feature embedding, image cue feature embedding and mask embedding in turn;

[0172] The background attention map is calculated based on the weight extraction result, and the background attention map is element-wise multiplied with the multi-scale perceptual encoder feature embedding and then input into the spatial attention module for feature enhancement;

[0173] The local detail features and image hint features are embedded for channel splicing and input into the convolution module for feature fusion, and the ultrasound image is segmented in combination with the texture information of the ultrasound image;

[0174] The mask embedding and local detail features are channel-concatenated and input into the upsampling layer to restore the feature map size and obtain the ultrasound image segmentation result.

[0175] It should be noted that the degradation of SAM performance is related to the coupling effect between bad prompts and mask segmentation. During the prompt generation process, the presence of bad prompts (such as points outside the mask or bounding boxes that are significantly larger than the mask) can mislead the mask generation. In addition, accurate prompts (such as high-quality point prompts and a large number of point prompts) can significantly improve the performance of SAM. However, SAM is very sensitive to the randomness of prompts, which can lead to serious performance degradation. Therefore, the present invention designs a DE-Decoder that uses multi-level fusion of image and prompt embedding to accurately predict the segmentation results, thereby reducing the dependence of the segmentation results on prompts.

[0176] The first half of DE-Decoder follows the original design of SAM and embeds the features of MA-Encoder into F ma and the hint embedding F of the hint encoder p Pass it into cross-attention to get the mask embedding F k The second half of DE-Decoder mainly combines the segmentation results and F k Decoupled, it receives features from three parts, namely feature embedding F ma , the mask embedding Fk And the intermediate feature F of DE-Decoder c , which specifically include:

[0177] Embed the features into F ma , mask embedding F k and the intermediate feature F c A 3×3 convolution module is used to unify the feature dimensions to 64×64×64; where F k With clear segmentation boundary features, F ma It contains the global context information of the main branch and the local details of the auxiliary branch. c Contains shallow texture information of the multi-scale network.

[0178] In order to fully explore F k To provide segmentation details and alleviate the impact of bad prompts on the results, the present invention uses the sigmoid function to extract weights, and then calculates the background attention map through it and compares it with F ma After element-wise multiplication, it is passed into the spatial attention module for further feature enhancement.

[0179] For F c Features, and compare it with F p Channel splicing is performed and passed into the 3×3 convolution module for feature fusion, guiding the network to segment the ultrasound image with the help of texture information.

[0180] The two features are concatenated in the channel dimension and passed to the final upsampling layer to restore the feature map size to obtain the final segmentation result F o , whose expression is:

[0181] F o =Up(Conv(Sat((1-S(F k )⊙F ma ))+Conv(F k ΘF c )))

[0182] In the formula, F o represents the ultrasound image segmentation result; Up represents the upsampling layer; Conv represents the convolution module; Sat represents the spatial attention module; S represents the Sigmoid function; ⊙ represents the dot product; Θ represents the channel splicing; F k represents mask embedding; F ma represents the multi-scale perceptual encoder feature embedding; F c Represents local detail features.

[0183] In summary, the DE-Decoder designed in the present invention can maximize the global context information and local detail information extracted by the encoder by multi-level fusion of mask embedding features and MA-Encoder features, while also alleviating the dependence of the segmentation results on adverse cues.

[0184] The multi-scale perceptual decoupling ultrasound image segmentation method provided by the present invention is further described below in conjunction with specific embodiments:

[0185] 1. Dataset:

[0186] In order to verify the segmentation effect of the model of the present invention in ultrasound images, 41,100 images and 79,564 masks were collected from 10 challenging public data sets to construct the US40K data set. These data come from the following data sets: UDIAT, BUSI, BUSIS, DDTI, TN3K, TNSCUI, XMUSet, TG3K, CAMUS and HMCQU. The segmentation targets contained in these data sets include six categories: breast lesions, thyroid nodules, thyroid, left ventricle (LV), myocardium (MYO) and left atrium (LA). In order to verify the performance of the model proposed in the present invention, the data set is divided into a training set and a test set in a ratio of 4:1. Due to the small amount of data in the UDIAT data set, it is only used as a test set in US40K. The specific data distribution is shown in Table 1.

[0187] Table 1 Introduction to US40K dataset

[0188] Dataset Number Mask Train Test Segmentation target UDIAT 163 163 - 163 Breast lesion BUSI 647 647 517 130 Breast lesion BUSIS 562 562 449 113 Breast lesion DDTI 637 637 509 128 Thyroid nodule TN3K 3493 3493 2794 699 Thyroid nodule TNSCUI 3644 3644 2915 729 Thyroid nodule XMUSet 6788 6788 5430 1358 Thyroid nodule TG3K 3585 3585 2868 717 Thyroid gland CAMUS 19232 57696 15385 11541 LV, MYO, LA HMC-QU 2349 2349 1879 470 MYO US40K 41100 79564 32746 8354 Above 6 categories

[0189] 2. Verification settings and evaluation methods:

[0190] To ensure the consistency of data processing, the present invention uniformly adjusts all input images to 256×256 pixels. In the model training stage, the initial weights of the model are derived from the pre-trained SAM-B weights, where the parameters of the SAM encoder remain frozen and only the parameters of the prompt encoder and decoder are trained. The Adam optimizer was selected in the optimization process, with an initial learning rate of 0.0001, and the learning rate is changed to 0.1 times the original every 10 epochs, and the entire training process lasts for 30 epochs.

[0191] The present invention adopts the Dice loss function to better measure the performance of the model in the segmentation task. The Dice coefficient (%) and the 95th percentile Hausdorff distance (HD95) are used as evaluation indicators to comprehensively evaluate the segmentation quality and boundary accuracy of the model.

[0192] 3. Comparative experiment:

[0193] 3.1 Quantitative Experiments

[0194] The main models compared in this invention are divided into two categories, one is the specific task model, U-Net and DeepLabV3+ based on CNN, Trans-UNet based on ViT, and CTO, a hybrid model based on CNN and ViT. One is the large model based on SAM, including SAM, Fine-Tuning SAM (FT-SAM), SAM-Med2D, SAMUS, MSA, SAMed, and MedSAM. In order to intuitively feel the difference between the large model and the specific task model, the prompt type P6 mode, which is more widely used in practical applications, was selected for result comparison. The results are shown in Table 2, and the optimal value is indicated in bold.

[0195] Table 2 Quantitative comparison of MAD-SAM and other methods on seven different datasets

[0196]

[0197] In Table 2, the following three phenomena can be observed:

[0198] First, due to the differences in data collection sample standards or GT delineation standards in different data centers, even for the same segmentation target, the same model has different segmentation effects in different data sets. For example, the TN3K data set and the XMUSet data set, although they are both segmenting thyroid nodules, the models based on specific tasks such as U-Net, DeepLabV3+ and Trans-UNet have large deviations in the results produced on the two data sets. The difference between U-Net and DeepLabV3+ on the two data sets reached 17.31%, the difference between DeepLabV3+ was 21.18%, and the difference between Trans-UNet was 19.71%. However, it is worth noting that for the interactive SAM-based model, the Dice difference between the two data sets on FT-SAM, MedSAM and SAM-Med2D does not exceed 1%, which highlights the adaptability of the interactive model to cross-datasets. What is more commendable is that the model proposed in the present invention not only has the best segmentation effect on the two data sets, but also has a difference of only 0.34%, showing excellent generalization and adaptability.

[0199] Second, even under the same data center and outlining requirements, the segmentation difficulty is different for different segmentation targets, resulting in different segmentation effects produced by the same model. For example, in the CAMUS dataset, there are significant differences in the segmentation results of LA and LV, and the results show that LA segmentation is easier. In the specific task model, U-Net achieved a Dice of 93.17% in the segmentation of LA, while on the LV segmentation target of the same dataset, the Dice was only 88.70%, and other specific task models also showed a similar trend. However, the SAM-based model has a smaller difference in this regard, which once again proves the applicability of the interactive model to ultrasound image segmentation. Compared with other SAM models, the average Dice improvement rates of the MAD-SAM of the present invention on CAMUS-LV and CAMUS-LA were 2.44% and 3.00%, respectively, showing excellent segmentation performance.

[0200] Third, on the US40K dataset, the specific task model performs poorly due to the significant size differences between different segmentation targets and the domain differences between different datasets. The SAM-based model benefits from the hint information and significantly outperforms the specific task model. Specifically, the average Dice value of the specific task model is only 53.91%, and the average HD95 value is 64.64. In contrast, the average Dice of the SAM-based model is 90.07%, which is 36.16% higher than the specific task model, which highlights the superiority of the base model with hint information in large datasets. In summary, whether it is for different datasets with the same segmentation target, different segmentation targets in the same dataset, or large datasets containing various segmentation targets, the MAD-SAM of the present invention always shows the best segmentation performance and applicability. This is attributed to the powerful feature extraction capability of the multi-scale perception module in the encoder, which can effectively capture the shape and boundary features of different segmentation targets. In addition, DE-Decoder also optimizes the impact of hint information on the segmentation results, thereby improving the segmentation performance and robustness of the model.

[0201] 3.2 Qualitative Experiments

[0202] In order to further verify the versatility of MAD-SAM in ultrasound image segmentation, the present invention randomly selected several samples for visual analysis. Figure 3The figure shows the excellent performance of MAD-SAM in dealing with small, medium and large segmentation objects. For small objects (first and second rows), due to the anatomical complexity of breast and thyroid tissues in ultrasound images, DeepLabV3+ and Trans-UNet have large segmentation errors, resulting in segmentation results that are significantly larger than GT. Although SAM-based models generally perform better with accurate cues, FT-SAM has obvious over-segmentation, while SAM-Med2D shows some under-segmentation. In contrast, MAD-SAM as well as SAMUS and SAMed have more accurate segmentation results.

[0203] For medium-sized objects (fourth row), when there are multiple segmentation targets, the specific task model can correctly locate and segment the target region. However, the SAM-based model is limited by the cue information and can only segment the specified target, while MAD-SAM can correctly segment all targets in this case. This is thanks to DE-Eecoder, which effectively reduces the dependence on cue information, thereby improving robustness and segmentation accuracy.

[0204] Finally, for large objects (fifth and sixth rows), U-Net, DeepLabV3+, and Trans-UNet show obvious over-segmentation in thyroid nodule segmentation and a certain degree of under-segmentation in thyroid organ segmentation. FT-SAM tends to predict most of the area within the prompt box as thyroid organs, while SAM-Med2D and SAMUS mistakenly predict other tissues as thyroid tissue. In contrast, only MAD-SAM can accurately segment objects and performs well in segmenting ultrasound image objects of different sizes. This ability is mainly attributed to MA-Encoder, which captures multi-scale features and boundary details through auxiliary branches and fine-tunes the main branch. This improves the feature extraction ability of the encoder, enabling MAD-SAM to accurately segment objects with large size variations.

[0205] In general, in different data domains, segmentation targets and different sample differences, specific task models often show problems of under-segmentation and over-segmentation, and the boundary and shape segmentation of the target is not accurate enough. The basic model based on SAM integrates the prompt information, which can effectively help the network locate and segment the target. However, poor prompt information directly affects the segmentation performance of these models. It is worth noting that the DE-Decoder designed by the present invention effectively alleviates these problems and achieves accurate segmentation results. Figure 4 As shown in the figure, the red border represents the GT, the light blue box represents the box prompt information, the green dot represents the positive point prompt information, and the light yellow mask represents the prediction result.

[0206] 4. Ablation experiment:

[0207] Taking SAM as the baseline network, this paper conducts ablation analysis on each key module of MAD-SAM and systematically tests the impact of adding pre-trained weights, MA-Encoder and DE-Decoder. As shown in Table 3, in all 6 datasets, the proposed modules significantly improve the performance of the baseline network, verifying the effectiveness of MAD-SAM in ultrasound image segmentation.

[0208] Table 3 Ablation experiments of MAD-SAM on 6 datasets

[0209]

[0210] First, loading the pre-trained weights significantly improves the segmentation performance of the baseline network. On TG3K, CAMUS-LA, and US40K datasets, Dice improves by 13.01%, 13.29%, and 17.29%, respectively. This substantial improvement is mainly attributed to the weights learned by SAM on very large datasets, which can be used for simple segmentation of ultrasound images through transfer learning.

[0211] Secondly, adding MA-Encoder also brings significant performance improvements. Specifically, on the US40K dataset, Dice is improved by 4.12% and HD95 is reduced from 6.484 to 3.776. On other datasets, the average Dice is improved by 1.98%. These results show that MA-Encoder greatly improves the segmentation performance on large-scale datasets. This improvement may be due to the fact that the segmentation targets in the US40K dataset vary in size, allowing MA-Encoder to capture multi-scale perceptual features through its auxiliary branches, thereby optimizing the main branches, which makes the feature extraction more comprehensive and detailed, thereby significantly improving the segmentation performance.

[0212] Finally, the introduction of DE-Decoder further improves the network performance. On the BUSI dataset, HD95 decreased by 0.80, while on the TG3K dataset, Dice increased by 0.47%. This improvement is mainly due to the fact that DE-Decoder can integrate input features from different dimensions through multi-level fusion, thereby reducing the model's dependence on prompt information and enhancing the model's robustness and segmentation performance.

[0213] 5. Generalization experiment:

[0214] In order to evaluate the impact of the US40K dataset constructed by the present invention on the generalization performance of the model, the present invention uses the model trained on US40K to test a specific dataset, and the results are shown in Table 4.

[0215] Table 4 Experimental results of model generalization using US40K dataset loading weights

[0216]

[0217] SAM's large model showed improved performance on most datasets. Specifically, after training on US40K, the Dice of U-Net, DeepLabv3+, Trans-UNet, and CTO decreased by 3.82%, 5.0%, 6.93%, and 0.19% respectively when tested on the BUSI dataset; when tested on the TN3K dataset, the Dice decreased by an average of 1.73%; only on the XMUSet dataset, the Dice of U-Net and DeepLabV3+ increased by 0.34% and 0.09% respectively. In contrast, the average Dice index of the large model based on SAM on the BUSI and TN3K datasets increased by 0.54% and 0.11% respectively, and only on the XMUSet dataset, there was a certain degree of performance degradation.

[0218] The reason why the proprietary model trained on a large-scale dataset has poor generalization performance on the proprietary dataset, while the large model based on SAM has better generalization performance, may be that the US40K dataset contains a variety of segmentation targets, and the correlation between different targets is low, and the proprietary model is difficult to handle segmentation tasks of multiple categories. The large model based on SAM can better locate the target position and complete the segmentation because it integrates the prompt information. This conclusion can also be seen from the data in Table 2. In summary, the introduction of the US40K dataset has a significant effect on improving the generalization performance of the large model based on SAM. At the same time, it is worth affirming that after the model is trained on US40K, the best effect is obtained when it is tested on various specific datasets, indicating the wide applicability of the model to ultrasound image segmentation.

[0219] 6. Impact of different prompts on performance:

[0220] In order to analyze the impact of different prompt information on the performance of MAD-SAM, the present invention conducted experiments on multiple data sets using six different prompt types. The experimental results are summarized in Table 5.

[0221] Table 5 MAD-SAM performance experiments using different prompt information

[0222]

[0223] As the number of cue points increases, the segmentation performance of MAD-SAM on most datasets shows an upward trend. Specifically, compared with P1 cue information, applying P4 cue information to the model can improve Dice by 5.89% on the BUSI dataset, 5.27% on the TN3K dataset, and 13.11% on the XMUSet dataset. However, when using P1 and P2 cue information, the changes are not significant. Only the Dice of the TN3K dataset increased slightly by 0.69%, while the Dice of the BUSI dataset decreased by 0.95% and the Dice of the XMUSet dataset decreased by 1.05%. The reason why there is no obvious difference between P1 and P2 cues may be that positive and negative points are input to the cue encoder at the same time during testing, instead of generating negative point information based on the difference area between the prediction map and the GT value.

[0224] Compared with point hints, box hints have considerable advantages. With P5 hint information, Dice increases by 7.44%, 5.58%, 14.19% and 1.56% on average on the four datasets compared with P1, P2, P3 and P4 hint information. In addition, HD95 is optimized by 11.89, 3.99, 10.11 and 1.59 on average. Combining box information and point information (P6) produces better results compared with using box information and point information alone. Specifically, when adopting a positive point and a box hint, Dice achieves 91.42%, 88.11%, 87.76% and 95.37% on the four datasets, respectively. These findings highlight the effectiveness of utilizing comprehensive hint information in ultrasound image segmentation, especially combining box and point hints to improve the segmentation performance of MAD-SAM.

[0225] It should be noted that, similar to SAM and other interactive segmentation methods, when training the model of the present invention, the simulation method of interactive segmentation is borrowed. In order to further explore the sensitivity of the model to the prompt information, the present invention carefully designs 6 types of prompts: one positive point (P1), one positive point and one negative point (P2), 2 positive points and 2 negative points (P3), 5 positive points and 5 negative points (P4), one box (P5), one box and one positive point (P6). It is worth noting that although SAM can receive different types of prompt information at the same time, in order to explore the impact of prompt information on the segmentation performance of large models, the present invention uses only one type of prompt information for each data set for training and testing. Therefore, 6 different model weights are trained for each data set.

[0226] For point hints, the present invention first randomly selects a positive point from the ground truth (GT) mask, then flattens the GT into a one-dimensional vector, and uses a uniform sampling method to obtain other positive points. The present invention uses the background as a randomly sampled area to obtain a certain number of negative points according to the same principle. For box hints, the present invention defines the bounding box as the maximum circumscribed rectangle of the GT, while allowing each coordinate to have a maximum offset of five pixels to increase the naturalness and diversity of the hint.

[0227] During the training process of the point prompt model, the present invention simulates a real human-computer interaction scenario. Each round of training includes multiple iterations. In the first iteration, the present invention prompts according to the preset number of points. Subsequently, the present invention performs up to 8 random iterations. Starting from the second iteration, the present invention randomly selects 1, 4 or 8 points from the error area between the previous mask prediction and the true value as subsequent sparse prompts, allowing the user to perform interactive segmentation of single points or multiple points. Such operations not only simulate the process of actually outlining the target in the clinical environment, but also enhance the model's adaptability to uncertainty and complexity prompts.

[0228] In summary, with the help of the above-mentioned technical scheme of the present invention, the present invention can enhance the feature learning ability of the encoder for the target through a multi-scale perceptual encoder, and embed a shape boundary adaptive perception module in the encoder, which can more accurately capture the shape and edge information of the target; at the same time, in order to reduce the coupling between the segmentation result and the prompt information, the decoupling enhancement decoder is used to enhance the features through cross-attention and multi-level fusion features from the multi-scale perceptual encoder, thereby reducing the proportion of prompt information in the feature learning process and reducing the sensitivity of the network to bad prompt information. Finally, the fused features are input into the spatial attention module for feature enhancement, thereby obtaining the final segmentation result, thereby improving the segmentation quality of the ultrasound image.

[0229] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-scale perceptual decoupling ultrasound image segmentation method, characterized in that: The multi-scale perceptual decoupling ultrasound image segmentation method includes: Inputting the collected ultrasound image into a multi-scale perceptual encoder, and using the multi-scale perceptual encoder to generate a multi-scale perceptual encoder feature embedding in the ultrasound image; Using the cue encoder to map the diluted cues, we get the image cue feature embedding; The multi-scale perceptual encoder feature embedding and the image prompt feature embedding are fused at multiple levels, and the multi-level fusion results are feature enhanced to obtain the ultrasound image segmentation result.

2. The multi-scale perceptual decoupling ultrasound image segmentation method according to claim 1, characterized in that: The step of inputting the acquired ultrasound image into the multi-scale perceptual encoder and using the multi-scale perceptual encoder to generate a multi-scale perceptual encoder feature embedding in the ultrasound image comprises: The backbone network in the multi-scale perceptual encoder is used to perform preliminary encoding processing on the ultrasound image to obtain the global context features of the image; The auxiliary network in the multi-scale perceptual encoder is used to extract detail features in the ultrasound image, and the detail features are sequentially spliced ​​and dimensionally transformed to obtain local detail features of the ultrasound image. The local detail features are integrated into the backbone network for fine-tuning, and a resonance mechanism is introduced between the backbone network and the auxiliary network to achieve collaborative optimization between feature maps.

3. The multi-scale perceptual decoupling ultrasound image segmentation method according to claim 2, characterized in that: The global context features of the image obtained by performing preliminary encoding processing on the ultrasound image using the backbone network in the multi-scale perceptual encoder include: Train the SAM encoder on a predefined dataset and use the SAM encoder weights to initialize the backbone network of the multi-scale perceptual encoder; Use the predefined convolutional layer of the backbone network to perform preliminary encoding processing on the ultrasound image; The initially encoded ultrasound image passes through several visual transformers including windowed attention and residual connections to obtain the global context features of the image.

4. The multi-scale perceptual decoupling ultrasound image segmentation method according to claim 3, characterized in that: The method of extracting detail features from the ultrasound image using the auxiliary network in the multi-scale perceptual encoder and sequentially performing splicing and dimension conversion processing on the detail features to obtain local detail features of the ultrasound image includes: Down-sampling the ultrasound image using a predetermined convolution module to obtain a number of initial feature maps; The initial feature map is used as the input of the shape boundary adaptive perception module, and the shape boundary adaptive perception module is used to capture the shape and boundary information in the initial feature map to obtain a detail feature map; The detail feature map is sequentially subjected to channel splicing and dimension conversion to obtain the local detail features of the ultrasound image.

5. The multi-scale perceptual decoupling ultrasound image segmentation method according to claim 4, characterized in that: The use of the shape boundary adaptive perception module to capture the shape and boundary information in the initial feature map to obtain the detail feature map includes: Extract the shape information weight from the initial feature map, multiply the shape information weight with the initial feature map, and then perform channel concatenation to obtain a shape enhanced feature map; The filter is used to sharpen the edge features of the ultrasonic image segmentation target to obtain an edge feature map, and the edge feature map is integrated and channel-joined with the initial feature map to obtain an edge enhancement feature map; The shape enhancement feature map and the edge enhancement feature map are channel-wise concatenated, and the final enhanced feature map is generated by combining the broadcast mechanism.

6. The multi-scale perceptual decoupling ultrasound image segmentation method according to claim 5, characterized in that: The channel splicing of the shape enhancement feature map and the edge enhancement feature map and combining the broadcast mechanism to generate the final enhancement feature map includes: The shape enhancement feature map and the edge enhancement feature map are channel-joined to obtain a fused feature map; The global average pooling layer is used to perform feature abstraction and dimension lifting and lowering on the fused feature map in turn to obtain an enhanced attention map. Calculate the channel dimension weight, multiply the enhanced attention map by the channel dimension weight, and obtain the feature map after channel dimension enhancement; Get the average value of the position corresponding to the feature map after channel dimension enhancement in each channel dimension, and calculate the weight of the spatial dimension; Combined with the broadcast mechanism, the spatial dimension weighted fusion feature maps are added together to obtain the final enhanced feature map.

7. The multi-scale perceptual decoupling ultrasound image segmentation method according to claim 6, characterized in that: The method of integrating local detail features into the backbone network for fine-tuning and introducing a resonance mechanism between the backbone network and the auxiliary network to achieve collaborative optimization between feature maps includes: The global context features and local detail features of the image are multiplied by the weights of the backbone network and the auxiliary network respectively to achieve feature resonance between the backbone network and the auxiliary network. The local detail features extracted by the auxiliary network are integrated into the multi-head attention module of the backbone network for fine-tuning to generate multi-scale perceptual encoder feature embedding.

8. The multi-scale perceptual decoupling ultrasound image segmentation method according to claim 7, characterized in that: The method of integrating the local detail features extracted by the auxiliary network into the multi-head attention module of the backbone network for fine-tuning to generate a multi-scale perceptual encoder feature embedding includes: The key vector and value vector are calculated by combining the local detail features, and the query vector is calculated based on the backbone features generated after window attention and normalization of the local detail features; Based on the key vector, value vector and query vector, the fusion feature is calculated through the attention mechanism; The backbone features and fusion features are residually connected and added to the multi-head attention module of the backbone network to generate the final fusion features; The final fused features are added to the global context features to obtain the multi-scale perceptual encoder feature embedding.

9. The multi-scale perceptual decoupling ultrasound image segmentation method according to claim 8, characterized in that: The multi-scale perceptual encoder feature embedding and the image prompt feature embedding are multi-level fused, and the multi-level fusion result is feature enhanced to obtain the ultrasound image segmentation result, including: Input the multi-scale perceptual encoder feature embedding and the image cue feature embedding into the cross attention to obtain the mask embedding; The predefined convolutional module is used to perform feature dimension conversion and weight extraction on multi-scale perceptual encoder feature embedding, image cue feature embedding and mask embedding in turn; The background attention map is calculated based on the weight extraction result, and the background attention map is element-wise multiplied with the multi-scale perceptual encoder feature embedding and then input into the spatial attention module for feature enhancement; The local detail features and image hint features are embedded for channel splicing and input into the convolution module for feature fusion, and the ultrasound image is segmented in combination with the texture information of the ultrasound image; The mask embedding and local detail features are channel-concatenated and input into the upsampling layer to restore the feature map size and obtain the ultrasound image segmentation result.

10. The multi-scale perceptual decoupling ultrasound image segmentation method according to claim 9, characterized in that: The expression of the ultrasonic image segmentation result is: F o =Up(Conv(Sat((1-S(F k )⊙F ma ))+Conv(F k ΘF c ))) In the formula, F o represents the ultrasound image segmentation result; Up represents the upsampling layer; Conv represents the convolution module; Sat represents the spatial attention module; S represents the Sigmoid function; ⊙ represents the dot product; Θ represents the channel splicing; F k represents mask embedding; F ma represents the image encoder features; F c Represents local detail features.

Images