Ultrasonic medical image segmentation method based on CNN and SAM fusion

Abstract

The invention provides a CNN and SAM fusion-based ultrasonic medical image segmentation method CASAM, and aims to construct a CNN and SAM ViT double-branch collaborative framework for the problems of low contrast, strong speckle noise, fuzzy anatomical boundary, complex lesion form, difficulty in small target segmentation and the like of an ultrasonic image. A dual-channel feature fusion module is arranged in the CNN branch to extract fine-grained local texture and edge information; local features are injected into ViT branches through a cross-branch low-rank adapter, and the characterization capability of small-scale details and key boundaries is enhanced under low parameter overhead; a medical cue adapter is introduced into a ViT branch, semantic and local clues are fused through image embedding, lesion boundary expression is enhanced in combination with edge information, and therefore lesion positioning and segmentation precision is improved. Experimental results show that higher precision and stronger generalization are achieved in multiple types of ultrasonic segmentation tasks, GPU video memory and computing overhead are reduced, and the method is suitable for resource-limited clinical terminal deployment.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to an ultrasound medical image segmentation method based on the fusion of CNN and SAM, applied to ultrasound image segmentation tasks of the thyroid and thymus. Background Technology

[0002] Medical image segmentation is a core task of computer-aided diagnostics (CAD), aiming to accurately extract organs, tissues, and lesion regions from medical images to support disease detection, treatment planning, and prognostic assessment. With the rapid development of deep learning, especially convolutional neural network (CNN) technology, significant breakthroughs have been achieved in medical image segmentation. However, despite the high accuracy achieved by existing methods in specific tasks, practical applications still face challenges such as data scarcity, high annotation costs, and insufficient model generalization ability.

[0003] Segment Anything Model (SAM), as a pre-trained general image segmentation framework, possesses strong generalization and transferability capabilities, generating segmentation results based on cues such as points, bounding boxes, and text, and is widely used in computer vision. However, the performance of SAM in medical image segmentation remains limited, primarily because medical images typically exhibit low contrast, blurred boundaries, and complex structures, making it difficult to effectively transfer the general features of SAM. Furthermore, medical image segmentation models often require training on domain-specific data, and the lack of labeled medical datasets makes it challenging to construct medical segmentation models with good generalization ability.

[0004] To improve the adaptability of SAM in medical image segmentation, researchers have proposed several fine-tuning strategies, such as MedSAM, MSA, and SAMed. While these methods improve computational efficiency, the use of 16×16 image patch tokenization sacrifices detailed features, leading to decreased segmentation performance when dealing with complex geometries and small-scale lesions. Furthermore, these methods typically require high input resolution, resulting in excessive computational complexity and limiting inference speed.

[0005] To address these issues, researchers proposed SAMUS, aiming to reduce computational complexity while maintaining the generality of SAM. However, SAMUS's cross-branch attention mechanism increases the computational burden, and its adoption of a traditional CNN architecture fails to effectively capture fine-grained features in ultrasound images.

[0006] To address the limitations of SAMUS in medical image segmentation, this paper proposes a novel ultrasound medical image segmentation method—CASAM—based on the fusion of CNN and SAM. CASAM introduces a medical cue adapter into the ViT image encoder, combining a dual-channel feature fusion module with a cross-branch low-rank adapter to enhance the model's ability to segment local details and complex structures. Through these innovative designs, CASAM successfully improves the accuracy and efficiency of SAM in medical image segmentation, providing a more efficient and accurate solution for medical image segmentation. Summary of the Invention

[0007] Purpose of the invention: The present invention aims to provide an ultrasound medical image segmentation method based on the fusion of CNN and SAM, so as to effectively improve the accuracy and efficiency of medical image segmentation.

[0008] Technical solution: A method for segmenting ultrasound medical images based on the fusion of CNN and SAM, comprising the following steps:

[0009] Step 1: Introduce a medical prompt adapter in the ViT image encoder. The Medical PromptAdapter is located between the Patch Embedding module and the WindowTransformer module in the ViT image encoder. It takes the Patch embedding feature output by the backbone network and the gradient prompt information Grad extracted from the medical image as input. Through a cascaded multilayer perceptron and nonlinear mapping structure, it achieves explicit modeling and injection of medical prior prompt information while keeping the overall network parameter quantity and computational quantity controllable.

[0010] Step 2: The medical image is processed by introducing a parallel CNN branch and a dual-channel feature fusion module to fuse high-frequency details and global semantic information;

[0011] Step 3: The output of the medical image processed by ViT's Norm & Add in Step 1 and the output of Step 2 are efficiently fused with local and global information through the cross-branch low-rank adapter CT-SLoRA Adapter. After being decoded by the Mask Decoder, the final segmentation mask is predicted, which is the lesion region in the medical image.

[0012] Based on the above scheme, as a preferred option, in step 1, the input medical image is processed by the Patch Embedding module to divide the image into multiple small patches, thereby extracting local features of the image. The local features are then injected with spatial location information through the Position Embedding module to ensure that the model can understand the spatial distribution of image features and maintain the structural information of the image. After the Adapter module enhances and reconstructs the features, they are fused with the output of Med PromptAdapter.

[0013] Based on the above scheme, as a preferred option, a Prompt Encoder module is introduced to convert the point prompts from the user into structured information that the model can understand.

[0014] Based on the above scheme, as a preferred embodiment, after the input medical image is processed by the patch embedding module of the backbone encoder, the original image is divided into N non-overlapping image patches, and a patch-level feature sequence is obtained through linear projection. During medical image preprocessing, a gradient cue information is constructed and converted into patch-level cue features. In the feature fusion stage, the Med Prompt Adapter aligns the patch embedding features and gradient cue features according to the patch index and then concatenates or linearly combines them to obtain a joint input representation. The joint input representation is then input into an adjustable multilayer perceptron subnetwork. In the process, intermediate cue features of stage i are obtained. A GELU nonlinear activation function is applied to these intermediate cue features. After nonlinear mapping, the features are passed through an upprojected multilayer perceptron. Dimensional restoration and adaptation are performed, and incremental features are output. The incremental features are then combined with the original patch using a residual method to obtain patch features adapted to medical prompts.

[0015] Based on the above scheme, as a preferred approach, each patch is linearly projected onto the token dimension D to form a patch token sequence. Position Embedding injects spatial location information into the tokens, and the output of Position Embedding forms a Position Feature, which is then fed into a trainable Adapter module to obtain incremental features. These features are then injected into the main token stream through residual addition. The output of Patch Embedding forms a Patch Feature, which also enters the trainable Adapter module. A cue vector is generated through trainable mapping and added to the token sequence. Window Transformer × 2 performs two windowing Transformer calculations on the input sequence, performs normalization, and then inputs it into the Multi-head Attention module to obtain the output MHA(X). A trainable Adapter is inserted after Multi-head Attention for efficient parameter fine-tuning. The output of the Adapter is normalized and residually connected, and then fed into the Multi-head Attention module again to obtain the second attention output MHA(X). Simultaneously, CT-SLoRA... The Adapter is used to fuse the token features from the ViT branch with the convolutional features output from step 2 to obtain the fused features. Finally, the fused features are added element-wise to the second MHA(X) and the output of the Adapter branch to obtain the output features of this stage.

[0016] Based on the above scheme, as a preferred option, in step 2, the medical image undergoes dual-channel feature fusion. The dual-channel fusion module consists of a high-channel path and a low-channel path. A set of basic convolutional sub-modules is set on each branch, which performs a concatenated operation of 3×3 convolution, batch normalization, and GELU activation on the input features to achieve preliminary linear mapping and nonlinear transformation of the original features. Simultaneously, the numerical range and distribution of the two features are normalized. After the basic convolution, each branch is further concatenated with 6 sets of recursive convolution ReConv+GELU units to form a lightweight recursive residual convolution structure. Each ReConv unit uses a 3×3 convolution kernel, reducing the number of parameters while maintaining spatial resolution. Through the residual connection between the convolution output and the input features, and through 6 successive stacks of ReConv3×3+GELU, each branch can fully mine local contextual information, refining and enhancing texture details, structural contours, and target boundaries, thereby obtaining the enhanced features of the upper branch. and lower branch features The two enhanced features are respectively normalized by a 3×3 convolution and BatchNorm, and the two features are structurally aligned in terms of channel dimension and statistical distribution to obtain aligned features. and Subsequently, the aligned features from the two paths are input into the fusion unit, where element-wise addition operations are used to achieve consistent fusion of multi-scale semantics, resulting in unified features. ,

[0017] Adaptive recalibration of fused features: First, F fuse Apply GELU activation to obtain F a =GELU(F fuse Then, a linear transformation is performed on the channel dimension using a 1×1 convolution to obtain F. b =Conv1×1(F a ), then for F b Applying BatchNorm normalization, we obtain F. c =BN(F b Finally, the features are mapped to the (0,1) interval using the Sigmoid function, generating an attention weight map A=σ(F) of the same size as the fused features. c In the output stage, the attention weight map and the fused features are multiplied element-wise to obtain the final output feature O = A⊙F. fuse , where “⊙” represents element-wise multiplication.

[0018] In step 2, the final output features of the dual-channel fusion module are fused. Perform a max pooling operation to achieve initial downsampling.

[0019] Then, The input is fed into the initial convolutional submodule ConvModule to obtain... ,

[0020] Obtaining initial convolutional features Subsequently, the CNN module constructs a basic unit of "GHPA–MaxPool–GHPA–Conv" and repeats this unit four times to form a multi-level local feature extraction and multi-axis attention enhancement structure, outputting features. .

[0021] Based on the above scheme, as a preferred option, the CT-SLoRA Adapter adopts the "cross-branch gating + low-rank linear adaptation (LoRA)" approach: first, a spatially related gating weight is generated jointly by CNN and ViT features, and then the ViT features are incrementally modulated with low-rank parameters. This achieves the fusion of local and global information while keeping the backbone frozen or with a small amount of training, and reduces additional GPU memory and parameter overhead.

[0022] Based on the above scheme, as a preferred option, the CT-SLoRA Adapter performs element-wise fusion of CNN Features and ViT Features, and processes the fused features... Calculate two types of statistical descriptions, corresponding to the Max and Mean branches in the graph, respectively. The Max branch of max pooling is obtained. The mean of the average pooling branch is obtained By channel-dimensional pooling, we obtain: The two are spliced ​​together using channel dimensions, and the spliced ​​result is... Entering Conv7×7 in the figure, a gated response is generated. Then, the gated value is compressed to the [0,1] interval by Sigmoid to obtain the final cross-branch gated weight map. The fused features are then subjected to element-wise multiplication gate, and the gated increment is injected into the ViT feature stream as a residual. After completing the cross-branch gated injection, it enters the serial low-rank adaptation sub-layer, which is: LoRA Linear→GELU→LoRALinear.

[0023] Compared with the prior art, the present invention has the following significant advantages:

[0024] 1. The CASAM model proposed in this invention combines a medical cue adapter, a convolutional neural network (CNN) branch, and a ViT encoder, effectively improving the accuracy and efficiency of medical image segmentation. Specifically, CASAM captures semantic and local features through the medical cue adapter and enhances the detailed representation of lesion areas using edge information, greatly improving the model's segmentation accuracy for ultrasound images. Simultaneously, CASAM combines a convolutional neural network (CNN) with the ViT model, optimizing the fusion of local and global information, further enhancing the model's performance in complex medical image segmentation tasks.

[0025] 2. This invention innovatively designs a dual-channel feature fusion module, which adaptively fuses high-frequency details and global semantic information, significantly enhancing the model's ability to perceive complex structures and ambiguous boundary regions without significantly increasing computational burden. This module effectively enhances attention to lesion regions through a local-global attention mechanism, further improving segmentation accuracy.

[0026] 3. The cross-branch low-rank adapter proposed in this invention achieves efficient fusion of local and global information by optimizing the feature interaction between the CNN and ViT encoders. This design not only enhances the model's ability to represent detailed features but also reduces memory consumption to a certain extent, ensuring a balance between computational efficiency and memory usage.

[0027] 4. This invention enhances the image embedding and edge information capture capabilities in the ViT branch by introducing a medical cue adapter. Especially in ultrasound images, it improves the segmentation accuracy of lesion boundaries and details, enabling the model to accurately identify lesion areas even in complex backgrounds.

[0028] 5. On both the BUSI and TN3K medical image segmentation benchmarks, the proposed CASAM (Ours) achieves state-of-the-art results: On the BUSI dataset, the **Dice coefficient** is improved to 87.47%, and the **HD (Hausdorff distance)** is reduced to 24.01, representing an improvement of 1.59 percentage points and a reduction of 1.96 (HD decreased by 7.55%) compared to the current strongest baseline SAMUS (85.88%, 25.97); on the TN3K dataset, the Dice coefficient reaches 85.49%, and the HD (Hausdorff distance) is reduced to 27.43, representing an improvement of 1.04 percentage points and a reduction of 1.54 (HD decreased by 5.32%) compared to SAMUS (84.45%, 28.97). These results demonstrate that the proposed method possesses stronger segmentation accuracy and boundary delineation capabilities in scenarios with small targets and complex boundaries.

[0029] 6. Table 2 shows the ablation results for the BUSI dataset. The baseline (Group 1) Dice / HD was 85.88% / 25.97. Adding the DF-Module alone (Group 2) significantly improved the Dice to 87.01% and reduced the HD to 24.67; adding the Med-Prompt-Adapter (Group 3) or CT-SLoRA-Adapter (Group 4) alone also brought stable improvements. The performance was further enhanced after combining modules, with DF+CT-SLoRA (Group 6) showing the most significant improvement in Se and IoU (Se 90.08%, IoU 79.25%). The optimal results were achieved when all three modules were enabled simultaneously (Group 7): Acc 97.86%, Se 90.11%, Dice 87.47%, IoU 79.59%, and HD 24.01. These figures represent improvements of +1.16, +3.06, +1.59, and +2.69 respectively compared to the baseline, while HD decreased by 1.96, validating the effectiveness of module complementarity and synergy. Attached Figure Description

[0030] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the ultrasound medical image segmentation method based on CNN and SAM fusion according to an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the network structure of the medical prompt adapter module according to an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the network structure of the dual-channel feature fusion module according to an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram of the cross-branch low-rank adapter module network structure according to an embodiment of the present invention;

[0035] Figure 6 This is a comparison chart of the comparative experimental results of the embodiments of the present invention;

[0036] Figure 7 This is a bar chart comparing the evaluation indicators of the comparative experiments in this invention embodiment;

[0037] Figure 8 This is a radar chart comparing the evaluation indicators of the comparative experiments in this invention. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0039] As attached Figure 1-8 As shown, an ultrasound medical image segmentation method based on CNN and SAM fusion includes the following steps:

[0040] A medical prompt adapter is introduced into the ViT image encoder. Located between the Patch Embedding and WindowTransformer modules, the Med Prompt Adapter captures semantic information and local features of the image through the image embedding module and enhances the edge features of lesion areas using the edge information fusion module, thereby improving the model's segmentation accuracy in medical images. Using the patch embedding features output from the backbone network and the gradient prompt information (Grad) extracted from the medical image as input, a cascaded multilayer perceptron and nonlinear mapping structure are employed to explicitly model and inject medical prior prompt information while maintaining controllable overall network parameters and computational cost. This enhances the model's ability to perceive lesion areas and tissue boundaries.

[0041] The Adaptive Vision Transformer (ViT) branch framework is an efficient medical image segmentation method that combines several innovative modules to fully leverage the potential of ViT and enhances the model's expressiveness and detail perception capabilities through an adapter mechanism. First, the input medical image is processed by the Patch Embedding module, which segments the image into multiple patches to extract local features. These local features are then injected with spatial location information through the Position Embedding module, ensuring that the model understands the spatial distribution of image features and preserves the structural information of the image.

[0042] Building upon this foundation, the Med Prompt Adapter module further enhances the model's ability to identify lesion areas and its sensitivity to details by transforming prior medical knowledge into adaptive cues. This module explicitly models medical cues using cascaded multilayer perceptrons and nonlinear mappings, while ensuring that the network's computation and parameter count are kept within reasonable limits, thereby improving the model's performance and efficiency.

[0043] Next, the two Adapter modules enhance and reconstruct features at different stages of image encoding, further optimizing image representation. The Window Transformer module and the Multi-head Attention module work together to extract local contextual information and capture global information. Through two Window Transformer operations, the model can effectively fuse local and global information, achieving a balance between handling details and broad contextual information, and significantly improving segmentation accuracy.

[0044] Next, the CT-SLoRA Adapter module deeply fuses ViT features with CNN features to capture more detailed image features, further enhancing the model's ability to perceive image details, especially in the identification and segmentation of complex lesion regions. Through the synergistic effect of these modules, the overall architecture not only significantly improves accuracy in medical image segmentation tasks but also effectively reduces computational complexity, enabling stable operation in resource-constrained clinical environments.

[0045] Next, after further decoding by the Mask Decoder, the model can predict the final segmentation mask, which is the lesion region in the medical image. The Mask Decoder module performs image segmentation based on features from the ViT branch, generating an accurate segmented image that shows the boundaries and features of the lesion region.

[0046] Furthermore, to further improve the model's accuracy and adaptability, the system introduces a Prompt Encoder module, specifically designed to process point prompts from the user. The Prompt Encoder module transforms these point prompts into structured information that the model can understand, further enhancing its ability to locate and identify target regions. By combining point prompt information, the model can better understand the specific location and shape of lesions, thereby improving segmentation accuracy.

[0047] Overall, the above process, through step-by-step feature extraction, fusion, and decoding, ultimately generates an accurate segmentation mask. This process not only fully utilizes ViT's global perception capabilities but also incorporates adapter and prompting mechanisms, making the model more adaptable and accurate in medical image segmentation tasks, especially performing exceptionally well when dealing with complex or small lesion regions.

[0048] Furthermore, in step 1, the GELU activation function in the medical prompt adapter is used for nonlinear transformations of each layer to ensure the smoothness and stability of the feature mapping. Specifically, assume the input medical image is:

[0049] After processing by the patch embedding layer of the backbone encoder, the original image is divided into N non-overlapping image patches, and the patch-level feature sequence is obtained through linear projection:

[0050]

[0051] in, This represents the Patch Embedding mapping, where D is the embedding feature dimension. The Patch embedding branch can be viewed as performing a low-level semantic encoding on the original grayscale medical image. Its advantages are: on the one hand, it converts the two-dimensional pixel grid into a sequence form, which is convenient for subsequent Transformer structure modeling; on the other hand, it compresses the data dimension and reduces the computational burden of the backbone network while ensuring that spatial structural information is not completely lost.

[0052] Meanwhile, in this embodiment, a gradient cue map is constructed during image preprocessing, denoted as ;

[0053]

[0054] This gradient map can be transformed into patch-level cue features using spatial gradient operators such as Sobel and Laplacian, as well as edge detection operators, through similar patch partitioning and linear mapping operations.

[0055]

[0056] in, For gradient hint mapping function, Gradient cueing provides feature dimensions. The role of the gradient cueing branch is to explicitly encode structural information such as lesion boundaries, tissue interfaces, and areas of drastic gray-scale changes, providing medically significant auxiliary clues for subsequent cueing and adaptation networks.

[0057] During the feature fusion stage, the Med Prompt Adapter concatenates or linearly combines the patch embedding feature P and the gradient cue feature Q according to the patch index to obtain the joint input representation:

[0058]

[0059] in This indicates a concatenation operation based on feature dimensions. For dimension-aligned linear transformations, This is the joint representation dimension. At each patch location, this joint representation simultaneously includes the semantic content of the original image and the gradient cue information for that location, enabling the establishment of a correspondence between "texture / intensity information – structure / boundary information" at the patch level.

[0060] like Figure 3 As shown in the upper part, the joint representation of Z input to an adjustable multilayer sensor subnetwork In the process, the intermediate cue features of stage i are obtained:

[0061]

[0062] in, It can be a two- or multi-layer fully connected network with hidden layers and linear transformation parameters. This network can learn independently for different encoding stages i, thereby achieving hierarchical prompt adjustment. Through this "stage-adjustable" design, the adaptation module can emphasize fine-grained boundary information for shallow features and focus on global lesion morphology and contextual information for deep features, making the prompt effect more refined and controllable.

[0063] Subsequently, the intermediate prompt features were analyzed. Apply the GELU nonlinear activation function:

[0064]

[0065] Compared to traditional ReLU, GELU activation is smoother in output and gradient, which is beneficial for learning more refined distinction boundaries in scenarios such as medical images with coherent gray-level distribution and subtle intensity differences, reducing gradient oscillations, and improving the expressive power of cue features and training stability.

[0066] After the nonlinear mapping, the cue features are passed through an upprojected multilayer perceptron. Perform dimensional restoration and adaptation:

[0067]

[0068] in, Projecting the intermediate cue features back to the same dimension D as the original patch embedding, the output is a set of "cue increment" features with the same shape as P. The advantage of this upprojection structure is that it does not change the original feature dimensions of the backbone network, but injects medical cues into the backbone features in an additive manner, thereby ensuring compatibility with the ViT and CNN hybrid architecture.

[0069] Finally, this embodiment can use a residual method to combine the incremental features of the prompt with the original patch embedding to obtain the patch features adapted for medical prompts:

[0070]

[0071] The aforementioned residual fusion enables the Med Prompt Adapter to selectively enhance features related to lesions and organ boundaries without disrupting the original semantic expression of the backbone network. For regions that do not require modification, the prompt increment can automatically approach zero, thereby achieving "fine-tuning" prompt control over the backbone network. Figure 3 The lower half illustrates this process with a diagram: the top shows the original ultrasound image and its segmented color patch features, and the bottom shows the corresponding gradient map and its gray patch hint features; after being fused, nonlinearly mapped and up-projected in the Med Prompt Adapter, the two are combined to obtain a set of enhanced patch features that simultaneously contain the original texture and gradient structure information, which are then reconstructed by the subsequent network into a clearer lesion salient area.

[0072] As attached Figure 2 As shown, this embodiment provides an Adaptive VisionTransformer Branch, which is used to perform efficient transfer learning on target domain data through several lightweight adaptation modules while freezing the backbone parameters.

[0073] The input is a two-dimensional ultrasound medical image with a size of, for example, 256×256. Let the input image be... , where H=W=256.

[0074] The Patch Embedding module divides the input image into several non-overlapping or partially overlapping image patches. Let the patch size be P×P, then the number of patches is...

[0075]

[0076] Each patch is linearly projected onto the token dimension D, forming a patch token sequence:

[0077]

[0078] The Patch Embedding in the figure is marked with a snowflake, indicating that its weights are frozen to stabilize the low-level visual representation and reduce the number of training parameters.

[0079] Position Embedding is used to inject spatial location information into a token. Its output can be represented as:

[0080]

[0081] The Position Embedding in the diagram also features a snowflake icon, indicating that its parameters are frozen.

[0082] Before proceeding to the subsequent Transformer computation, this scheme sets lightweight adapters for location features and patch features respectively, and also introduces an additional Med Prompt Adapter to achieve stronger structural priors and semantic cues for the target domain.

[0083] The Position Embedding output forms a "Position Feature," which is fed into a trainable Adapter module to obtain incremental features.

[0084]

[0085] Then, it is injected into the main token stream through residual addition:

[0086]

[0087] The significance of this approach is that, even when the main position encoding is fixed, the deformation of the target domain and the differences in imaging geometry can still be compensated for by a small number of parameters.

[0088] The output of Patch Embedding forms a "Patch Feature", which is also fed into the trainable Adapter module.

[0089]

[0090] This adaptation branch is used to perform low-cost correction of target domain offsets such as texture distribution, noise model, and device differences when the patch embedding is frozen.

[0091] The Med Prompt Adapter at the bottom of the diagram originates from the Patch Embedding branch, indicating that it generates "hinting increments" based on the patch token and injects the mainstream before entering the Transformer.

[0092] Generate cue vectors from trainable mappings And add it to the token sequence:

[0093]

[0094] In the diagram, "Window Transformer×2" indicates that the input sequence is processed twice using a windowed Transformer. The core principle is to divide the token into multiple windows based on its spatial neighborhood, and then perform self-attention within each window to reduce computational complexity and enhance local structure modeling capabilities.

[0095] Suppose there are K windows after the window is divided, and each window contains For each token, the token subsequence of the k-th window is... implement:

[0096]

[0097] The two stacks can be represented as:

[0098]

[0099] Following the Window Transformer, the attached diagram shows a typical Transformer substructure flow from left to right, including:

[0100] In the diagram, "Norm&Add" represents normalizing the input and performing residual connections. Its typical form is:

[0101]

[0102] in This is a combined mapping for subsequent sub-layers.

[0103] The "Multi-head Attention" in the diagram is used to model long-range dependencies in the token sequence. Typically, for each attention head h:

[0104]

[0105] The output MHA(X) is obtained by splicing multiple heads together and linearly mapping them.

[0106] After the first Multi-head Attention, a trainable Adapter is inserted for efficient parameter fine-tuning.

[0107] The Adapter module consists of a dimensionality reduction layer (Down), a nonlinear GELU, and a dimensionality increase layer (Up), forming a residual connection with the input. Let the input features be... If we reduce the dimension to the bottleneck dimension d, then:

[0108]

[0109] in: For Down projection; For GELU nonlinearity; For Up projection; "+" indicates residual addition.

[0110] This structure guarantees that: even during training... , Even with a small number of parameters, an effective domain adaptation increment can be applied to the backbone representation while avoiding the destruction of the pre-trained representation.

[0111] Residual addition at the fusion point

[0112]

[0113]

[0114] This "+" also represents residual fusion: it adds the outputs of the two branches element-wise, requiring consistent dimensions. It is equivalent to adding the outputs of the main features... A feature from the Adapter is superimposed on top.

[0115]

[0116] Similar to the Norm & Add steps above, we again normalize the Adapter output and perform residual connections. This ensures stability during information transfer and avoids gradient vanishing or exploding issues during model training.

[0117] Figure 2 At the top is a Conv Module, which is typically used to extract local features from the input image, especially when processing image data. This module mainly processes the input image through convolution operations to extract spatially local features.

[0118] Assume the input features are Where H is the height, W is the width, and C is the number of channels, the convolution operation can be represented as:

[0119]

[0120] in, It is a convolution kernel; after the convolution operation, the feature map... This can be used for further processing. The convolution module here extracts low-level feature information from the input image.

[0121] The figure is marked with ×4, indicating that the Conv Module will be applied four times to gradually extract deeper features and enhance the expressive power of the features through multiple convolutional layers.

[0122] Next, the CT-SLoRA Adapter in the diagram is used for efficient fine-tuning of the model. The purpose of the cross-branch low-rank adapter is to enhance the model's performance on a specific task with minimal parameter adjustments without retraining the entire model. This adapter is typically inserted into a pre-trained model to adapt it to a new task.

[0123] The calculation formula for this module is as follows:

[0124]

[0125] Input features, shape is . : Dimensionally reduced matrix, shape is . : An upgraded matrix with the following shape . : Activation function, typically GELU. : Combination operation of dimensionality reduction, activation, and dimensionality increase. +: Residual connection, which combines input features with adapter output.

[0126] First, input features After Norm & Add operations. These Norm & Add operations include LayerNormalization and residual joins. Assume the input is... The output is The Norm&Add operation can then be represented as:

[0127]

[0128] Then, the normalized output It is passed to the CT-SLoRA Adapter. This adapter will then receive the input. Adjustments are made through a low-rank adaptation mechanism. The adapter calculation formula is as follows:

[0129]

[0130] Meanwhile, another branch is the Conv Module, which is used to extract local features from the input image. Assume the input is... The output of the convolution operation is :

[0131]

[0132] Convolution results This will be passed directly to the CT-SLoRA Adapter and merged with the result from the Norm&Add output. As before, the adapter's output will be:

[0133]

[0134] Then, after the above processing, the input features are further processed by Multi-head Attention to capture long-range dependencies. Assume that after the preceding processing, the resulting input is... The formula for calculating Multi-head Attention is:

[0135]

[0136] The outputs of multiple heads are concatenated and linearly transformed to obtain the final multi-head attention output:

[0137]

[0138] Finally, the core part of the diagram involves fusing the output of the CT-SLoRA Adapter with the output of the Multi-head Attention. This fusion can be achieved through addition or concatenation operations. Let's assume the output from the CT-SLoRA Adapter is... The output obtained from Multi-head Attention is Then the fusion operation can be expressed as:

[0139]

[0140] Ultimately, the features after fusion This will be used for subsequent task processing.

[0141] In the previous steps, the model has undergone processing by multiple modules, ultimately outputting the fused features. This feature fusion combines the outputs from the CT-SLoRA Adapter and Multi-head Attention. This fused feature will then be fed into the Mask Decoder to generate the prediction mask.

[0142] Mask Decoders are primarily used to generate masks for a task based on fused features, and are commonly used in image segmentation tasks. In image segmentation tasks, the mask is typically a binary image that marks the target region and the background region.

[0143] The pseudo-formula for Mask Decoder can be expressed as:

[0144]

[0145] in The fused features, after passing through the Mask Decoder, output as a prediction mask. The mask represents the location of the target region in the input image, and is usually a binary matrix of the same size as the input image.

[0146] Next, the model's input includes a Point Prompt, which is typically a user-provided point or location information related to the target region in the image. The Point Prompt helps the model locate the target region more accurately.

[0147] The point prompt is passed to the Prompt Encoder, which converts the point prompt information into a feature representation suitable for model processing. Through the Prompt Encoder, the point prompt is encoded into a feature vector, helping the Mask Decoder better understand and use the location information.

[0148] The pseudo-formula for Prompt Encoder can be expressed as:

[0149]

[0150] in The input is a dotted hint, which the Prompt Encoder transforms into the same features as the Mask Decoder. This will be used in conjunction with other features from the Mask Decoder to generate an accurate mask.

[0151] The Mask Decoder and Prompt Encoder work together, utilizing information extracted from the Point Prompt to help generate the target mask. Specifically, the Prompt Encoder provides information related to the target region in the input image, while the Mask Decoder uses this information, along with other context from the fused features, to generate the final predicted mask.

[0152] Therefore, the final mask output can be expressed as:

[0153]

[0154] here, It is a feature resulting from the fusion of all the preceding modules. It is a dot hint message encoded by the Prompt Encoder.

[0155] Furthermore, in this embodiment, a dual-channel feature fusion module is proposed to perform depth enhancement and adaptive fusion on feature maps from different semantic levels and different numbers of channels, so as to improve the medical image segmentation network's ability to represent lesion regions.

[0156] Overall, the input ultrasound medical image enters this module simultaneously via two branches: the upper branch receives high-level semantic features with 128 channels, and the lower branch receives low-level detail features with 64 channels. First, a set of basic convolutional submodules is set up on each branch, performing a concatenated operation of 3×3 convolution, batch normalization, and GELU activation on the input features. This achieves preliminary linear mapping and nonlinear transformation of the original features, while simultaneously normalizing the numerical range and distribution of the two feature paths, laying a stable foundation for subsequent recursive convolution enhancement. After the basic convolution, each branch further concatenates 6 sets of recursive convolution (ReConv) + GELU units to form a lightweight recursive residual convolution structure. Each ReConv unit uses a 3×3 convolution kernel, reducing the number of parameters while maintaining spatial resolution. The residual connection between the convolution output and the input features effectively alleviates the vanishing gradient problem and stabilizes network training. Through six successive stackings of ReConv3×3+GELU, each branch can fully exploit local contextual information, refining and enhancing texture details, structural contours, and target boundaries, thereby obtaining the enhanced upper branch features. (128 channels) and lower branch features (64 channels). After recursive enhancement, the two enhanced features are respectively normalized by a 3×3 convolution and BatchNorm, i.e., Conv3×3→BN is performed on the upper branch and Conv3×3→BN is performed on the lower branch, in order to further extract local information. The two features are then structurally aligned in terms of channel dimension and statistical distribution to obtain aligned features. and Subsequently, the aligned features from the two paths are input into the fusion unit, where element-wise addition operations are used to achieve consistent fusion of multi-scale semantics, resulting in unified features. The fused feature can be represented as a multi-channel 3D tensor, which simultaneously encodes high-level semantic information and low-level boundary details. Based on this, the module constructs a compact attention generation branch to adaptively recalibrate the fused feature: firstly, it... Apply GELU activation to obtain To introduce richer nonlinear expressions; then, a linear transformation is performed in the channel dimension using 1×1 convolution, resulting in... This step is equivalent to performing a full channel connectivity operation at each spatial location, which is beneficial for explicitly modeling the correlation between different channels; subsequently, for Applying BatchNorm normalization, we get To stabilize the numerical range and improve training convergence, the Sigmoid function is used to map the features to the (0,1) interval, generating an attention weight map of the same size as the fused features. In the output phase, the attention weight map and the fused features are multiplied element-wise to obtain the final output features of the module. ,in" "" indicates element-wise multiplication. In this way, the module can significantly enhance important channels and key spatial locations, and effectively suppress irrelevant regions and background noise. Thus, without significantly increasing the overall number of parameters and computational complexity, it can significantly improve the discriminative characterization and segmentation accuracy of target tissues and lesion regions in medical image segmentation tasks.

[0157] Dual-branch recursive convolution enhancement

[0158] By utilizing the multi-layer stacking of ReConv3×3+GELU, feature refinement is performed on both the high-channel and low-channel branches.

[0159] Without significantly increasing parameters, enhancing the ability to express detailed information such as local texture, tissue structure and lesion boundaries is beneficial for the fine depiction of small-volume lesions and thin structures in complex medical scenarios.

[0160] Structured fusion of multi-scale features

[0161] By using two 3×3Conv+BN sub-modules, the coding features at different scales are aligned in terms of channel dimension and statistical distribution.

[0162] By adopting an element-by-element fusion approach, high-level semantic information and low-level detailed information are modeled in a unified manner to achieve a collaborative expression of the overall morphology and local boundary information of organs.

[0163] Attention-guided adaptive recalibration

[0164] An attention weight map is generated using GELU+1×1Conv+BN+Sigmoid to selectively emphasize the fused features in terms of spatial location and channel dimension.

[0165] It can adaptively enhance key responses related to the target organ or lesion region (such as lesion internal texture, organ outline, lesion-normal tissue boundary, etc.) and suppress background tissue and imaging noise, thereby improving the robustness and accuracy of medical image segmentation.

[0166] like Figure 2 As shown, in the specific process of the CNN branch, the output of the aforementioned dual-channel feature fusion module is denoted as...

[0167]

[0168] This feature map Information from different scales and channel branches has been integrated, and through attention recalibration, a relatively rich and discriminative representation of the lesion region and its boundary details has been formed. Based on this, a local feature extraction module based on a convolutional neural network is proposed to... Further multi-level convolutional modeling and multi-axis attention enhancement are performed to obtain high-quality feature representations suitable for fine segmentation of medical images.

[0169] First, regarding fusion features Perform a max pooling operation to achieve initial downsampling:

[0170]

[0171] in, Pooling kernel size (e.g.) ), Step size (e.g.) This step achieves two objectives: firstly, each downsampled feature point corresponds to a larger receptive field in the original image, which is beneficial for the network to capture the complete shape of the organ and the overall distribution of lesions; secondly, the reduction in spatial resolution effectively compresses redundant information, significantly reducing the computational load of subsequent GHPA modules and convolution operations, making it easier to deploy in high-resolution medical image scenarios. Furthermore, max pooling retains the "strongest response value" within the local window, highlighting and enhancing areas with high lesion intensity and large boundary gradients, thus completing a significant response screening at an early stage.

[0172] Then, Inputting into the initial convolutional submodule (ConvModule) yields:

[0173]

[0174] The ConvModule consists of a cascaded 3×3 convolution, a normalization layer, and a GELU activation function. The 3×3 convolution models local texture, edge, and shape information at a new scale, demonstrating excellent capture capabilities for details such as blood vessel course, organ contours, and lesion boundaries. Normalization effectively unifies the feature distribution across different cases and imaging parameters, mitigating the impact of data distribution differences on training stability. The GELU activation function, with its smoother gradient and output distribution, better expresses the numerous low-contrast lesion regions with subtle grayscale variations in medical images, thereby enhancing the network's nonlinear modeling capabilities and sensitivity to small lesions.

[0175] Obtaining initial convolutional features Subsequently, a basic unit of "GHPA–MaxPool–GHPA–Conv" is constructed within the module, and this unit is repeated four times to form a multi-level local feature extraction and multi-axis attention enhancement structure. Let the input of the t-th basic unit be... The output is , where t=1,2,3,4.

[0176] For the first GHPA (fine-grained multiaxial attention enhancement), the calculation process can be expressed as follows:

[0177]

[0178] Let the input of the GHPA module be The output is Internally, it can be divided into three steps: First, precoding convolution is performed.

[0179]

[0180] While maintaining spatial resolution, local contextual information is extracted through 3×3 convolution, and normalization and GELU activation are used to stabilize numerical distributions and enhance nonlinear representation; secondly, in Constructing multi-axis attention weights

[0181]

[0182] This function explicitly models correlations across multiple dimensions, including the horizontal axis, vertical axis, and channel dimension, enabling it to capture elongated structures extending along a specific direction and complementary information between different channels. Finally, it obtains the desired results through attention recalibration and residual connections.

[0183]

[0184] in This represents element-wise multiplication. The residual form can be viewed as a representation of the original characteristic... On top of this, a gain term modulated by attention is superimposed. This approach avoids damaging the original representation and helps alleviate gradient vanishing, ensuring stable and easily convergent training. Overall, multi-axis attention weights... By assigning higher weights to spatial locations associated with lesions and lower weights to other background areas, the discriminative power of the features is significantly improved.

[0185] Following the first GHPA module, to further expand the receptive field and construct multi-scale feature representations, the output is... Perform max pooling again:

[0186]

[0187] By inserting a downsampling operation within each basic unit, the network progressively expands its receptive field in the depth direction, enabling subsequent deep features to perceive a wider range of contextual information. This is particularly important for characterizing the overall morphology of organs and the relationship between tumors and distant tissues. Simultaneously, the downsampled features provide a "coarse-scale" input to the second GHPA module, making subsequent attention modeling more biased towards global and cross-regional dependencies, thus forming a two-level attention structure combining "fine-scale" and "coarse-scale" approaches.

[0188] The second GHPA module is used for multi-axis attention enhancement on coarse-scale features, and its calculation process is as follows:

[0189]

[0190] The internal text can also be written as

[0191]

[0192] in and These represent the multi-axis attention weights and pre-encoded features calculated on the downsampled features, respectively. Performing GHPA at a lower spatial resolution is equivalent to using fewer feature points to cover a larger area of ​​the original image, thus more effectively modeling long-range dependencies between organs and between lesions and surrounding distal tissues. In conjunction with the local details focused on by the previous GHPA, this coarse-scale GHPA focuses on global semantic consistency and overall structural constraints. The cascading of these two approaches can simultaneously consider lesion details and overall morphology, which is particularly crucial for medical image segmentation tasks.

[0193] After completing two-stage GHPA and intermediate downsampling, The input is fed into the Convolutional Integration Submodule (ConvModule) to obtain the output of this basic unit:

[0194]

[0195] The ConvModule here also adopts a 3×3 convolution + normalization + GELU structure. On the one hand, the post-convolution can locally smooth and reorganize the attention-weighted features generated by GHPA, suppressing noise amplification caused by overly sharp attention weights; on the other hand, this convolutional module can reorganize and compress the channel dimension, making the number of channels output at each level match the interface requirements of subsequent decoders or other network modules, and further fusing information between different channels. Furthermore, since the GHPA module is wrapped within the ConvModule before and after, the overall structure still maintains the form of a typical CNN backbone, facilitating seamless embedding into existing U-Net-like or encoder-decoder-based medical segmentation networks, exhibiting good compatibility and portability. The aforementioned "GHPA–MaxPool–GHPA–Conv" basic unit... This process is repeated four times consecutively along the depth direction, i.e., at t=1, 2, 3, and 4, respectively, yielding the results. , , , Through these four progressively layered processing steps, each layer performs multi-axis attention modeling and convolutional integration at a coarser scale, thereby achieving a hierarchical feature extraction process from "local details → mesoscale structure → global semantics". Compared to using only a single-layer attention mechanism, this multi-level structure better meets the practical needs of medical images, which require attention to both minute lesions and overall organ morphology, and significantly improves feature representation capabilities while keeping computational costs under control.

[0196] Finally, after one initial MaxPool and ConvModule cycle and four levels of GHPA–MaxPool–GHPA–Conv basic units, the module output is recorded as:

[0197]

[0198] This output feature It also possesses multi-scale local texture features, multi-axis global contextual information, and a stable numerical distribution after multiple convolutions and normalizations.

[0199] As attached Figure 5 As shown, the CT-SLoRA Adapter is used to achieve efficient fusion of global features of the ViT branch and local detail features of the CNN branch. The module adopts a "cross-branch gating + low-rank linear adaptation (LoRA)" approach: first, the CNN and ViT features jointly generate a spatially correlated gating weight, and then the ViT features are incrementally modulated with low-rank parameters. This achieves the fusion of local and global information while keeping the backbone frozen or with minimal training, and reduces additional GPU memory and parameter overhead.

[0200] The inputs on the left side of the module include:

[0201] CNN Feature: denoted as It is used to provide fine-grained information such as local texture, edges, and organ boundaries;

[0202] ViT Feature: denoted as It is used to provide information such as global dependencies and long-range semantic relationships.

[0203] In one implementation, the channel counts of both are aligned before entering the fusion node to satisfy the same-dimensional constraint:

[0204]

[0205] Then on the left side of the image, " Perform element-by-element merging at the node:

[0206]

[0207] in This represents element-wise addition. To enable the ViT branch to absorb local details of the CNN branch "by position and by content," the module performs feature fusion. Calculate two types of statistical descriptions, corresponding to the Max and Mean branches in the graph, respectively:

[0208] Max pooling branch (Max): This yields...

[0209] Average pooling branch (Mean): This yields...

[0210] Through channel pooling, we obtain:

[0211] Then, at node "C" in the diagram, the two are concatenated at the channel dimension:

[0212]

[0213] spliced Entering Conv7×7 in the diagram, neighborhood structural information is aggregated through a larger receptive field to generate a gated response:

[0214]

[0215] The gating values ​​are then compressed to the [0,1] interval using a Sigmoid function, resulting in the final cross-branch gating weight graph (attention graph):

[0216]

[0217] The gating weight A is jointly determined by CNN and ViT, thus reflecting the "context-triggered" characteristic: when a CNN branch gives a strong local response at a certain position, the gating will increase the fusion strength at that position; otherwise, it will suppress noise or unreliable details.

[0218] The diagram shows Sigmoid followed by " A node in the form of "" indicates that the gating weights are used to modulate the features and merged with the main branch residuals. A preferred implementation is element-wise multiplication gating of the fused features:

[0219]

[0220] in This is element-wise multiplication.

[0221] The gated increment is then injected into the ViT feature stream as a residual:

[0222]

[0223] Through the above design, the ViT branch can effectively absorb the local details of the CNN branch while maintaining the original global semantic structure. In addition, since the gating is a sparse / restricted [0,1] weight map, the regions that actually participate in the enhancement are automatically selected, thereby reducing invalid computation and memory usage to a certain extent.

[0224] After completing the cross-branch gated injection, a low-rank adaptor sublayer is further connected on the right side of the figure. Its structure is as follows: LoRA Linear (low-rank linear) → GELU → LoRA Linear (low-rank linear).

[0225] This sublayer is used to perform operations with very few parameters. Perform trainable incremental mapping. Preferably, LoRA restricts the weight increments of the linear transformation to a low-rank decomposition form:

[0226]

[0227] This transforms large matrix training into training with two small matrices, significantly reducing the number of parameters and GPU memory.

[0228] Will Consider it as a vector input expanded by position. Then the low-rank adaptation can be represented as:

[0229]

[0230] in These correspond to two LoRA Linear layers, with GELU being a non-linear activation. The output is then restored to the shape of the feature map to obtain the following:

[0231]

[0232] Through the aforementioned combination of "cross-branch gating + low-rank adaptation," the ViT branch can selectively absorb local detail features provided by the CNN branch while maintaining the advantages of global modeling, achieving efficient fusion of local and global information. Since the gating weights A limit the enhancement to key regions, and LoRA uses low-rank decomposition to train incremental parameters, this module improves detail representation capabilities while reducing the scale of additional parameters and memory consumption during training, making it suitable for applications with small sample sizes, strong domain shifts, or limited computing power.

[0233] Experiments were conducted using four publicly available datasets. The TN3K dataset is a thyroid ultrasound image dataset designed specifically for ultrasound image segmentation tasks. The DDTI dataset is a publicly available thyroid ultrasound image dataset containing high-quality thyroid ultrasound images widely used in medical segmentation research. The BUSI dataset is a breast ultrasound image dataset containing ultrasound images of various sizes, ranging from 256×256 to 1072×768. The UDIAT dataset is designed specifically for breast mass detection and segmentation tasks and contains 163 precisely annotated breast ultrasound images. To evaluate the generalization ability of different models, the DDTI and UDIAT datasets were kept invisible during training and validation.

[0234] The comparison results of the ultrasound medical image segmentation method based on CNN and SAM fusion using the embodiments of the present invention with other models are shown in Table 1:

[0235] Table 1 Comparative Experimental Results

[0236]

[0237] We conducted systematic comparative experiments on two publicly available datasets, BUSI and TN3K, to evaluate the performance of various mainstream segmentation methods. Overall, U-Net performed moderately, achieving Dice of 78.56% and 79.41% on BUSI and TN3K, respectively, with relatively high HD values ​​of 33.21 and 33.55. Notably, TransUNet and H2Former also achieved good results on these two datasets, but still did not surpass our method. SAMUS achieved some improvement on BUSI, with Dice reaching 85.88% and HD decreasing to 25.97. In contrast, our proposed CASAM achieved state-of-the-art results on both datasets; on BUSI, Dice improved to 87.47% and HD further decreased to 24.01; on TN3K, Dice reached 85.49% and HD was 27.43. Considering these metrics, CASAM significantly outperformed the comparative methods on both benchmarks, demonstrating stronger segmentation accuracy and more stable boundary consistency. This demonstrates its excellent performance in medical image segmentation tasks, especially its advantages in detail capture and boundary accuracy.

[0238] To verify the effectiveness of the improved modules in this embodiment of the invention in improving model performance, six ablation experiments were conducted under the same parameter settings, focusing on the three innovative points. The results of the ablation experiments are shown in Table 2.

[0239] Table 2 Ablation Experiment Results

[0240]

[0241] Table 2 presents the ablation results on the BUSI dataset, where "√" indicates that the corresponding module was enabled. This paper sequentially introduces the dual-channel feature fusion module (DF-Module), the medical prompt adapter (Med-Prompt-Adapter), and the cross-branch low-rank adapter (CT-SLoRA-Adapter) onto the baseline model SAMUS, and evaluates their effects group by group. Group 1 is the baseline model, with Acc at 96.70% and Dice at 85.88%. Group 2, with the addition of the DF-Module, saw Acc increase to 97.56%, and Se and Dice also significantly improve. Group 4, with the CT-SLoRA-Adapter enabled, further improved all indicators, with Acc reaching 97.44%. Group 5, with both the Med-Prompt-Adapter and DF-Module added, continued to improve overall indicators, especially Se and IoU. Finally, Group 7, with all modules enabled, achieved the best performance, with Acc at 97.86%, Se at 90.11%, Dice at 87.47%, and a significant reduction in HD. Overall, the collaborative integration of multiple adapters significantly enhanced model robustness and segmentation accuracy, improving Acc by 1.16 percentage points and Se by 3.06 percentage points compared to the baseline, validating the key role of adapters in performance optimization.

[0242] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered by the present invention.

Claims

1. An ultrasonic medical image segmentation method based on CNN and SAM fusion, characterized in that, Comprising the following steps: Step 1: Introduce a medical prompt adapter in the ViT image encoder, the medical prompt adapter Med PromptAdapter is located between the patch embedding module and the window transformation module in the ViT image encoder, taking the patch embedding features output by the backbone network and the gradient prompt information Grad extracted from the medical image as input, through the cascaded multi-layer perceptron and nonlinear mapping structure, while keeping the overall network parameter quantity and computational quantity controllable, realizing the explicit modeling and injection of medical prior prompt information; Step 2: The medical image is introduced into the CNN and the dual-channel feature fusion module to fuse high-frequency details and global semantic information; Step 3: The output of the medical image after the Norm&Add processing of the ViT in step 1 and the output of step 2 are fused through the cross-branch low-rank adapter CT-SLoRA Adapter to output the local and global information, and then the mask decoder is decoded to predict the final segmentation mask, i.e. the lesion area in the medical image. 2.The CNN and SAM fusion-based medical ultrasound image segmentation method of claim 1, wherein, In step 1, the input medical image is processed by the Patch Embedding module to divide the image into multiple small blocks Patch, thereby extracting the local features of the image. The local features are then injected with spatial position information through the Position Embedding module to ensure that the model can understand the spatial distribution of image features and maintain the structural information of the image. After the Adapter module enhances and reconstructs the features, they are fused with the output of the Med Prompt Adapter. 3.The CNN and SAM fusion based medical ultrasound image segmentation method of claim 1, wherein, The Prompt Encoder module is introduced to convert the point prompt Point Prompt from the user into structured information that the model can understand. 4.The CNN and SAM fusion based medical ultrasound image segmentation method of claim 1, wherein, After the input medical image is processed by the patch embedding module of the backbone encoder, the original image is divided into N non-overlapping image blocks patch, and a patch-level feature sequence is obtained through linear projection. A gradient prompt information is constructed during the preprocessing of the medical image, and the gradient prompt information is converted into patch-level prompt features. In the feature fusion stage, the Med Prompt Adapter aligns the patch embedding features and the gradient prompt features according to the patch index, and then performs splicing or linear combination to obtain a joint input representation. The joint input representation is input into an adjustable multi-layer perception subnetwork In this way, the intermediate prompt features of stage i are obtained. A GELU nonlinear activation function is applied to the intermediate prompt features. After nonlinear mapping, a projection multi-layer perception performs dimension recovery and adaptation, outputs prompt incremental features, and combines the prompt incremental features with the original patch embedding in a residual manner to obtain patch features after medical prompt adaptation. 5.The CNN and SAM fusion based medical ultrasound image segmentation method of claim 1, wherein, Each patch is linearly projected to the token dimension D to form a patch token sequence, and the Position Embedding injects spatial position information into the token, and the Position Embedding output forms a Position Feature, which is then input into a trainable Adapter module to obtain an incremental feature, which is injected into the main token stream through residual addition. The Patch Embedding output forms a Patch Feature, which is also input into a trainable Adapter module to generate a prompt vector through a trainable mapping and added to the token sequence. The Window Transformerx2 performs two windowed Transformer calculations on the input sequence, performs normalization and connects with a residual connection, and then inputs the Multi-head Attention module to obtain the output MHA(X). A trainable Adapter is inserted after the Multi-head Attention to efficiently fine-tune the parameters. The output of the Adapter is normalized and connected with a residual connection, and then input into the Multi-head Attention module to obtain the second attention output MHA(X). At the same time, the CT-SLoRA Adapter is used to fuse the token features from the ViT branch and the convolution features output in step 2 to obtain the fused features. Finally, the fused features are element-wise added to the second MHA(X) and the output of the Adapter branch to obtain the output features of this stage. 6.The CNN and SAM fusion based medical ultrasound image segmentation method of claim 1, wherein, In step 2, the medical image is fused by a dual-channel feature fusion, and the dual fusion module Dual Fusion Module is composed of a high-channel path and a low-channel path. A set of basic convolution sub-modules is arranged on each branch, that is, a series of 3x3 convolution, batch normalization and GELU activation operations are performed on the input features to realize the preliminary linear mapping and nonlinear transformation of the original features, and the numerical range and distribution of the two features are standardized. After the basic convolution, each branch further concatenates 6 groups of recursive convolution ReConv+GELU units to form a lightweight recursive residual convolution structure; each ReConv unit uses a 3x3 convolution kernel to reduce the parameter amount while keeping the spatial resolution unchanged, and is connected through the residual connection between the convolution output and the input feature. After 6 times of ReConv3x3+GELU step-by-step superposition, each branch can fully mine the local context information, and the texture details, structure contours and target boundaries are refined and enhanced, so as to obtain the enhanced upper branch feature and the lower branch feature respectively. The two enhanced features are respectively passed through a layer of 3x3 convolution and BatchNorm normalization, and the two features are structurally aligned in the channel dimension and statistical distribution to obtain the aligned features and Then, the two aligned features are input into the fusion unit to complete the consistent fusion of multi-scale semantics through element-level addition operation to obtain the unified feature , Adaptive re-scaling of fusion features: First, F fuse is applied with GELU activation to obtain F a =GELU(F fuse ), then linear transformation is performed on the channel dimension through 1x1 convolution to obtain F b =Conv1x1(F a ), then BatchNorm normalization is applied to F b to obtain F c =BN(F b ), finally the feature is mapped to the interval (0, 1) through the Sigmoid function to generate an attention weight map A=σ(F c ), in the output stage, the attention weight map is multiplied element by element with the fusion feature to obtain the final output feature of the module O=A⊙F fuse , where "⊙" represents element-wise multiplication operation.

7. The CNN and SAM fusion-based medical ultrasound image segmentation method of claim 6, wherein, In step 2, the fusion features of the dual-channel fusion module are subjected to a max-pooling operation to achieve initial down-sampling , and then are input into an initial convolution sub-module ConvModule to obtain , After obtaining the initial convolutional features The CNN module internally builds a basic unit of "GHPA-MaxPool-GHPA-Conv" and repeats the unit 4 times as a whole to form a multi-level local feature extraction and multi-axis attention enhancement structure, and outputs the features . 8.The CNN and SAM fusion based medical ultrasound image segmentation method of claim 7, wherein, The CT-SLoRA Adapter adopts the "cross-branch gating + low-rank linear adaptation (LoRA)" method: first, a spatially related gating weight is generated from the CNN and ViT features, and then the ViT features are incrementally modulated with low-rank parameters, thereby realizing the fusion of local and global information while keeping the backbone frozen or with a small amount of training, and reducing the additional memory and parameter overhead. 9.The CNN and SAM fusion based medical ultrasound image segmentation method of claim 8, wherein, The CT-SLoRA Adapter performs element-wise fusion on the CNN Feature and the ViT Feature, and obtains the fused feature Two types of statistical descriptions are calculated, corresponding to the Max and Mean branches in the figure, respectively. The maximum pooling branch Max obtains , and the average pooling branch Mean obtains , and through channel dimension pooling, we obtain: The two are spliced in the channel dimension, and the spliced enters Conv7x7 in the figure to generate a gating response, and then the gating value is compressed to the [0, 1] interval through Sigmoid to obtain the final cross-branch gating weight map. The element-wise multiplication gating is performed on the fused feature, and then the gated increment is injected into the ViT feature stream in a residual manner. After completing the cross-branch gating injection, the concatenated low-rank adaptation sublayer is entered, which is LoRA Linear→GELU→LoRA Linear.

Images