Weakly supervised medical image segmentation method based on feature reference enhancement prompt and system thereof

Abstract

The application discloses a weakly supervised medical image segmentation method based on feature reference enhancement prompt and a system thereof, and belongs to the technical field of image analysis. Step 1: a reference feature prompt adapter for generating dense feature prompts is constructed; step 2: based on the adapter of step 1, a cross-domain medical image double encoder is constructed; step 3: based on the cross-domain medical image double encoder of step 2, a bidirectional fusion module is used to uniformly align and bidirectionally fuse cross-domain features; and step 4: based on the bidirectional fusion of step 3, a consistency double decoder is used to decode and reconstruct features output by the double encoder module under the reference features as prompts. The application is used to solve the problems of unstable pseudo label quality, high false detection rate and sensitivity to hyperparameters in weakly supervised medical image segmentation, and overcomes the defects that existing methods are difficult to fully utilize prompt information and cannot capture fine-grained structures.

Description

Technical Field

[0001] This invention belongs to the field of image analysis technology, specifically relating to a weakly supervised medical image segmentation method and system based on feature reference enhancement cues. Background Technology

[0002] Medical image segmentation accurately identifies target organs, tissues, or lesion areas at the pixel level, providing reliable evidence for clinical diagnosis and pathological research. Medical image analysis is an important tool in current clinical medicine and life science research, playing a crucial role in disease diagnosis, treatment, and rehabilitation. Doctors and researchers use medical images to understand detailed information about tissues and organs, perform quantitative analysis, real-time monitoring, and treatment planning for specific organs, in order to formulate appropriate treatment plans.

[0003] With the continuous development of artificial intelligence technology, intelligent medical image-assisted analysis has become a hot topic in the field of medical imaging research, but it still faces many limitations in practical applications. Complete medical image segmentation mask annotation is a very time-consuming and labor-intensive task; pixel-level annotation of targets in medical images is inefficient and costly. In contrast, sparse annotation is easier to obtain than dense annotation, but the annotation information it provides is incomplete, making it difficult to meet high-precision segmentation requirements. Secondly, the number of annotations is limited; medical image data is far less than large-scale training datasets of natural images, especially in rare disease research, where relevant datasets may only contain dozens of samples, resulting in a small number of labeled images and a large number of unlabeled images. Therefore, it is necessary to study medical image analysis methods for constrained problems, overcome the problems of low annotation quality and insufficient annotation, and promote the development and application of the medical imaging field.

[0004] Weakly supervised learning significantly reduces the reliance on fine-grained annotations by fully exploiting the potential of sparsely labeled data, providing a more cost-effective solution for medical image analysis. A common strategy in weakly supervised medical image segmentation is to augment sparse annotations by generating pseudo-labels, using the predictions of a randomized hybrid network model as supplementary supervision to guide the model training process. However, the effectiveness of such methods largely depends on the quality of the pseudo-labels. In practical applications, model-generated pseudo-labels inevitably contain noise, requiring the introduction of complex constraints or regularization strategies to ensure the reliability of the supervision signal.

[0005] Unlike traditional models, the SegmentAnythingModel (SAM) segmentation model features an interactive design, performing segmentation tasks based on input cues, including both sparse and dense cues. This cue-driven mechanism provides a natural and efficient way to implement weakly supervised segmentation and pseudo-label generation. However, existing weakly supervised methods have not fully explored the potential of the SAM cue mechanism. Recent research has attempted to use hand-drawn annotations as cue input to SAM to obtain coarse masks, and then improve their quality through a series of complex post-processing steps. However, this strategy is prone to many false detections in the pseudo-mask and is highly sensitive to hyperparameter settings, making the model heavily dependent on the quality of pseudo-labels, thus limiting its generalization ability. Furthermore, due to the lack of dense cue information, this method struggles to capture the fine-grained structural features of the target, limiting its performance in fine-grained segmentation tasks.

[0006] In summary, existing weakly supervised methods still have significant limitations in utilizing the cueing mechanism of SAM (Sparse Awareness Model): on the one hand, they directly rely on pre-trained encoders and lack task adaptation capabilities, making pseudo-labels susceptible to noise; on the other hand, sparse cueing is insufficient to provide adequate structural information, making it difficult for the model to obtain refined target representations. Therefore, how to fully utilize dense cueing and improve the reliability of pseudo-labels has become a key issue driving the further development of weakly supervised medical image segmentation. Summary of the Invention

[0007] This invention provides a weakly supervised medical image segmentation method and system based on feature reference enhancement cues, which solves the problems of unstable pseudo-label quality, high false detection rate and sensitivity to hyperparameters in weakly supervised medical image segmentation. At the same time, it overcomes the shortcomings of existing methods that are unable to make full use of cues and cannot capture fine-grained structures.

[0008] This invention is achieved through the following technical solution: A weakly supervised medical image segmentation method based on feature reference enhancement cues, the weakly supervised medical image segmentation method comprising the following steps: Step 1: Construct a reference feature cue adapter for generating dense feature cues; Step 2: Based on the adapter from Step 1, construct a cross-domain medical image dual encoder; Step 3: Based on the cross-domain medical image dual encoder of Step 2, a bidirectional fusion module is used to uniformly align and bidirectionally fuse cross-domain features; Step 4: Based on the bidirectional fusion in Step 3, a consistent dual decoder is used to decode and reconstruct the features output by the dual encoder module with reference features as a prompt.

[0009] Furthermore, the reference feature cue adapter in step 1 includes a reference feature extractor and a cue adapter; The input to the reference feature extractor includes semantic features from the cross-domain SAM encoder. Structural features of lightweight U-Net encoders The semantic features of the cross-domain SAM encoder The channel attention mechanism is used to enhance representative semantic information; the structural features of the lightweight U-Net encoder. Employing spatial attention mechanisms to enhance representative structural information; Based on representative semantic and structural information, element-level fusion is performed through channel attention and spatial attention mechanisms to obtain a selective attention mask that adaptively weights the two features to generate reference features. The cue adapter uses reference features from a reference feature extractor to construct a dense cue embedding that is category-specific and has multi-scale expressive power.

[0010] Furthermore, the cross-domain medical image dual encoder in step 2 includes two feature extraction branches with complementary characteristics; The first encoder adopts a SAM-based cross-domain medical image encoder, which outputs a global semantic representation with strong generalization ability; The second encoder employs a lightweight U-Net convolutional network, specifically designed for extracting local structures and spatial details from medical images.

[0011] Furthermore, the bidirectional fusion in step 3 aligns the global features from the Transformer encoder and the local features from the convolutional neural network encoder through upsampling and downsampling. This feature-level coupling maps them to the same feature space, strengthening the representation consistency between the two branches and achieving effective cross-branch feature interaction and supplementation.

[0012] Furthermore, step 4, the consistent dual decoder, specifically includes a SAM decoding branch and a U-Net decoding branch. The SAM decoding branch uses the fused global semantic features and the cue embeddings generated by the FRP-Adapter to generate pseudo-labels; the U-Net decoding branch generates prediction results with stronger boundary sensitivity based on the convolutional structure; while maintaining structural independence, the two branches achieve dual alignment at the semantic and structural levels through consistency loss.

[0013] A weakly supervised medical image segmentation system based on feature reference enhancement cues is disclosed. The system employs the aforementioned weakly supervised medical image segmentation method based on feature reference enhancement cues, and includes the following steps: Reference Feature Cueing Adapter: Used to generate dense feature cues; Cross-domain medical image dual encoder: used to simultaneously acquire global semantic information and local fine-grained structural features of medical images; Bidirectional fusion module: performs unified alignment and bidirectional fusion of cross-domain features; Consistent Dual Decoder: The features output by the dual encoder module are decoded and reconstructed with reference features as a cue.

[0014] Furthermore, the reference feature cue adapter includes a reference feature extractor and a cue adapter; The input to the reference feature extractor includes semantic features from the cross-domain SAM encoder. Structural features of lightweight U-Net encoders The semantic features of the cross-domain SAM encoder Channel attention mechanism is used to enhance representative semantic information; Structural features of the lightweight U-Net encoder Employing spatial attention mechanisms to enhance representative structural information; Based on representative semantic and structural information, element-level fusion is performed through channel attention and spatial attention mechanisms to obtain a selective attention mask that adaptively weights the two features to generate reference features. The cue adapter uses reference features from a reference feature extractor to construct a dense cue embedding that is category-specific and has multi-scale expressive power.

[0015] Furthermore, the cross-domain medical image dual encoder includes two feature extraction branches with complementary characteristics; The first encoder adopts a SAM-based cross-domain medical image encoder, which outputs a global semantic representation with strong generalization ability; The second encoder employs a lightweight U-Net convolutional network, specifically designed for extracting local structures and spatial details from medical images.

[0016] Furthermore, the bidirectional fusion module aligns global features from the Transformer encoder and local features from the convolutional neural network encoder through upsampling and downsampling. By mapping these features to the same feature space through this feature-level coupling, the consistency of representation between the two branches is enhanced, enabling effective cross-branch feature interaction and supplementation.

[0017] Furthermore, the consistent dual decoder specifically includes a SAM decoding branch and a U-Net decoding branch. The SAM decoding branch uses the fused global semantic features and the cue embeddings generated by the FRP-Adapter to generate pseudo-labels; the U-Net decoding branch generates prediction results with stronger boundary sensitivity based on the convolutional structure; while maintaining structural independence, the two branches achieve dual alignment at the semantic and structural levels through consistency loss.

[0018] The beneficial effects of this invention are: This invention achieves high-precision segmentation of cardiac MRI data by introducing dense reference feature cues and a cross-branch consistent pseudo-label generation strategy, which greatly reduces the cost of manual annotation.

[0019] This invention achieves more stable, refined, and generalizable pseudo-label generation and weakly supervised training.

[0020] This invention replaces full annotation with low-cost doodle-style weak annotation, which only requires rough line hints to achieve effective constraints on organ regions, allowing unlabeled data to participate in model training and significantly reducing the manpower consumption of medical image annotation.

[0021] By extracting dense feature hints and combining them with feature consistency constraints, this invention can generate more stable and clearly defined pseudo-labels for unlabeled data, effectively suppressing training bias caused by pseudo-label noise and achieving more reliable semi-supervised learning results.

[0022] Thanks to the introduction of cue enhancement and consistency regularization, this invention maintains stable performance even under conditions of different scanning devices and significant differences in imaging quality, making it highly clinically applicable. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the structure of the feature reference prompt adapter of the present invention.

[0024] Figure 2 This is a schematic diagram of the overall architecture of the weakly supervised medical image segmentation system constructed in this invention.

[0025] Figure 3 This is a schematic diagram of the visualization results of the weakly supervised segmentation experiment conducted by this invention on the ACDC and MSCMR datasets. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0029] The following is in conjunction with the appendix to this application specification. Figure 1-3 The technical solutions in the embodiments of this application are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0030] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0031] Implementation Method 1 This embodiment provides a weakly supervised medical image segmentation method based on feature reference enhancement cues, which addresses the problems of how to achieve efficient and high-quality cross-branch feature fusion between Transformer semantic features and CNN structural features; how to generate category-specific and fully expressive cue embeddings from sparse doodle annotations; and how to utilize bi-branch feature interaction to enhance the semantic consistency and structural fine-grainedness of cue embeddings, thereby improving segmentation accuracy under weak supervision.

[0032] The weakly supervised medical image segmentation method includes the following steps: Step 1: Construct a reference feature cue adapter for generating dense feature cues; Step 2: Based on the adapter from Step 1, construct a cross-domain medical image dual encoder; Step 3: Based on the cross-domain medical image dual encoder of Step 2, a bidirectional fusion module is used to uniformly align and bidirectionally fuse cross-domain features; Step 4: Based on the bidirectional fusion in Step 3, a consistent dual decoder is used to decode and reconstruct the features output by the dual encoder module with reference features as a prompt.

[0033] Furthermore, the reference feature cue adapter in step 1 includes a reference feature extractor and a cue adapter; The input to the Reference Feature Extractor (RFE) includes two types of feature maps: semantic features from the cross-domain SAM encoder. and structural features from the lightweight U-Net encoder To address the two types of features mentioned above, this invention introduces channel attention and spatial attention mechanisms to enhance representative semantic and structural information.

[0034] Channel attention enhancement: for semantic features Global average pooling is performed to obtain the importance response of each channel; this response is then fed into a multilayer perceptron (MLP) to generate a channel attention map. ∈ This attention map is used to evaluate the importance of each channel of the Transformer semantic features.

[0035] Spatial attention enhancement: in structural features Apply a 1×1 convolution to generate a spatial attention map. ∈ This attention map is used to characterize the salience of structural features at different spatial locations.

[0036] Reference Feature Extraction and Generation: This invention performs element-wise fusion of channel attention maps and spatial attention maps to obtain a selective attention mask. , ∈ Where σ represents the Sigmoid activation function. Then, a selective attention mask is used. Adaptive weighting of the two features is applied to generate a reference feature:

[0037]

[0038] Here, ⊙ represents element-wise multiplication. This process can dynamically allocate feature contributions across spatial and channel dimensions, achieving complementary fusion of features from semantic and structural branches to form high-quality reference features.

[0039] The cue adapter uses the aforementioned reference features to construct dense cue embeddings that are category-specific and multi-scale expressive. At the heart of this module lies a two-step interactive process to progressively enrich the learnable query.

[0040] This invention introduces a set of learnable query vectors Q as the basic representation for prompt construction. The query vectors are first compared with semantic reference features. By interacting with cross-attention and self-attention mechanisms, a representation with global semantic information is obtained. :

[0041] Subsequently, the enhanced query vector and structural reference features Further interaction is needed to inject fine-grained spatial and edge structure information:

[0042] Enhanced vector after two-stage interaction It is transformed into dense feature hints through a convolutional projection layer. .

[0043] Furthermore, the cross-domain medical image dual encoder in step 2 includes two feature extraction branches with complementary characteristics, used to simultaneously acquire global semantic information and local fine-grained structural features of the medical image.

[0044] First, the first encoder employs a SAM-based cross-domain medical image encoder. With the original encoder frozen, a low-rank decomposition (LoRA) efficient fine-tuning technique is introduced to adapt the original SAM image encoder across domains, enabling it to better understand the feature differences in texture, organ structure, and imaging patterns of medical images. This encoder can output a global semantic representation with strong generalization capabilities, providing stable high-level semantic features for subsequent cue construction and pseudo-label generation.

[0045] Secondly, the second encoder employs a lightweight U-Net convolutional network, specifically designed for extracting local structures and spatial details from medical images. This lightweight design ensures the model maintains high feature representation capabilities even with limited computational resources, making it particularly suitable for capturing contour boundaries, organ morphology, and small-scale targets. This encoder outputs high-resolution feature maps with good spatial consistency, effectively complementing the semantic information of the first encoder. By combining these two encoders, the dual-encoder system constructed in this invention can balance semantic understanding and detail preservation in medical image segmentation tasks, effectively alleviating the problem of insufficient feature representation faced by a single encoder under weak supervision. This provides a reliable feature foundation for the stable operation of the subsequent reference feature cue encoder and the consistency dual decoder.

[0046] Furthermore, the bidirectional fusion in step 3 aligns the global features from the Transformer encoder and the local features from the convolutional neural network encoder through upsampling and downsampling. This feature-level coupling maps them to the same feature space, strengthening the representation consistency between the two branches and achieving effective cross-branch feature interaction and supplementation.

[0047] Specifically, the bidirectional fusion module is built on the features output by the dual encoders. By designing an adaptive feature transformation structure, it maps the global context features extracted by the Transformer to the convolutional feature space, and injects the local geometry and boundary information of the CNN into the Transformer token representation to achieve bidirectional interaction.

[0048] The specific steps of downward merging from global to local are: The Transformer features from the SAM encoder are fused with convolutional features of the corresponding scale from U-Net after linear projection, dimension matching, and spatial reconstruction. Scale alignment is achieved using Convolutional Downsampling (Conv2D) and Average Pooling (AvgPool) to effectively supplement the semantic representation capabilities of the CNN branches with the long-range dependency information of the Transformer.

[0049] The upward fusion from local to global is specifically as follows: The local boundary and texture features in the convolutional encoder are reorganized into a sequence consistent with the number of Transformer tokens through convolutional upsampling (Conv2D) and interpolation insertion, and then periodically embedded into the Transformer encoder through a feature injection mechanism. In this invention, the Transformer receives local supplementation from the CNN every three layers, effectively enhancing its ability to perceive target shape, edge details, and local changes.

[0050] Building upon the aforementioned bidirectional information exchange, the bidirectional fusion module constructs a unified latent space, enabling the two types of representations to continuously couple across multiple scales. This structure not only enhances the consistency of cross-domain features but also significantly improves the segmentation model's ability to simultaneously model structural details and semantic regions, providing a high-quality fusion feature foundation for subsequent pseudo-label generation and consistent decoding.

[0051] Furthermore, the consistency dual decoder in step 4 is used to decode and reconstruct the features output by the dual encoder module with reference features as cues, and to align and strengthen the cross-branch prediction results through a consistency learning mechanism, thereby improving segmentation accuracy and model robustness under weak supervision. The consistent dual decoder in step 4 specifically includes the SAM decoding branch and the U-Net decoding branch; The consistent dual decoder consists of a SAM decoding branch and a U-Net decoding branch. The SAM decoding branch uses fused global semantic features and cue embeddings generated by the FRP-Adapter to produce pseudo-labels; the U-Net decoding branch generates predictions with stronger boundary sensitivity based on convolutional structures. While maintaining structural independence, the two branches achieve dual alignment at both the semantic and structural levels through consistency loss.

[0052] In the feature reference hint adapter of this invention, the query vector is optimized through two stages of interaction. The material is fed into the convolutional projection module and transformed into dense mask embeddings through a convolutional layer. This embedding incorporates global semantic information from the Transformer branch and local structural information from the CNN branch, encoding the shape distribution and class relevance of the target region in a multi-granular space. Simultaneously, sparse features generated from sparse doodle annotations... As another type of cue information, it is input into the cue encoder to guide the SAM decoder to generate predictions based on human annotations.

[0053] The SAM decoding branch uses the global semantic features output by the Transformer encoder. Based on this, a cue encoder encodes cues, including sparse or dense cues, into cue vectors aligned with the feature space. Based on the cue encoder, this invention generates pseudo-labels and output predictions respectively. High-quality pseudo-tags generated based on dense cueing Embedded through dense masking The SAM decoder, encoded as dense cue input, is used to generate pseudo-labels with fine structure:

[0054] Dense cueing can adaptively fuse semantic and structural features, capture high-resolution structural information such as local boundaries and texture details, complete and smooth sparse annotations, and provide stable self-supervised signals for consistency loss. The pseudo-label serves as the core signal for "consistency supervision" and is used to guide the output alignment of the two branches.

[0055] Sparse cue-driven prediction output Use sparse graffiti As a hint:

[0056] This output serves as the primary supervision branch, used to calculate the loss against the actual weak annotations (graffiti annotations). Its features include: It strongly relies on the spatial priors provided by sparse annotations; it utilizes global semantic modeling capabilities to obtain a relatively complete regional response; and it serves as the main prediction result corresponding to the true weak annotations (used for supervision loss).

[0057] The convolutional decoding branch receives local structural features from the CNN encoder. Generate a second-path prediction result with fine-grained boundary details:

[0058] The prediction provides boundary-sensitive, texture-friendly local structural information; serves as the main output of the second branch, supervised by sparse annotations; and participates in the cross-branch consistency loss with pseudo-labels.

[0059] In summary, this invention constructs a dual-branch prediction system consisting of a Transformer semantic branch and a CNN structural branch, and combines two types of multi-granularity cue mechanisms—sparse cueing and dense cueing—to generate the following three types of output signals: Sparse cue-driven supervised prediction : Generated by the SAM decoder under the guidance of sparse graffiti hints, used for accurate supervision with weak annotations; Dense hint-driven pseudo-tags : Obtained by dense hints generated by the SAM decoder based on fused features, used to construct cross-branch consistency supervision; CNN branch output : Generated by the U-Net decoder based on local structural features, serving as a structure-sensitive prediction for the second branch.

[0060] This mechanism effectively achieves complementary synergy between Transformer global semantic modeling and CNN local structural characterization, and significantly improves medical image segmentation performance through pseudo-label consistency learning under conditions of minimal annotation.

[0061] Figure 1The adapter can automatically generate dense prompts based on the multi-scale features of the input medical images to guide segmentation learning under weak supervision.

[0062] Figure 2 The system includes: (1) A cross-domain medical image dual encoder is used to extract semantic features of the style domain and the content domain respectively; (2) Bidirectional feature fusion module, used to realize the interaction and compensation between features of the two encoders; (3) Reference feature cue encoder, used to... Figure 1 Dense cueing is injected into the feature stream to enhance weak supervision signals; (4) Consistent dual decoder, used to improve segmentation stability and boundary quality under weak supervision by cross-branch consistency constraints.

[0063] Figure 3 The paper demonstrates the cardiac structure segmentation performance obtained by the method of the present invention under weak annotation conditions, including the segmentation effect of the left ventricle, right ventricle and myocardial region, and compares it with the actual annotated area to verify the superiority of the present invention.

[0064] Implementation Method 2 This embodiment provides a weakly supervised medical image segmentation system based on feature reference enhancement cues. The weakly supervised medical image segmentation system uses a weakly supervised medical image segmentation method based on feature reference enhancement cues as described in Embodiment 1. The weakly supervised medical image segmentation system includes the following steps: Reference Feature Cueing Adapter: Used to generate dense feature cues; Cross-domain medical image dual encoder: used to simultaneously acquire global semantic information and local fine-grained structural features of medical images; Bidirectional fusion module: performs unified alignment and bidirectional fusion of cross-domain features; Consistent Dual Decoder: The features output by the dual encoder module are decoded and reconstructed with reference features as a cue.

[0065] Furthermore, the reference feature cue adapter includes a reference feature extractor and a cue adapter; The input to the reference feature extractor includes semantic features from the cross-domain SAM encoder. Structural features of lightweight U-Net encoders The semantic features of the cross-domain SAM encoder The channel attention mechanism is used to enhance representative semantic information; the structural features of the lightweight U-Net encoder. Employing spatial attention mechanisms to enhance representative structural information; Based on representative semantic and structural information, element-level fusion is performed through channel attention and spatial attention mechanisms to obtain a selective attention mask that adaptively weights the two features to generate reference features. The cue adapter uses reference features from a reference feature extractor to construct a dense cue embedding that is category-specific and has multi-scale expressive power.

[0066] Furthermore, the cross-domain medical image dual encoder includes two feature extraction branches with complementary characteristics; The first encoder adopts a SAM-based cross-domain medical image encoder, which outputs a global semantic representation with strong generalization ability; The second encoder employs a lightweight U-Net convolutional network, specifically designed for extracting local structures and spatial details from medical images.

[0067] Furthermore, the bidirectional fusion module aligns global features from the Transformer encoder and local features from the convolutional neural network encoder through upsampling and downsampling. By mapping these features to the same feature space through this feature-level coupling, the consistency of representation between the two branches is enhanced, enabling effective cross-branch feature interaction and supplementation.

[0068] Furthermore, the consistent dual decoder specifically includes a SAM decoding branch and a U-Net decoding branch. The SAM decoding branch uses the fused global semantic features and the cue embeddings generated by the FRP-Adapter to generate pseudo-labels; the U-Net decoding branch generates prediction results with stronger boundary sensitivity based on the convolutional structure; while maintaining structural independence, the two branches achieve dual alignment at the semantic and structural levels through consistency loss.

[0069] Consistency supervision loss based on pseudo-labels: In this embodiment, to enhance the robustness of the overall segmentation system in weakly labeled scenarios, this invention constructs a cross-branch consistency learning mechanism based on pseudo-labels. Specifically, the system first generates dense cues using a reference feature cue adapter and inputs them into a dual-branch decoder to obtain the SAM branch outputs. With U-Net branch output Subsequently, the system uses the pseudo-tags generated by the SAM branch. As an additional monitoring signal.

[0070] Pseudo-labels By combining dense cueing with the SAM decoder The output then constitutes a self-supervised estimate of the true segmentation result. When weak annotations (such as graffiti annotations) are insufficient to cover the complete anatomical structure, pseudo-labels can provide denser and more structurally complete supervision information.

[0071] To ensure consistency between the two branches at the output level, this invention constructs a cross-branch consistency loss to constrain... and All must be matched with the pseudo-tags generated by the system. Alignment ensures that the two branches produce similar and reliable results even under weak labeling conditions.

[0072] The consistency loss is calculated as follows:

[0073] In this invention, the Dice loss measures the degree of overlap between the prediction and the pseudo-label in terms of region and contour, while the cross-entropy loss provides pixel-level classification constraints. A weight of 0.5 is intentionally assigned to the cross-entropy loss in this invention to prevent overfitting when the pseudo-label is not entirely accurate, thus balancing the contributions of the two types of losses.

[0074] Supervised loss based on graffiti annotation: In this embodiment, monitoring loss Used to measure branch output , With sparse graffiti markings The differences between them. This invention employs a weighted combination of Dice loss and cross-entropy loss to balance structural consistency and pixel accuracy:

[0075] In this design, the weight of the Dice loss is set to 0.8, mainly because doodle annotations usually only provide structural cues and are difficult to cover the complete target. A higher Dice weight can encourage the model to pay more attention to the shape integrity of the target region, thereby improving the structural recovery ability under weak annotation conditions.

[0076] Total loss function design: To simultaneously integrate consistency supervision from sparse graffiti annotations and pseudo-labels, this invention constructs the following total loss function.

[0077] in, Used to adjust the intensity of supervision from real weak annotations; The importance of adjusting pseudo-label consistency supervision; in this example =1.0, .

[0078] This weight design means that the system gives equal importance to real weakly labeled signals and self-supervised signals, while using a lower consistency weight to avoid false label noise interfering with network training.

[0079] Model training method steps: Based on the above loss design, the training method of this invention includes the following steps. During the training phase, this invention employs a freezing strategy for the SAM image encoder, introducing low-rank adaptation (LoRA) modules only in the decoder and cue encoder for parameter fine-tuning. This approach maintains the stable feature representation capability of the large model while significantly reducing training overhead and improving model learning efficiency under weak supervision.

[0080] Step S1: Simultaneously feed the input medical image into the cross-domain dual encoder to generate global features. (From the fine-tuned SAM image encoder) and structural features From (U-Net encoder). Among them: the SAM image encoder is kept completely frozen, preserving its general visual priors learned on large-scale data; a LoRA module is added inside the SAM image encoder to adapt to the new distribution in the medical imaging field in a low-rank manner; new task features are learned through incremental matrix, so that training can achieve good transfer performance while significantly reducing the number of parameters.

[0081] Step S2: Generate reference feature hints The reference features output from the dual encoders are input into the reference feature cue adapter. After feature extraction through channel attention and spatial attention, dense feature cues are generated through self-attention and cross-attention. .

[0082] Step S3: Dual decoder output prediction and pseudo-label generation For the SAM branch, respectively based on the clues from real graffiti annotations. Below and dense feature hints generated Generate prediction results and pseudo-tags For the U-Net branch output convolution prediction results .

[0083] Step S4: Calculate the supervision and consistency loss using graffiti labels and pseudo-labels respectively.

[0084] This approach achieves efficient transfer learning while ensuring model stability.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A weakly supervised medical image segmentation method based on feature reference enhancement cues, characterized in that, The weakly supervised medical image segmentation method includes the following steps: Step 1: Construct a reference feature cue adapter for generating dense feature cues; Step 2: Based on the adapter from Step 1, construct a cross-domain medical image dual encoder; Step 3: Based on the cross-domain medical image dual encoder of Step 2, a bidirectional fusion module is used to uniformly align and bidirectionally fuse cross-domain features; Step 4: Based on the bidirectional fusion in Step 3, a consistent dual decoder is used to decode and reconstruct the features output by the dual encoder module with reference features as a prompt.

2. The weakly supervised medical image segmentation method according to claim 1, characterized in that, The reference feature cue adapter in step 1 includes a reference feature extractor and a cue adapter; The input to the reference feature extractor includes semantic features from the cross-domain SAM encoder. Structural features of lightweight U-Net encoders The semantic features of the cross-domain SAM encoder The channel attention mechanism is used to enhance representative semantic information; the structural features of the lightweight U-Net encoder. Employing spatial attention mechanisms to enhance representative structural information; Based on representative semantic and structural information, element-level fusion is performed using channel attention and spatial attention mechanisms to obtain a selective attention mask that adaptively weights the two features to generate reference features. The cue adapter uses reference features from a reference feature extractor to construct a dense cue embedding that is category-specific and has multi-scale expressive power.

3. The weakly supervised medical image segmentation method according to claim 1, characterized in that, The cross-domain medical image dual encoder in step 2 includes two feature extraction branches with complementary characteristics; The first encoder adopts a SAM-based cross-domain medical image encoder, which outputs a global semantic representation with strong generalization ability; The second encoder employs a lightweight U-Net convolutional network, specifically designed for extracting local structures and spatial details from medical images.

4. The weakly supervised medical image segmentation method according to claim 1, characterized in that, The bidirectional fusion in step 3 aligns global features from the Transformer encoder and local features from the convolutional neural network encoder through upsampling and downsampling. This feature-level coupling maps to the same feature space, strengthening the representation consistency between the two branches and achieving effective cross-branch feature interaction and supplementation.

5. The weakly supervised medical image segmentation method according to claim 2, characterized in that, Step 4, the consistent dual decoder, specifically includes the SAM decoding branch and the U-Net decoding branch; The SAM decoding branch uses the fused global semantic features and the cue embeddings generated by the FRP-Adapter to produce pseudo-labels; the U-Net decoding branch generates prediction results with stronger boundary sensitivity based on the convolutional structure; while maintaining structural independence, the two branches achieve dual alignment at the semantic and structural levels through consistency loss.

6. A weakly supervised medical image segmentation system based on feature reference enhancement cues, characterized in that, The weakly supervised medical image segmentation system uses a weakly supervised medical image segmentation method based on feature reference enhancement cues as described in any one of claims 1-5, and the weakly supervised medical image segmentation system includes the following steps: Reference Feature Cueing Adapter: Used to generate dense feature cues; Cross-domain medical image dual encoder: used to simultaneously acquire global semantic information and local fine-grained structural features of medical images; Bidirectional fusion module: performs unified alignment and bidirectional fusion of cross-domain features; Consistent Dual Decoder: The features output by the dual encoder module are decoded and reconstructed with reference features as a cue.

7. The weakly supervised medical image segmentation system according to claim 6, characterized in that, The reference feature cue adapter includes a reference feature extractor and a cue adapter; The input to the reference feature extractor includes semantic features from the cross-domain SAM encoder. Structural features of lightweight U-Net encoders The semantic features of the cross-domain SAM encoder The channel attention mechanism is used to enhance representative semantic information; the structural features of the lightweight U-Net encoder. Employing spatial attention mechanisms to enhance representative structural information; Based on representative semantic and structural information, element-level fusion is performed through channel attention and spatial attention mechanisms to obtain a selective attention mask that adaptively weights the two features to generate reference features. The cue adapter uses reference features from a reference feature extractor to construct a dense cue embedding that is category-specific and has multi-scale expressive power.

8. The weakly supervised medical image segmentation system according to claim 6, characterized in that, The cross-domain medical image dual encoder includes two feature extraction branches with complementary characteristics; The first encoder adopts a SAM-based cross-domain medical image encoder, which outputs a global semantic representation with strong generalization ability; The second encoder employs a lightweight U-Net convolutional network, specifically designed for extracting local structures and spatial details from medical images.

9. The weakly supervised medical image segmentation system according to claim 6, characterized in that, The bidirectional fusion module aligns global features from the Transformer encoder and local features from the convolutional neural network encoder through upsampling and downsampling. This feature-level coupling maps the features to the same feature space, enhancing the representation consistency between the two branches and achieving effective cross-branch feature interaction and supplementation.

10. The weakly supervised medical image segmentation system according to claim 6, characterized in that, The consistent dual decoder specifically includes a SAM decoding branch and a U-Net decoding branch. The SAM decoding branch uses the fused global semantic features and the cue embeddings generated by the FRP-Adapter to generate pseudo-labels; the U-Net decoding branch generates prediction results with stronger boundary sensitivity based on the convolutional structure; while maintaining structural independence, the two branches achieve dual alignment at the semantic and structural levels through consistency loss.

Images