A method and system for semantic-guided endoscopic bleeding image segmentation

Abstract

The application discloses a semantic-guided endoscope bleeding image segmentation method and system, relates to the technical field of computer systems based on specific computing models and medical image processing, and extracts hierarchical visual features of an endoscope image and semantic features of natural language prompt text; the semantic features are injected into a hierarchical visual feature extraction process through a semantic adapter to obtain semantic-enhanced visual features; a coarse segmentation mask of a bleeding area is generated through a prior segmentation head; a multi-modal prompt set is constructed based on the coarse segmentation mask and the semantic features; the endoscope image, the semantic-enhanced visual features and the multi-modal prompt set are input into a segmentation base model, and a final bleeding area segmentation mask is output after fine segmentation. Through the design of a lightweight semantic adapter, natural language semantics is efficiently injected into hierarchical visual features, and a two-stage segmentation strategy is combined to realize accurate bleeding segmentation from semantic perception to boundary refinement.

Description

Technical Field

[0001] This invention relates to the fields of computer systems and medical image processing technology based on specific computational models, and particularly to a semantically guided method and system for endoscopic bleeding image segmentation. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] In endoscopic minimally invasive surgery, real-time and accurate segmentation of intraoperative bleeding areas is crucial for ensuring surgical safety. Especially in liquid environments such as spinal endoscopy, blood spreads and its boundaries become blurred, and its color is similar to that of tissue, leading to insufficient precision of traditional segmentation methods.

[0004] Despite significant advancements in automated segmentation technology in medical imaging, achieving bleeding segmentation in spinal endoscopy scenarios presents greater challenges than in traditional imaging (such as CT and MRI) or other endoscopic scenarios (such as gastrointestinal endoscopy). The unique "underwater environment" of spinal endoscopy causes blood to diffuse irregularly in a cloud-like pattern within saline solution, exhibiting extremely blurred boundaries. The difficulties stem primarily from two aspects: first, the highly variable morphology and blurred contours of the bleeding area make precise depiction difficult; second, strong color similarity—fresh blood may resemble cauterized tissue, while diluted or aged blood easily blends into a dark background, leading to significant semantic ambiguity.

[0005] Existing deep learning-based segmentation methods face two main challenges: first, relying solely on visual features makes it difficult to solve highly semantically ambiguous problems (such as distinguishing between bleeding and similar backgrounds); second, existing methods that combine language prompts fail to deeply and efficiently inject semantic information into the core process of visual feature extraction, thus having limited guiding effect.

[0006] Existing technologies, such as traditional image processing methods based on saturation thresholds and pixel similarity, do not involve deep semantic understanding; other existing technologies enhance segmentation by constructing and interacting visual graphs with predefined semantic graphs, and their semantic expression is relatively static and fixed. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention provides a semantically guided endoscopic bleeding image segmentation method and system, aiming to achieve accurate identification of bleeding areas.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention provides a semantically guided endoscopic hemorrhage image segmentation method, comprising: Acquire intraoperative endoscopic images and corresponding natural language prompts; Feature extraction is performed on the endoscopic images and natural language prompts to obtain hierarchical visual features and semantic features; the semantic features include word-level semantic features and global semantic features. The semantic features are injected into the extraction process of the hierarchical visual features through a semantic adapter to obtain semantically enhanced visual features; Based on the semantically enhanced visual features and the semantic features, a coarse segmentation mask for the bleeding region is generated using a priori segmentation head; Based on the coarse segmentation mask and the semantic features, a multimodal cue set including bounding box cue, mask cue, and language cue is constructed; The endoscopic image, the semantically enhanced visual features, and the multimodal cue set are input into the segmentation base model, and refined segmentation is performed through its mask decoder to output the final bleeding region segmentation mask.

[0009] In a further technical solution, the semantic adapter includes a multi-scale convolution module and a multi-head cross-attention fusion module.

[0010] A further technical solution involves using a multi-scale convolution module to extract and fuse features from different receptive fields of the input hierarchical visual features to obtain enhanced spatial features. The multi-head cross-attention fusion module uses global semantic features as keys and values, and enhanced spatial features as queries, to perform feature alignment and fusion, resulting in semantically enhanced visual features. After selection and output, multi-level semantically enhanced visual features are obtained.

[0011] A further technical solution is that the prior segmentation head includes a global context prior module, a local semantic alignment module, and an upsampling projector. The global context prior module integrates multi-level semantically enhanced visual features and global semantic features, and introduces coordinate attention enhancement to obtain spatial perception features.

[0012] In a further technical solution, the local semantic alignment module performs fine-grained alignment between spatially perceived features and word-level semantic features to obtain text-aligned enhanced visual features; the upsampling projector upsamples the text-aligned enhanced visual features and maps them to generate a coarse segmentation mask.

[0013] A further technical solution involves constructing a multimodal tooltip set that includes bounding box hints, mask hints, and language hints, specifically as follows: Extract the bounding box from the coarse segmentation mask as a bounding box cue; Use the coarse segmentation mask as a mask hint; The global semantic features are mapped to the embedding space of the prompt encoder of the segmentation base model to obtain language prompts.

[0014] A further technical solution is that the segmentation base model employs a combined loss:

[0015] in, , , , To balance the weight hyperparameters, Text visual contrast loss in coarse segmentation masks The text visual contrast loss for the final segmentation mask, For the Dice loss of the coarse segmentation mask, The Dice loss for the final segmentation mask.

[0016] Secondly, the present invention provides a semantically guided endoscopic hemorrhage image segmentation system, comprising: The data acquisition module is configured to acquire intraoperative endoscopic images and corresponding natural language prompts. The multimodal feature extraction module is configured to extract features from the endoscopic image and the natural language prompt text respectively to obtain hierarchical visual features and semantic features; the semantic features include word-level semantic features and global semantic features. A multimodal feature fusion module is configured to inject the semantic features into the extraction process of the hierarchical visual features through a semantic adapter to obtain semantically enhanced visual features; The prior segmentation module is configured to generate a coarse segmentation mask for the bleeding region based on the semantically enhanced visual features and the semantic features through the prior segmentation head. The prompt generation module is configured to: construct a multimodal prompt set including bounding box prompts, mask prompts, and language prompts based on the coarse segmentation mask and the semantic features; The fine segmentation module is configured to: input the endoscopic image, the semantically enhanced visual features, and the multimodal cue set into the segmentation base model, perform fine segmentation through its mask decoder, and output the final bleeding region segmentation mask.

[0017] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a semantically guided endoscopic hemorrhage image segmentation method as described in the first aspect.

[0018] Fourthly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a semantically guided endoscopic hemorrhage image segmentation method as described in the first aspect.

[0019] The above one or more technical solutions have the following beneficial effects: This invention achieves precise bleed segmentation from semantic perception to boundary refinement by designing a lightweight semantic adapter to efficiently inject natural language semantics into hierarchical visual features and combining it with a two-stage segmentation strategy.

[0020] This invention proposes a bleed image segmentation model, namely the SGBA-Net framework, which injects linguistic semantics into visual features, enabling the model to have semantic perception capabilities of bleed regions. The bleed image segmentation model designs a lightweight semantic adapter, fusing the bleed semantic information extracted by CLIP into the multi-layer features of the DINOv2 encoder, thus imbuing early visual features with bleed semantic capabilities and reducing visual confusion. SGBA-Net employs a two-stage coarse-to-fine strategy: first, a coarse mask is generated by the prior segmentation head (PSH), and then this mask is used as a spatial cue input to SAM2 to obtain high-precision boundaries.

[0021] To validate SGBA-Net, this invention constructs the SeBlood dataset, which is the first dataset with pixel-level annotations for spinal endoscopic bleeding, covering various stages of real intraoperative bleeding and providing a standard benchmark for subsequent research. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of a semantically guided endoscopic hemorrhage image segmentation method according to an embodiment of the present invention; Figure 2 This is a structural diagram of the hemorrhage image segmentation model according to an embodiment of the present invention; Figure 3 This is a structural diagram of the semantic adapter according to an embodiment of the present invention; Figure 4 This is a structural diagram of the global context prior module in an embodiment of the present invention; Figure 5 This is a structural diagram of the local semantic alignment module in an embodiment of the present invention; Figure 6This is a qualitative comparison chart of different methods on the SeBlood test set according to the embodiments of the present invention, wherein (a) represents microbleeds, (b) represents patchy hemorrhages, and (c) represents diffuse hemorrhages. Detailed Implementation

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0027] Traditional convolutional neural networks (CNNs), such as the U-Net series, can effectively capture multi-scale features, but are limited by their local receptive fields, often resulting in fragmented or incomplete segmentation in diffuse hemorrhage regions. Transformer-type models, such as Swin-Unet, have the ability to model long-range dependencies on global context, but are still susceptible to semantic confusion in highly similar regions. Recent large-scale visual models have provided new opportunities for medical image analysis; however, each of these models still has its limitations. Therefore, effectively fusing semantic and visual information has become a key challenge in achieving high-quality hemorrhage segmentation.

[0028] This invention proposes a method and system for precise segmentation of bleeding areas in endoscopic surgery by integrating natural language semantics and a large visual model. It achieves early fusion of semantics and visual depth through dynamic language prompts and a parameter-efficient adapter, and performs fine-tuning in conjunction with the basic segmentation model.

[0029] Example 1 like Figure 1 As shown, this embodiment discloses a semantically guided endoscopic hemorrhage image segmentation method, which includes the following steps: S1: Acquire intraoperative endoscopic images and corresponding natural language prompts; In this embodiment, a single-frame RGB image (endoscopic image) I from a spinal endoscopy surgery video is acquired and its size is uniformly adjusted; in this embodiment, it is adjusted to 448×448. The natural language prompt text T corresponding to the RGB image is set to "red bleeding area".

[0030] S2: Extract features from the endoscopic image and natural language prompt text respectively to obtain hierarchical visual features and semantic features; In this embodiment, bleeding areas in endoscopic scenarios often lack obvious visual features, and their color and shape are easily confused with surrounding tissues: fresh bleeding may resemble cauterized tissue, while old bleeding will blend into a dark background, making it difficult for the visual model to accurately identify it. To alleviate semantic ambiguity, this invention introduces natural language prompts as priors to guide the model to focus on areas related to bleeding.

[0031] like Figure 2 As shown, this invention proposes a bleeding image segmentation model, namely the SGBA-Net framework. SGBA-Net fuses the semantic prior extracted by CLIP with the visual features acquired by DINOv2 through a semantic adapter. Subsequently, the prior segmentation head (PSH) combines global contextual prior (GCP), local semantic alignment (LSA), and a dynamic upsampling projector to generate an initial segmentation mask. Finally, SAM2 refines the prior mask using multimodal cue embedding, achieving high-precision segmentation of intraoperative bleeding areas from coarse to fine. The SGBA-Net framework is a multimodal, semantically guided segmentation framework that injects linguistic priors into visual features, enabling the model to have semantic awareness of bleeding areas, thereby reducing visual ambiguity.

[0032] The bleeding image segmentation model is designed with a semantically guided multimodal encoder, including a text encoder and a visual encoder. The pre-trained visual encoder extracts hierarchical visual features from the endoscopic image, while the pre-trained text encoder extracts semantic features from the cue text.

[0033] (1) A pre-trained CLIP (ViT-L / 14) model is used as the text encoder. Input cue text T, and the resulting text embedding sequence is obtained. :

[0034] Text Embedded Sequence After passing through the CLIP Transformer, a contextualized semantic representation is obtained. Further, word-level semantic features and global semantic features are extracted, represented as follows:

[0035]

[0036] in, Preserving word-level semantic features helps align local details with boundaries. For projection layer operations; For contextualized semantic representation; The length of the text embedding sequence; For the embedded dimension; Provides overall guidance information for global semantic features; This is a pooling operation.

[0037] (2) The pre-trained DINOv2-L / 14 model was used as the visual encoder.

[0038] DINOv2 is pre-trained on large-scale unlabeled data through self-supervised learning, which can capture structured visual knowledge and generate hierarchical features that remain stable under changes in illumination, viewpoint, and partial occlusion. This is especially crucial for highly dynamic intraoperative environments.

[0039] Input the RGB image I to the visual encoder and extract the output of its layers {2, 6, 10, 14, 18, 22} as multi-level visual features. The formula is expressed as:

[0040] in, Represents a visual encoder. Indicates the first Layer feature map, , , They represent the first The spatial dimension and number of channels of the layer feature map Indicates the number of network layers.

[0041] S3: The semantic features are injected into the extraction process of the hierarchical visual features through a semantic adapter to obtain semantically enhanced visual features; In this embodiment, to enhance the visual encoder's sensitivity to the concept of "bleed," a lightweight semantic adapter module is designed, which uses global semantic features... Visual features are fused and embedded into the Transformer layer of DINOv2. By injecting linguistic priors early in the network, the semantic adapter guides feature learning to focus on regions associated with bleeding.

[0042] The semantic adapter includes a multi-scale convolutional module and a multi-head cross-attention (MHCA) fusion module, used for semantic and visual feature alignment and fusion at different levels of the visual encoder. The multi-scale convolutional module extracts and fuses features from different receptive fields of the input hierarchical visual features to obtain enhanced spatial features; the MHCA fusion module uses global semantic features as keys and values, and enhanced spatial features as queries, to perform feature alignment and fusion to obtain semantically enhanced visual features. After selection, the output yields multi-level semantically enhanced visual features.

[0043] The multi-scale convolution module consists of a dimensionality-reduced linear layer (Down), a ReLU activation function, a 1×1 convolutional layer, a 3×3 convolutional layer, and a 5×5 convolutional layer, connected sequentially, with skip connections introduced between the multiple convolutional layers. The multi-head cross-attention fusion module includes a flatten layer (Flatten), a dimensionality-reduced linear layer (Down), a cross-attention module (Cross Attention), residual connections, layer normalization (Add & Norm), and an upsampling projector (Up).

[0044] like Figure 3 As shown, for each layer of selected visual features First, after dimensionality reduction and ReLU activation, we obtain... Next, convolutional layers with kernels of 1×1, 3×3, and 5×5 are used to extract multi-scale features, and these features are then combined with... The residuals are added together to obtain the enhanced spatial features. Finally, the result after dimensionality reduction using a linear layer... For Key and Value, after flattening layer processing For the query, the data is fused using a multi-head cross-attention module, ultimately outputting the semantically enhanced visual features of this layer. (i.e., visual features after semantic injection).

[0045] Specifically, each visual feature It will undergo multi-scale convolution to capture spatial information under different receptive fields, represented as:

[0046]

[0047]

[0048]

[0049]

[0050] in, Indicates primary features, This represents the output features of a 1×1 convolutional layer. This represents the output features of a 3×3 convolutional layer. This represents the output features of a 5×5 convolutional layer. This process allows the "bleed" semantics to continuously guide the allocation of visual attention during feature extraction.

[0051] Subsequently, the multi-head cross-attention module (MHCA) integrates global semantic features. With enhanced spatial features Alignment:

[0052] in As the output of the current Transformer layer, it is passed to the next layer, enabling semantically guided feature extraction to be carried out throughout the entire network. By injecting semantic information into each layer, language cues can continuously guide visual features, allowing the model to maintain strong perception and segmentation capabilities even when the boundaries are blurred or the bleed area is similar to the background.

[0053] To fully utilize hierarchical feature information, this invention selects two intermediate layers and the final layer from DINOv2 as output:

[0054] in, Enhance visual features for multi-level semantics. For the first Semantic-enhanced visual features output by the layer For the first Semantic-enhanced visual features output by the layer For the first Semantic-enhanced visual features output by the layer .

[0055] S4: Based on the semantically enhanced visual features and the semantic features, a coarse segmentation mask (preliminary mask) for the bleeding region is generated using a priori segmentation head. In this embodiment, although multi-level semantically enhanced visual features (visual embeddings) With global semantic features (text embedding) While some alignment has been achieved during the encoding process, further semantic fusion is still needed to achieve accurate cross-modal fusion and precise target localization. To address this, the bleeding image segmentation model proposes a Prior Segmentation Head (PSH), comprising a Global Context Prior (GCP) module, a Local Semantic Alignment (LSA) module, and an Up-Sample Projector. The PSH receives semantic features from the text encoder (…). , ) and multi-level semantically enhanced visual features from the visual encoder The features are then fused to output a fused feature map. and prior mask , as a prompt input for SAM2.

[0056] The prior segmentation head receives the fused multi-level visual features. and semantic features ( , ).

[0057] (1) Global Context Prior Module The global context prior module integrates multi-level semantically enhanced visual features and global semantic features, and introduces coordinate attention enhancement to obtain spatial perception features.

[0058] like Figure 4 As shown, in order to effectively integrate multi-level visual information and further utilize global semantic features... Multi-level semantic enhancement visual features For the initial alignment between them, a global context prior module was designed.

[0059] First, the semantically enhanced visual feature channels at each level are aligned and concatenated, and then fused through a 1×1 convolution to obtain a unified feature. A coordinate attention module is introduced to explicitly encode spatial coordinate information, generate a coordinate attention graph, and integrate it with... Adding them together yields the spatial perception features. At the same time, After projection, it serves as a guide, interacting with semantically enhanced visual features at each level through MHCA to strengthen global semantic consistency.

[0060] Specifically, the channels will first be adjusted using 1×1 convolutions to accommodate subsequent cross-modal interactions:

[0061] in, Enhance visual features with multi-level semantics after channel adjustment.

[0062] To align with visual features, global semantic features Mapped to the shared embedding space via linear projection:

[0063] in, It is a global semantic embedding feature.

[0064] Subsequently, the MHCA module was used to perform cross-modal fusion with global semantic embedding features as the key and value, and the channel-adjusted multi-level semantically enhanced visual features as the query.

[0065] in, This is a multi-scale fusion feature.

[0066] This process enables sentence-level semantics to effectively guide visual features, focusing the model on the target region. Next, multi-scale fused features are concatenated and a unified visual representation is generated using a 1×1 convolution.

[0067] in, To unify characteristics.

[0068] This approach preserves hierarchical feature information while reducing redundant representations. Considering that spatial accuracy is crucial for endoscopic surgical navigation, this invention... CoordConv (Coordinate Attention Enhancement) is introduced. By explicitly encoding spatial coordinate information, the model's ability to locate the bleed region is enhanced, thereby alleviating the inherent limitation of standard convolution being insensitive to spatial location.

[0069] in, This refers to spatial perception features.

[0070] (2) Local semantic alignment module The local semantic alignment module performs fine-grained alignment between spatially aware features and word-level semantic features to obtain text-aligned enhanced visual features.

[0071] like Figure 5 As shown, this invention designs an LSA module based on global context priors to achieve fine-grained local localization, which is crucial for accurately depicting irregular bleeding boundaries. The LSA module employs a hierarchical attention mechanism to progressively inject token-level semantic information into visual features, thereby achieving more accurate cross-modal alignment.

[0072] spatial perception features With word-level semantic features The splicing process involves sequentially passing through a multi-head self-attention layer and a multi-head cross-attention layer (using...). Deep cross-modal interaction is performed for Key / Value pairs, followed by a feedforward network and layer normalization to output finely aligned features. .

[0073] Specifically, LSA first identifies spatial perception features with spatial perception capabilities. fine-grained word-level semantic features The features are concatenated to construct an initial joint feature representation. This representation is then processed by an MHSA layer, and its output is used as the query for subsequent MHCA modules. The self-attention process is defined as follows:

[0074] Subsequently, Further enhance the semantic consistency of visual features and integrate them with textual semantics. Dynamic alignment is then performed. A feedforward network (FFN) is then used to enhance feature representation, and residual connections and layer normalization are employed to ensure training stability. The final generated text alignment-enhanced visual features are denoted as... :

[0075] This fine-grained alignment mechanism enables the model to associate local semantics (such as "red", "region", etc.) with specific pixel patterns, thereby generating semantically consistent and detailed feature maps.

[0076] (3) Upsampling projector The upsampling projector upsamples the text alignment enhancement visual features and maps them to generate a coarse segmentation mask.

[0077] In this embodiment, in order to extract from the decoded low-resolution feature map To obtain the initial segmentation mask, this invention designs a dynamic upsampling projector. Traditional upsampling methods (such as bilinear interpolation or deconvolution) rely on fixed parameters and lack adaptability to content. Therefore, this invention utilizes textual semantic features. Dynamically adjust the convolution kernel parameters to enhance the model's ability to perceive target regions.

[0078] A dynamically upsampling projector is used. The kernel parameters of this projector are determined by... Dynamically generated, thus reducing low resolution Upsampled and mapped to a coarse segmentation mask with the same resolution as the input image. .

[0079] Specifically, dynamic convolution kernels are derived from language features. generate:

[0080] in, , These represent the convolution weights and bias terms, respectively. The fused feature map from the LSA decoder is used to compute the prior cue mask. First, upsampling is performed to recover detailed information, followed by text-guided dynamic convolution.

[0081] in, For the initial mask of the prediction, and Achieve 4× upsampling. This design enables the upsampling process to be language-guided, adaptively recovering details relevant to the bleeding area while suppressing irrelevant background noise.

[0082] S5: Based on the coarse segmentation mask and the semantic features, construct a multimodal cue set including bounding box cue, mask cue, and language cue; In this embodiment, although PSH can generate a high-quality preliminary segmentation mask, further refinement is required using the SAM2 segmentation model to achieve the accuracy needed for clinical applications. After multimodal feature fusion and dynamic cue generation, the fused features are input into SAM2 to generate the final segmentation result. This stage consists of the SAM2 Prompt Encoder and the SAM2 Mask Decoder.

[0083] A multimodal cue set including bounding box cues, mask cues, and language cues is constructed. Specifically, bounding boxes are extracted from coarse segmentation masks as bounding box cues; coarse segmentation masks are used as mask cues; and global semantic features are mapped to the embedding space of the cue encoder of the segmentation base model to obtain language cues.

[0084] Specifically, for coarse segmentation masks Binarize the data, extract the connected region with the largest area, and calculate its minimum bounding rectangle as the bounding box cue. It serves as a mask hint. It incorporates global semantic features. A linear layer is mapped to the embedding space of the SAM2 cue encoder as a language cue.

[0085] To provide more explicit spatial guidance, this invention uses a priori masks generated from PSH. A bounding box is automatically inferred from the data. This serves as a sparsity cue. Specifically, firstly, [the following is used as a sparsity cue]. Binarization yields a binary mask. Then, the largest connected component is extracted through connected component analysis, and its bounding rectangle is calculated:

[0086] in, The function for calculating the minimum bounding rectangle is given. The SAM2 cue encoder then encodes the bounding box cue and mask cue, generating a sparse cue embedding. and densely embedded prompts .

[0087] S6: Input the endoscopic image, the semantically enhanced visual features, and the multimodal cue set into the segmentation base model, perform fine segmentation through its mask decoder, and output the final bleeding region segmentation mask.

[0088] High-resolution visual features extracted from the original image and DINOv2 The sparse and dense cues (bounding boxes, masks, and speech) are input into the segmentation base model (SAM2). The SAM2 cue encoder encodes both sparse and dense cues (boxes and speech), while the mask decoder integrates image features and all cue information to output the final high-precision segmentation mask M. This step effectively corrects the coarse mask. There may be problems such as uneven boundaries and small holes in the process.

[0089] Specifically, sentence-level semantic vectors Mapping yields language prompt embeddings This is then concatenated with the sparse hints to obtain the enhanced sparse hint embedding. To maintain spatial consistency, this invention further incorporates a location embedding generated by the SAM2 location encoding module. Finally, all cues and image features are fed into the SAM2 mask decoder, which receives the image features, cue embeddings, and high-resolution features. Generate the final bleeding region segmentation mask (final segmentation mask):

[0090] in, This is the final segmentation mask.

[0091] A dataset of spinal endoscopic hemorrhage images was collected and labeled, and then divided into training and testing sets proportionally. During the training phase, the backbone parameters of CLIP, DINOv2, and SAM2 were frozen, and only the parameters of the semantic adapter, prior segmentation head, each projection layer, and cue mapping layer were optimized.

[0092] In this embodiment, the AdamW optimizer is used. The total loss function uses combined loss:

[0093] in, , , , To balance the weight hyperparameters, For the initial masking of text visual contrast loss, The text visual contrast loss for the final segmentation mask, For the initial mask's Dice loss, The Dice loss for the final segmentation mask.

[0094] To ensure that the mask predicted by the model can accurately align with the ground truth labels, the initial mask is... and final segmentation mask The Dice loss is calculated for all cases. The Dice loss is defined as follows:

[0095] in, This indicates Dice's loss. and They represent the first The predicted and actual values ​​of each pixel. This is a smoothing term.

[0096] The proposed SGBA-Net framework differs from traditional segmentation networks that rely solely on visual cues. SGBA-Net incorporates natural language semantics from CLIP into the DINOv2 feature learning process. To maintain parameter efficiency, most CLIP and DINOv2 parameters are frozen, and only the lightweight adapter and task-related modules are trained. The prior segmentation head (PSH) first generates a coarse mask of the bleeding region, which is then further refined by SAM2 under the guidance of multimodal cues, achieving high-precision segmentation of the bleeding region in spinal endoscopy images.

[0097] This invention effectively solves the segmentation problem of endoscopic bleeding areas caused by similar colors and blurred boundaries through deep semantic guidance and a two-stage segmentation strategy. It has the advantages of high precision, strong generalization and high parameter efficiency, and can be widely used for intraoperative assistance in various endoscopic surgeries.

[0098] In summary, this invention achieves: Precise semantic guidance: Through an innovative semantic adapter, language priors are injected early in the visual feature extraction process and at different levels, enabling the model to fundamentally perceive the concept of "bleeding" and effectively solving the semantic ambiguity problem caused by color similarity.

[0099] High segmentation accuracy: A two-stage strategy of "coarse segmentation with prior segmentation head + fine segmentation with SAM2" is adopted. First, the target region is locked using semantic information, and then the powerful boundary generation capability of SAM2 is used for refinement. While maintaining a high recall rate, pixel-level precision segmentation boundary is obtained.

[0100] Efficient parameters and strong generalization: Only lightweight adapters, prior segmenters and a small number of projection parameters are trained, freezing most of the weights of CLIP, DINOv2 and SAM2 backbone networks. This greatly reduces training costs and overfitting risks while fully inheriting the powerful generalization ability of the base model.

[0101] Flexible application: By modifying the input natural language prompt text (e.g., changing "red bleeding area" to "skin lesion area"), this method can be quickly transferred to other medical image segmentation tasks, demonstrating good scalability.

[0102] Experimental Results and Analysis: (1) Dataset Currently, publicly available surgical bleeding detection datasets are scarce, and there are no datasets specifically designed for spinal endoscopy scenarios. Unlike surgical data collected in an air environment, spinal endoscopy is typically performed in a liquid environment, leading to significant differences in the visual characteristics and imaging appearance of bleeding. To support the development and evaluation of the method in this invention, a novel spinal endoscopic bleeding segmentation dataset—SeBlood—was constructed. All images were carefully selected from over 80 videos of minimally invasive spinal endoscopic surgeries to ensure diversity and representativeness.

[0103] The dataset contains 1421 images, with 1136 used for training and 285 for testing. Each image is finely annotated at the pixel level to delineate the bleeding area. The annotation work was completed collaboratively by three experienced spine surgeons, and ambiguous examples were cross-validated to ensure consistency and accuracy. Notably, the SeBlood dataset includes both bleeding points (images capturing the initial occurrence of bleeding) and bleeding regions (images showing the extent of bleeding after it has spread), covering different stages of intraoperative bleeding.

[0104] The SeBlood dataset presents a significant challenge for automated hemorrhage segmentation. First, the morphology of the hemorrhage region varies considerably, and its boundaries are often blurred due to optical artifacts. Second, the color of the hemorrhage is highly similar to that of the surrounding tissue, especially in cases of old hemorrhage or coagulated tissue. These factors make SeBlood a challenging benchmark dataset for clinical segmentation.

[0105] (2) Implementation details In the experiments of this invention, SGBA-Net employed a pre-trained CLIP text encoder and a DINOv2-L / 14 visual encoder, with lightweight adapters inserted at layers 2, 6, 10, 14, 18, and 22 of the DINOv2-L to achieve cross-modal fusion. The input images were uniformly resized to 448×448, and the text prompt was set to "red bleeding region". The AdamW optimizer was used during training, with an initial learning rate of 0.0001, which decayed to 0.1 at the 35th epoch. The batch size was set to 32, and the total number of training epochs was 50. The experiments were conducted on three RTX 3090 (24GB) GPUs.

[0106] Model performance was evaluated using three metrics: mIoU, Dice, and P@50. mIoU measures the average intersection-union ratio (IU) between the predicted and ground truth segmentation masks; Dice measures the degree of overlap between the predicted and ground truth masks; and P@50 evaluates the accuracy of the predicted region.

[0107] (3) Performance on the SeBlood benchmark dataset To validate the effectiveness of SGBA-Net, comprehensive experiments were conducted on the newly constructed SeBlood benchmark dataset, and comparisons were made with various state-of-the-art segmentation methods, including the classic semantic segmentation architectures U-Net, U-Net++, and DeepLabv3+; the diffusion-based Seg U-KAN; the SAM-based medical segmentation method MedSA; and various referential segmentation models such as Restr, ETRIS, RefSeg, and DETRIS. To ensure fairness, all methods were implemented using their official code and evaluated under the same experimental settings.

[0108] 1) Quantitative comparison As shown in Table 1, SGBA-Net achieved state-of-the-art results across all evaluation metrics. Compared to traditional CNN architectures (such as U-Net++), SGBA-Net significantly improved mIoU from 59.97% to 77.61%, an improvement of 17.64%. This substantial performance leap demonstrates that relying solely on visual texture features is insufficient to address the high similarity between hemorrhage areas and background tissues in spinal endoscopy, making the introduction of semantic priors crucial. Furthermore, compared to the referential segmentation method RefSeg, SGBA-Net still maintains a significant advantage of 6.49% in mIoU (77.61% vs 71.12%), and also improves the Dice metric by 6.61%. This proves that the proposed "semantic adapter" and "coarse-to-fine" SAM2 guidance strategy can more effectively utilize language cues to locate ambiguous targets than traditional cross-modal fusion mechanisms.

[0109] Table 1. Quantitative comparison with existing methods on the SeBlood test set.

[0110] It is worth noting that SGBA-Net also outperforms MedSA (mIoU 70.01%), which is specifically optimized for medical images. Although MedSA also utilizes the segmentation capabilities of SAM, it lacks explicit textual semantic guidance and relies solely on visual adapters for fine-tuning. SGBA-Net's advantage lies in its explicit construction of "text-visual" alignment, demonstrating that in endoscopic scenes with severe color interference, verbal guidance is more discriminative than simple visual fine-tuning.

[0111] This invention further divides the test set into two categories for analysis: "bleeding points" and "bleeding regions".

[0112] Tiny Target Detection Capability (Bleeding Points): Bleeding points typically refer to minute bleeding in the early stages of surgery. These targets are extremely small and their features are not readily apparent, making detection challenging. In this scenario, SGBA-Net demonstrates superior sensitivity, achieving a mIoU of 69.07%, far exceeding MedSA (60.21%) and DeepLabv3+ (47.16%). Other models often overlook these tiny targets, leading to missed detections. SGBA-Net, however, leverages the textual features of CLIPs to identify tiny regions semantically consistent with "red bleeding" in the early encoding stage, significantly improving the recall rate for small targets.

[0113] Boundary integrity and accuracy (bleeding area): For diffuse, large-area bleeding areas, the challenge of segmentation lies in accurately depicting irregular and blurred boundaries. Thanks to the powerful mask decoding capabilities of SAM2, SGBA-Net achieved an impressive 98.19% Dice score and 96.89% P@50 on this subset. In comparison, although Seg U-KAN introduces the KAN mechanism, it still lags behind the method of this invention in boundary handling (Dice 82.19%). This demonstrates that SGBA-Net can not only "find" the target but also generate "pixel-perfect" masks that fit the anatomical structure, effectively overcoming the boundary blurring problem caused by blood diffusion in liquid environments.

[0114] 2) Qualitative comparison To intuitively evaluate SGBA-Net's perception capabilities in real surgical scenarios, this invention divides the test set samples into three typical challenging scenarios: microbleeds, patchy bleeding, and diffuse bleeding. Figure 6 This paper presents a visual comparison of representative samples from the SeBlood dataset and their segmentation results. The qualitative results further corroborate the conclusions of the quantitative experiments.

[0115] Microbleed detection (a): Microbleeds in the early stages of surgery typically occupy only a few pixels and are often lost in complex background textures, making them highly susceptible to false negatives. For example... Figure 6 As shown in (a), both the classic U-Net++ and the Transformer-based Seg U-KAN failed to identify the weak bleed signal, mainly due to the loss of small target features caused by multiple downsampling. In contrast, SGBA-Net successfully located the tiny region. This is thanks to the semantic adapter designed in this invention, which injects the linguistic prior of "red bleed" early in the feature extraction stage, significantly enhancing the model's sensitivity to weak color features and maintaining high recall even when the target is extremely small.

[0116] Delineating the boundaries of patchy hemorrhages (b): For irregularly shaped patchy hemorrhages, the challenge lies in accurately approximating the hemorrhage edges. From Figure 6 As can be observed in (b), although MedSA and DETRIS can locate the bleeding area, their predicted mask edges are either smooth or coarse, failing to capture irregular anatomical boundary details (i.e., undersegmentation). In contrast, the mask generated by SGBA-Net exhibits high spatial consistency with the ground truth (GT). This is because SGBA-Net incorporates the mask decoder of SAM2, utilizing high-resolution visual features to refine the coarse prior generated by PSH, thereby achieving pixel-level boundary fitting.

[0117] Maintaining the integrity of diffuse hemorrhage (c): Diffuse hemorrhage appears "cloudy" in the underwater environment, with blurred and semi-transparent boundaries, which easily leads to fragmentation of the segmentation results. For example... Figure 6 As shown in (c), the prediction results of Seg U-KAN and U-Net++ exhibit obvious breaks and holes, incorrectly segmenting the continuous bleeding cloud into multiple fragments; while DETRIS tends to include excessive background noise. In contrast, SGBA-Net generates complete and coherent segmented regions, effectively covering the blurred diffusion edges. This indicates that the global textual semantics introduced by CLIP effectively guides the model to understand the integrity of the "bleeding region," avoiding semantic ambiguity caused by unclear local textures.

[0118] These visual results demonstrate that by introducing linguistic cues to guide visual feature extraction and combining it with SAM2's coarse-to-fine segmentation strategy, SGBA-Net can effectively address common challenges in spinal endoscopy imaging, such as high color similarity and unclear bleeding boundaries, thereby achieving higher-quality bleeding region segmentation.

[0119] Example 2 This embodiment discloses a semantically guided endoscopic hemorrhage image segmentation system, including: The data acquisition module is configured to acquire intraoperative endoscopic images and corresponding natural language prompts. The multimodal feature extraction module is configured to extract features from the endoscopic image and the natural language prompt text respectively to obtain hierarchical visual features and semantic features; the semantic features include word-level semantic features and global semantic features. A multimodal feature fusion module is configured to inject the semantic features into the extraction process of the hierarchical visual features through a semantic adapter to obtain semantically enhanced visual features; The prior segmentation module is configured to generate a coarse segmentation mask for the bleeding region based on the semantically enhanced visual features and the semantic features through the prior segmentation head. The prompt generation module is configured to: construct a multimodal prompt set including bounding box prompts, mask prompts, and language prompts based on the coarse segmentation mask and the semantic features; The fine segmentation module is configured to: input the endoscopic image, the semantically enhanced visual features, and the multimodal cue set into the segmentation base model, perform fine segmentation through its mask decoder, and output the final bleeding region segmentation mask.

[0120] Example 3 This embodiment illustrates the application of the present invention in a gastrointestinal endoscopy setting. The main difference from Embodiment 1 is: The input images are from videos of gastrointestinal endoscopy examinations.

[0121] The natural language prompt text T can be set to "active bleeding area in the gastrointestinal tract" or "dark red blood clot" depending on the specific situation.

[0122] The training data needs to be replaced with a labeled dataset of bleeding from gastrointestinal endoscopy.

[0123] The framework of this invention requires no structural modifications and only minor adjustments on a new dataset to achieve effective segmentation of the bleeding region in this scenario, demonstrating the good domain adaptability of this invention.

[0124] Example 4 The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of Embodiment 1.

[0125] Example 5 The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method of Embodiment 1.

[0126] The steps and methods involved in the apparatuses of Embodiments 4 and 5 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0127] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0128] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0129] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A semantically guided endoscopic hemorrhage image segmentation method, characterized in that, include: Acquire intraoperative endoscopic images and corresponding natural language prompts; Feature extraction is performed on the endoscopic images and natural language prompts to obtain hierarchical visual features and semantic features; the semantic features include word-level semantic features and global semantic features. The semantic features are injected into the extraction process of the hierarchical visual features through a semantic adapter to obtain semantically enhanced visual features; The semantic adapter includes a multi-scale convolution module and a multi-head cross-attention fusion module; Based on the multi-scale convolution module, the input hierarchical visual features are extracted and fused with features from different receptive fields to obtain enhanced spatial features; The multi-head cross-attention fusion module uses global semantic features as keys and values, and enhanced spatial features as queries to perform feature alignment and fusion, thereby obtaining semantically enhanced visual features. After selection and output, multi-level semantically enhanced visual features are obtained. Based on the semantically enhanced visual features and the semantic features, a coarse segmentation mask for the bleeding region is generated using a priori segmentation head; The prior segmentation head includes a global context prior module, a local semantic alignment module, and an upsampling projector. The global context prior module integrates multi-level semantically enhanced visual features and global semantic features, and introduces coordinate attention enhancement to obtain spatial perception features. The local semantic alignment module performs fine-grained alignment between spatial awareness features and word-level semantic features to obtain text-aligned enhanced visual features; The upsampling projector upsamples the text alignment enhancement visual features and maps them to generate a coarse segmentation mask. Based on the coarse segmentation mask and the semantic features, a multimodal cue set including bounding box cue, mask cue, and language cue is constructed; The endoscopic image, the semantically enhanced visual features, and the multimodal cue set are input into the segmentation base model, and refined segmentation is performed through its mask decoder to output the final bleeding region segmentation mask.

2. The semantically guided endoscopic hemorrhage image segmentation method as described in claim 1, characterized in that, Construct a multimodal tooltip set that includes bounding box hints, mask hints, and language hints, specifically as follows: Extract the bounding box from the coarse segmentation mask as a bounding box cue; Use the coarse segmentation mask as a mask hint; The global semantic features are mapped to the embedding space of the prompt encoder of the segmentation base model to obtain language prompts.

3. The semantically guided endoscopic hemorrhage image segmentation method as described in claim 1, characterized in that, The segmentation-based model employs a combined loss: in, , , , To balance the weight hyperparameters, For the visual contrast loss of text in coarse segmentation masks, The text visual contrast loss for the final segmentation mask, For the Dice loss of the coarse segmentation mask, The Dice loss is used for the final segmentation mask.

4. A semantically guided endoscopic hemorrhage image segmentation system, characterized in that, include: The data acquisition module is configured to acquire intraoperative endoscopic images and corresponding natural language prompts. The multimodal feature extraction module is configured to extract features from the endoscopic image and the natural language prompt text respectively to obtain hierarchical visual features and semantic features; the semantic features include word-level semantic features and global semantic features. A multimodal feature fusion module is configured to inject the semantic features into the extraction process of the hierarchical visual features through a semantic adapter to obtain semantically enhanced visual features; The prior segmentation module is configured to generate a coarse segmentation mask for the bleeding region based on the semantically enhanced visual features and the semantic features through the prior segmentation head. The prompt generation module is configured to: construct a multimodal prompt set including bounding box prompts, mask prompts, and language prompts based on the coarse segmentation mask and the semantic features; The fine segmentation module is configured to: input the endoscopic image, the semantically enhanced visual features, and the multimodal cue set into the segmentation base model, perform fine segmentation through its mask decoder, and output the final bleeding region segmentation mask; The semantic adapter includes a multi-scale convolution module and a multi-head cross-attention fusion module; Based on the multi-scale convolution module, the input hierarchical visual features are extracted and fused with features from different receptive fields to obtain enhanced spatial features; The multi-head cross-attention fusion module uses global semantic features as keys and values, and enhanced spatial features as queries to perform feature alignment and fusion, thereby obtaining semantically enhanced visual features. After selection and output, multi-level semantically enhanced visual features are obtained. The prior segmentation head includes a global context prior module, a local semantic alignment module, and an upsampling projector. The global context prior module integrates multi-level semantically enhanced visual features and global semantic features, and introduces coordinate attention enhancement to obtain spatial perception features. The local semantic alignment module performs fine-grained alignment between spatial awareness features and word-level semantic features to obtain text-aligned enhanced visual features; The upsampling projector upsamples the text alignment enhancement visual features and maps them to generate a coarse segmentation mask.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the semantically guided endoscopic hemorrhage image segmentation method as described in any one of claims 1-3.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the semantically guided endoscopic hemorrhage image segmentation method as described in any one of claims 1-3.

Images