Visual language model-based optic disk atrophy weak supervision segmentation method

Abstract

The invention discloses an optic disc atrophy arc weakly supervised segmentation method based on a visual language model, and belongs to the field of fundus retina image processing. The method comprises the following steps: constructing a bimodal encoder comprising a visual feature encoder and a text encoder; a visual language multi-modal feature alignment mechanism is introduced, visual representation learning is guided by using priori knowledge in medical text description, cross-modal association of images and lesion semantics is established in a unified feature space, and accurate alignment of visual and text features is realized; generating a focus attention heat map based on the alignment features, designing an adaptive multi-channel prompt generation network, and automatically mining high-confidence positive and negative prompt points from the heat map; and taking the generated prompt point as a position guide input segmentation basic model SAM, and realizing fine extraction of the optic disk atrophy arc boundary under the weak supervision condition of only text description by utilizing the strong zero sample segmentation capability of the SAM.

Description

Technical Field

[0001] This invention belongs to the field of fundus retinal image processing and relates to a weakly supervised segmentation method for optic disc atrophy arc based on a visual language model. Background Technology

[0002] Peripapillary atrophy (PPA) is an important pathological feature of fundus diseases such as glaucoma and high myopia, characterized by degenerative changes in the retinal pigment epithelium and choroid around the optic disc. Its morphological features (such as the area of ​​atrophy, border morphology, and the extent of the beta zone) are closely related to the occurrence and progression of the disease. Therefore, accurate segmentation of the PPA is of crucial clinical significance for early glaucoma screening, disease monitoring, and personalized treatment planning. However, traditional clinical diagnosis relies on doctors manually delineating lesion areas in fundus images, which is inefficient, subjective, and has poor repeatability. Automated segmentation techniques are urgently needed to assist in diagnosis.

[0003] Currently, automatic segmentation methods for optic disc atrophy arcs mainly fall into two categories: traditional image processing techniques and fully supervised deep learning models. Traditional methods are mostly based on edge detection, region growing, or threshold segmentation, combined with prior shape constraints such as ellipse fitting. These methods rely on manually designed features and are extremely sensitive to image noise, uneven illumination, and contrast variations, making it difficult to handle situations where the atrophy arc has blurred boundaries and varied shapes. While fully supervised deep learning models (such as U-Net and its variants) significantly outperform traditional methods, their training heavily relies on large-scale, high-quality pixel-level labeled data. In fundus image analysis, obtaining pixel-level annotations not only requires a significant investment of time from professional ophthalmologists, but also, due to the complex spatial relationships between the PPA region and structures such as the optic disc and retinal vessels, coupled with the progressively blurred transitions of lesion boundaries, annotations from different doctors, and even from the same doctor at different times, often exhibit significant differences. This high cost of data acquisition and inconsistency in annotation severely restricts the generalization ability and practical deployment of fully supervised models in clinical settings.

[0004] To address the aforementioned issues, weakly supervised learning (WL) techniques have gradually become a research hotspot, aiming to achieve segmentation using low-cost annotations such as image-level labels, bounding boxes, or sparse points. However, existing WL segmentation methods still face significant challenges when directly applied to optic disc atrophy arcs: on the one hand, existing methods based on class activation mapping (CAM) often generate localization maps with low resolution and coarse boundaries, making it difficult to capture the fine edge structures of the PPA; on the other hand, simple attention mechanisms struggle to effectively model the complex multi-level contextual dependencies between the PPA and anatomical structures such as the optic disc and blood vessels, easily leading to topological errors in the segmentation results (such as holes or breaks). Therefore, how to achieve high-precision and robust segmentation of optic disc atrophy arcs using only low-cost weakly supervised signals combined with prior clinical semantic knowledge is a pressing technical challenge that needs to be addressed. Summary of the Invention

[0005] To address the problems of existing optic disc atrophy arc (PPA) segmentation methods' high dependence on costly pixel-level annotations and the inaccuracy and insufficient generalization ability of traditional weakly supervised methods in lesion boundary identification, the present invention aims to provide a weakly supervised segmentation method for optic disc atrophy arc based on a visual language model. This method can utilize low-cost weakly supervised signals and combine them with prior clinical semantic knowledge to achieve high-precision and robust segmentation of the optic disc atrophy arc.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] This invention discloses a weakly supervised segmentation method for optic disc atrophy arcs based on a visual language model, employing a framework of "semantic feature alignment - adaptive cue generation - fine segmentation of the base model". First, a bimodal encoder comprising a visual feature encoder and a text encoder is constructed, introducing a visual-language multimodal feature alignment mechanism. Prior knowledge from medical text descriptions (such as lesion location and morphological features) guides the learning of visual features, achieving cross-modal semantic association between the image and the lesion in the feature space. Second, a lesion attention heatmap is generated based on the aligned features, and an adaptive multi-channel cue generation network is designed to automatically mine high-confidence positive cue points (lesion area) and negative cue points (background area) from the heatmap. Finally, the generated cue points are used as location-guided input to the segmentation base model (Segment Anything Model, SAM). Leveraging the powerful zero-shot segmentation capability of SAM, fine extraction of the optic disc atrophy arc boundary is achieved under weak supervision with only text.

[0008] This invention discloses a weakly supervised segmentation method for optic disc atrophy arcs based on a visual language model, comprising the following steps:

[0009] S1: Construct a dual-modal encoder that includes a visual feature encoder and a text encoder; configure the visual encoder to extract multi-scale visual features from fundus images, and configure the text encoder to extract textual semantic features of optic disc atrophy arc lesions, thus establishing the foundation for feature extraction.

[0010] S2: The bimodal encoder is fine-tuned by comparative learning using fundus image and text datasets, so that the visual features and text features are semantically aligned in a unified feature space, and the fine-tuned bimodal encoder is obtained.

[0011] S3: Construct an adaptive multi-channel prompt generation network; use the fine-tuned bimodal encoder to generate self-supervised pseudo-labels based on lesion attention heatmaps, and train the network to enable it to map the coordinates of prompt points from the heatmap, thus obtaining the trained prompt generation network.

[0012] S4: Acquire the fundus image to be tested, calculate the similarity matrix between the image and the preset text features using the fine-tuned bimodal encoder, and generate a lesion attention heatmap; input the heatmap into the prompt generation network trained in step S3, and output segmentation prompt points.

[0013] S5: Input the fundus image to be tested and the segmentation prompt points into the segmentation base model. Use the prompt points as position guides to guide the segmentation base model to extract the boundary of the optic disc atrophy arc, so as to achieve automatic segmentation of the optic disc atrophy arc under weak text supervision.

[0014] Furthermore, the method for constructing the visual feature encoder in S1 is specifically as follows:

[0015] A visual Transformer is used as the frozen backbone network; the input fundus image is... Divided into Each size is Image blocks; each image block is mapped to a linear projection layer. The dimensional embedding vector, after being added with position encoding, is input into the Transformer layer; and a visual adapter is inserted in parallel between every two Transformer encoding blocks of the visual Transformer to obtain the visual feature encoder.

[0016] Furthermore, the method for constructing the text encoder in S1 is specifically as follows:

[0017] A pre-trained large-scale language model is used as the frozen backbone network; the input visual disc atrophy arc (PPA) related text description information is segmented; each segmented word is mapped to a D-dimensional embedding vector through an embedding layer to form a text vector sequence, which is then input into the Transformer encoding layer of the backbone network; and a text adapter is inserted between several Transformer layers of the language model to obtain the text encoder.

[0018] Furthermore, the specific method for establishing and fine-tuning the cross-modal feature alignment mechanism in S2 is as follows:

[0019] By constructing a comprehensive loss function Image features obtained in step S1 and text features Joint optimization training is performed, and the comprehensive loss function consists of global feature alignment loss, local feature alignment loss, cross-level consistency loss, and a regularization term. The specific calculation steps are as follows:

[0020] Step S2.1: Construct the global feature alignment loss At the overall level, the similarity of positive sample image-text pairings is improved through contrastive learning, and the loss function is defined as:

[0021]

[0022] in, Batch size; For the first The global feature vector of an image; For the first The positive sample text feature vectors corresponding to each image; For any number within the batch One text feature vector; Represents the cosine similarity function; This refers to temperature hyperparameters.

[0023] Step S2.2: Construct the local feature alignment loss At the local level, a multi-instance learning strategy is employed to perform weakly supervised matching between local image features and text vocabulary features. Since pixel-level annotation is not used, the loss function automatically selects the region with the highest response value in the image as a potential lesion area for optimization, defined as:

[0024]

[0025] in, A set of text terms describing PPA lesions; For position Visual feature vector at the location; as a word element The text feature vector; The image feature map with the highest similarity to the lesion text The loss function is a set of Top-K Regions in spatial locations; it guides the model to automatically focus on and locate lesion regions by maximizing the similarity between high-response regions and lesion text.

[0026] Step S2.3: Construct cross-level consistency loss At the cross-layer constraint level, to minimize the difference between the aligned distributions of shallow and deep features, the loss function is defined as:

[0027]

[0028] in, The set of network layers participating in the alignment; For the first Similarity distribution between layer visual features and text features; This serves as a reference similarity distribution; The Kullback-Leibler divergence; For the first Layer weight coefficients;

[0029] Step S2.4: Calculate the comprehensive loss function

[0030]

[0031] in, Let be the weight hyperparameter, satisfying ; This is the 2-norm regularization term for the adapter parameter set.

[0032] Furthermore, the construction and training method of the adaptive multi-channel prompt generation network in S3 is as follows:

[0033] Step 3.1: Constructing the Network Structure The adaptive multi-channel cue generation network adopts an encoder-decoder structure, and the input is the single-channel attention heatmap used for training. This heatmap comes from step S2.2. The output of the adaptive multi-channel cue generation network is a dual-channel cue probability map. The dual-channel prompt probability map includes a positive prompt probability map. Negative hint probability graph These correspond to the lesion boundary region and the high-confidence background region predicted by the model, respectively, and achieve collaborative learning of boundary recognition and background suppression within the same network structure.

[0034] Step 3.2: Construct self-supervised pseudo-labels using image processing algorithms from attention heatmaps Automatically generate pseudo-tag sets Attention heatmap Calculate gradient magnitude plots using gradient operators such as Sobel or Scharr. and to Thresholding and morphological skeletonization are performed to refine the boundary contour, which is then used as a positive cue pseudo-label. ; heatmap Normalization and inversion are performed to obtain the background confidence map. ,right Thresholding is performed, and high-confidence regions are retained as negative cue pseudo-labels. ;

[0035] Step 3.3: End-to-end network training to construct a dual-channel composite loss function containing positive and negative weights. Optimize the network:

[0036]

[0037] in, Selected from Dice loss or Focal Loss to alleviate the problem of foreground and background pixel imbalance; and These are the weighting coefficients;

[0038] Furthermore, the specific implementation method for generating the lesion attention heatmap and the cue points in S4 is as follows:

[0039] Step 4.1: Extract bimodal features from the inference stage, and analyze the fundus image to be tested. The visual encoder, which is fine-tuned in step S2, is input to the visual encoder of the dual-modal encoder to extract local visual feature maps. ,in The spatial resolution of the feature map is set; simultaneously, the preset target text prompt of the visual disc atrophy arc is input into the text encoder of the dual-modal encoder after fine-tuning in step S2 to obtain the corresponding global text feature vector. ;

[0040] Step 4.2: Calculate the image-text feature similarity matrix and calculate the text feature vector. With the local visual feature map Features of various spatial locations The cosine similarity between them yields a two-dimensional similarity matrix. :

[0041]

[0042] Among them, the two-dimensional similarity matrix Each position in the matrix The value indicates the degree of correlation between the local area of ​​the image and the preset semantic meaning of "optic disc atrophy arc". The higher the value, the higher the probability that the area is a PPA lesion.

[0043] Step 4.3: Generate a normalized attention heatmap, and then convert the two-dimensional similarity matrix... Upsampled to the same resolution as the input image The data was then normalized to obtain the final lesion attention heatmap. This heatmap visually reflects the spatial distribution probability of PPA lesions, and is used to achieve coarse localization of the lesion area.

[0044] Step 4.4: Cue point generation, generating the attention heatmap of the image to be tested. Input the trained network to obtain dual-channel output. ; Positive cue probability diagram in dual-channel output Perform non-maximum suppression (NMS) and Top-K filtering to select the response with the highest value. Each point serves as a positive cue point; the negative cue probability diagram in the dual-channel output. Perform nonmaximum suppression and Top-K filtering to select... A high-confidence background point is used as a negative cue point.

[0045] Furthermore, the specific implementation method for pixel-level fine segmentation using cue points in S5 is as follows:

[0046] Step 5.1: Constructing Multimodal Input Feature Embeddings. The preprocessed fundus image from Step S1 is input into the image encoder of the segmentation base model SAM to map the high-resolution image into low-resolution image feature embeddings. Simultaneously, the positive and negative cue point sets extracted in step S4 are input into the cue encoder of the segmentation base model; the spatial coordinate information of the cue points is mapped into sparse cue feature embeddings using learnable positional encoding. Positive cue points are encoded as foreground features, and negative cue points are encoded as background features, in order to distinguish lesion and non-lesion areas in the feature space.

[0047] Step 5.2: Feature Fusion and Mask Decoding: Embedding the Image Features and cue feature embedding The features are jointly input into the mask decoder of the segmentation base model for feature fusion and interaction, and the model outputs a predicted lesion probability map.

[0048] Step 5.3: To generate a binary segmentation mask, the probability map is processed with the Sigmoid activation function, and then converted into a binary mask by setting a probability threshold; the connected regions with a pixel value of 1 in the binary mask are the final predicted PPA lesion regions of the optic disc atrophy arc; the lesion segmentation result with fine edge structure is output by locating the main body of the lesion through positive prompts and suppressing background noise through negative prompts.

[0049] Beneficial effects:

[0050] 1. This invention discloses a weakly supervised segmentation method for optic disc atrophy arcs based on a visual language model. By introducing a weakly supervised learning mechanism, model training can be driven solely by image-level labels or simple textual semantic descriptions, eliminating the need for time-consuming and labor-intensive pixel-level manual annotation. This significantly reduces the reliance on expert annotation and the associated labor costs in medical image segmentation tasks. This invention not only greatly reduces the annotation workload of ophthalmologists but also improves the scalability of the weakly supervised segmentation method for optic disc atrophy arcs in scenarios where medical data is scarce.

[0051] 2. This invention discloses a weakly supervised segmentation method for optic disc atrophy arcs based on a visual language model. Through deep fusion of visual and linguistic multimodal information, it enhances the model's semantic understanding of lesion features. This invention utilizes the cross-modal alignment mechanism of a bimodal encoder to implicitly inject clinical prior knowledge from medical texts (such as "atrophy arc," "ring boundary," etc.) into the visual feature representation. This semantic enhancement mechanism enables the model to overcome difficulties such as low contrast and complex backgrounds in fundus images under low-annotation conditions, achieving adaptive understanding and recognition of specific lesion features.

[0052] 3. This invention discloses a weakly supervised segmentation method for optic disc atrophy arcs based on a visual language model, establishing a multi-level semantic alignment and consistency constraint mechanism to achieve accurate lesion localization across scales. By jointly optimizing global alignment loss, local multi-instance learning loss, and cross-level consistency loss, this invention forces the model to establish a unified semantic feature space across different network layers. This enables the model to automatically achieve scale-adaptive lesion region localization without pixel-level supervision by calculating the similarity distribution between visual features and text semantics, ensuring the accuracy of feature extraction.

[0053] 4. This invention discloses a weakly supervised segmentation method for optic disc atrophy arcs based on a visual language model. By constructing an adaptive multi-channel cue generation mechanism, it can improve the accuracy and robustness of weakly supervised segmentation. The cue generation network constructed in this invention can automatically generate positive cues (foreground) and negative cues (background) containing positional information based on attention heatmaps, and effectively suppress high-confidence background noise using negative cues. Combined with a segmentation base model (such as SAM), it can solve the oversegmentation and undersegmentation problems that are prone to occur in traditional weakly supervised methods, and significantly improve the ability to capture the blurred boundaries of optic disc atrophy arcs. Attached Figure Description

[0054] Figure 1 A flowchart of a weakly supervised segmentation method for optic disc atrophy arc based on a visual language model according to the present invention. Detailed Implementation

[0055] To better illustrate the purpose and advantages of the present invention, the invention will be further described below in conjunction with the accompanying drawings and examples.

[0056] Example 1:

[0057] like Figure 1 As shown in the figure, this embodiment discloses a weakly supervised segmentation method for optic disc atrophy arc based on a visual language model. The specific implementation steps are as follows:

[0058] This embodiment elaborates on step S1: constructing a dual-modal encoder that includes visual and text encoders; configuring the visual encoder to extract multi-scale visual features from fundus images, configuring the text encoder to extract textual semantic features of optic disc atrophy arc lesions, and establishing the foundation for feature extraction.

[0059] Step 1.1 Standardized preprocessing of fundus images.

[0060] To eliminate differences in illumination, contrast, and resolution in the original fundus images and to build a high-quality foundation for feature extraction, this embodiment follows the steps below to process the original fundus images. Preprocessing:

[0061] Step 1.1.1 Image Enhancement: Obtain the original color fundus image. Given that fundus images often exhibit uneven illumination, this embodiment employs the Limiting Contrast Adaptive Histogram Equalization (CLAHE) algorithm to enhance the image. Specific parameters are set as follows: the contrast limit threshold (Clip Limit) is set to 2.0, and the tile grid size is set to... This step effectively enhances the contrast between the optic disc atrophy arc (PPA) region and the surrounding retinal background.

[0062] Step 1.1.2 Visual disc localization and ROI extraction: The Hough Circle Transform is used to detect the visual disc region in the enhanced image, and the point with the highest response value is selected as the center coordinate of the visual disc. and obtain the viewing radius. To ensure complete coverage of the periphery atrophy area (including...) District and (Region), set the clipping expansion factor for the region of interest (ROI). That is to say, Centered on, the side length to be cut is A square area.

[0063] Step 1.1.3 Size Normalization: The cropped ROI region is scaled to the standard input resolution set in this embodiment, i.e., height, using a bilinear interpolation algorithm. ,width The image was then normalized to pixel values ​​(standardized to the [0, 1] interval) to obtain the preprocessed fundus image. .

[0064] Step 1.2. Construction of the visual encoder.

[0065] In this embodiment, the visual encoder is built on the Transformer architecture and aims to extract multi-scale visual features containing global semantics.

[0066] Step 1.2.1 Backbone Network Configuration: In this embodiment, a pre-trained ViT-B / 16 (VisionTransformer Base) is used as the backbone network. This network contains 12 Transformer encoding layers. To preserve the large-scale visual prior knowledge of the pre-trained model, the weight parameters of the backbone network are frozen during fine-tuning, preventing them from participating in gradient updates. In the specific Transformer network patch embedding process, the input image... Divided into sizes Non-overlapping image patches, where The resulting image patch sequence has a length of [length missing]. For the input resolution of this experiment, the sequence length is... Subsequently, after passing through a linear projection layer, each image patch is mapped to a dimension. The feature embedding vector.

[0067] Step 1.2.2 Construction and Operation of the Visual Adapter: To adapt the frozen backbone network to the optic disc shrinkage segmentation task, a visual adapter is inserted in parallel between every two coding blocks of the visual Transformer. The bottleneck dimension of the adapter is set as follows: The initial value of the learnable scaling factor is set to .

[0068] The characteristic of input to the adapter is: (i.e., the first) The specific computation steps of the visual adapter (output of the Transformer layer) are as follows:

[0069] (1) Dimensionality reduction projection: using the weight matrix High-dimensional features Mapping to a low-rank space yields intermediate features. The calculation formula is:

[0070]

[0071] At this point, the feature dimension was compressed from 768 to 64, achieving parameter efficiency.

[0072] (2) Nonlinear activation: For intermediate features Apply The activation function introduces a nonlinear feature transformation, resulting in... :

[0073]

[0074] (3) Dimensional Upgrading Projection: Using the weight matrix Activated features Restored to the original feature dimensions, resulting in :

[0075]

[0076] (4) Residual scaling connection: Convert the adapter output characteristics Scaling factor After weighting, the original output features of the backbone network are compared. By adding element by element, we get the first... The final output of the layer :

[0077]

[0078] Step 1.2.3 Feature Recombination and Output:

[0079] Take the output sequence of the last layer of the visual encoder (dimension: The classification tokens (CLSTokens) are removed and transformed into spatial feature maps through a reshape operation. .

[0080] The final local visual feature map The height of the feature map ,width Number of channels This feature map integrates the spatial information of the original image with the semantic information of the lesions learned through adapter fine-tuning, providing a feature foundation for the subsequent generation of high-precision lesion attention heatmaps.

[0081] Step 1.3. Construction of the text encoder and adapter module.

[0082] This step aims to build a text encoder capable of processing medical text descriptions related to peripapillary atrophy (PPA) and aligning them with visual features. The specific implementation process is as follows:

[0083] Step 1.3.1 Text preprocessing and embedding.

[0084] First, textual descriptions of paradiscal atrophy (PPA) corresponding to fundus images are obtained. A tokenizer is then used to segment the text descriptions, converting the natural language sentences into discrete token sequences. Subsequently, a learnable embedding layer maps each token to a dimensional [value missing]. Dense vectors form a text vector sequence. ,in For the sequence length, this embodiment truncates or pads the maximum text sequence length. , For the embedding dimension (e.g.) (Or determined based on the selected large model), ultimately forming a sequence of text vectors. .

[0085] Step 1.3.2 Freeze the backbone and extract features.

[0086] A pre-trained large-scale language model (such as the Qwen or LLaMA series) is used as the backbone network for feature extraction. This backbone network contains... A stacked Transformer encoding block. During training, to preserve the general semantic understanding capabilities of large-scale language models and reduce computational overhead, we freeze all original weight parameters of the backbone network, preventing them from participating in gradient updates. (Text vector sequence) The code passes through each Transformer encoding block in turn, and uses its multi-head self-attention mechanism to capture long-distance semantic dependencies.

[0087] Step 1.3.3 Embedding and computation of the Text Adapter module.

[0088] To adapt the general language model to specific ophthalmic medical description tasks, we inserted a lightweight text adapter module (TextAdapter) between several Transformer layers of the language model (or after the FFN layer).

[0089] Each text adapter module contains a bottleneck architecture for outputting features from the intermediate layers of the backbone network. Perform a low-rank residual transformation. The specific calculation steps are as follows:

[0090] (1) Dimensional reduction projection: through the dimensionality reduction matrix The input 4096-dimensional feature vector is projected into a low-rank space. In this embodiment, to achieve a balance between parameter count and performance, the rank of the bottleneck layer is set to 1. The 64-dimensional low-dimensional features were calculated. :

[0091]

[0092] at this time, .

[0093] (2) Nonlinear activation: for low-dimensional features Apply a nonlinear activation function (Usually GELU or ReLU is used) to increase the nonlinear expressive power of the model and obtain the activated features. :

[0094]

[0095] (3) Dimensional recovery: through the dimension-upgrading matrix The activated features are restored to their original dimension of 4096 to obtain the adapter's transformed features. :

[0096]

[0097] Note: In this process, only (size ) and (size ) are trainable parameters.

[0098] (4) Residual connection: the output of the adapter Features of the original output of the backbone network (or input features) (Depending on the specific connection method) Element-by-element summation is performed to form the final adapted feature representation. :

[0099]

[0100] in This is a learnable scaling factor (optional) used to adjust the contribution weights of the adapter branches. Through the above structural design, the text adapter module, without disrupting the original language model's pre-trained parameters, only fine-tunes a small number of parameters. and This enables efficient transfer and enhancement of specific semantics in the PPA domain (such as descriptions like "blurred shrinkage arc boundary" or "crescent shape").

[0101] Step 1.3.4 Cross-modal feature alignment.

[0102] After processing through all encoding layers and adapters, the vectors corresponding to specific tokens (such as [EOS] or [CLS]) output by the text encoder are extracted as global text features. Finally, these text features are mapped to the same feature space dimension as the visual encoder (e.g., [image of a visual encoder]) via a one-dimensional linear projection head. This enables semantic alignment of visual and textual features on the same dimension, laying the foundation for subsequent cross-modal fusion and image-text matching.

[0103] This embodiment elaborates on step S2 in detail: using the fundus image and text dataset to perform comparative learning and fine-tuning on the bimodal encoder, forcing the visual features and text features to achieve semantic alignment in a unified feature space, thereby obtaining the fine-tuned bimodal encoder;

[0104] The model was fine-tuned using a pre-constructed fundus image dataset, specifically by calculating the comprehensive loss function. And then backpropagation is performed.

[0105] Step S2.1: Calculate the global feature alignment loss This loss is used to optimize image-text matching at the overall level. During training, the batch size is set. That is, 32 pairs of image and text data are input simultaneously in each iteration. The InfoNCE loss calculation formula is as follows:

[0106]

[0107] At that time, the temperature hyperparameter It was specifically assigned the value 0.07. This value has been experimentally verified to enable the model to focus on positive samples ( At the same time, maintain an appropriate level of rejection for difficult negative samples to avoid an overly smooth or sharp distribution.

[0108] Step S2.2: Calculate the local feature alignment loss This loss is used to automatically locate the lesion area. In the calculation formula:

[0109]

[0110] At that time, in order to... This embodiment selects the region most likely to contain the lesion from a set of image patches. The size is set to 50. This means that the algorithm will automatically select the 50 image patch locations (approximately 5% of the total image area) that have the highest similarity to text descriptions such as "shrinkage arc" as salient regions to participate in the loss calculation, thereby achieving accurate localization of lesions without pixel-level annotation.

[0111] Step S2.3: Calculate cross-level consistency loss To constrain the consistency of features between shallow and deep layers, in the formula:

[0112]

[0113] In the middle, the set of network layers participating in alignment Specifically, layers 3, 6, 9, and 12 were selected. These four layers represent the shallow, middle, and deep features of the network, respectively. The weight coefficients of each layer are... The value is set to 0.25 to ensure that semantic information at different depths contributes equally to the final feature representation.

[0114] Step S2.4: Calculate the comprehensive loss function The final optimization objective is derived by weighted summation of the components:

[0115]

[0116] The values ​​for each weight hyperparameter are assigned as follows: Global Loss Weight Local loss weights Consistency loss weight Regularization term weights .

[0117] This weighting strategy ( It prioritizes the semantic alignment of the global and local models while also taking into account the consistency and generalization ability within the model. For adapter parameters Regularization, with coefficients set to (Implicit in the regular optimizer settings, or here as) (part of)

[0118] This embodiment elaborates on step S3 in detail: constructing an adaptive multi-channel prompt generation network; using the fine-tuned bimodal encoder to generate self-supervised pseudo-labels based on lesion attention heatmaps, and training the network to enable it to map the coordinates of prompt points from the heatmap, thus obtaining the trained prompt generation network;

[0119] This step aims to utilize the attention heatmap obtained in step S2 to automatically generate high-quality visual cues (Prompts) without manual pixel-level annotation, providing accurate guidance for subsequent segmentation models.

[0120] Step S3.1: Construct an adaptive multi-channel cue generation network.

[0121] This step builds a lightweight encoder-decoder network (Prompt Generator) specifically designed to refine the coarse attention heatmap.

[0122] The input settings for this step are as follows:

[0123] The network input is the single-channel attention heatmap calculated in step S2.2. To capture the details, first... The original heatmap is upsampled to the same size as the original image using bilinear interpolation. Resolution.

[0124] The network structure is as follows:

[0125] The encoder contains 3 convolutional blocks, each consisting of "3x3 convolution + ReLU + max pooling", with the number of channels being [32, 64, 128] respectively, used to extract structural features of the heatmap.

[0126] The decoder contains three upsampled blocks, which fuse encoder features through skip connections to ultimately recover the final image. Resolution.

[0127] The output is defined as a two-channel convolutional layer that outputs feature maps. The probability diagram for positive prompts output by channel 1. This characterizes the potential boundary of the lesion; channel 2 outputs a negative probability map. This represents the high-confidence background region. This dual-channel design achieves a synergistic mechanism of "boundary sharpening" and "background suppression" within the same network.

[0128] Step S3.2: Construct a self-supervised pseudo-label set.

[0129] To train the network described above, this embodiment utilizes traditional image processing algorithms to analyze the input attention heatmap. Automatically generate pseudo-tag sets The specific parameter assignment and processing flow is as follows:

[0130] (1) Positive hint pseudo-label Generation (Boundary Extraction):

[0131] First, the Sobel gradient operator (with a kernel size of 3) is applied to the normalized attention heatmap to calculate the gradient magnitude map. .

[0132] Next, regarding Perform thresholding. Set a gradient threshold. Pixels with a gradient magnitude greater than 0.4 are initially identified as edge candidate points.

[0133] Finally, a morphological skeletonization algorithm is applied to refine the coarse edges into fine contours of a single pixel width, serving as positive cue pseudo-labels. This forces the network to learn more precise lesion morphology boundaries.

[0134] (2) Negative prompt pseudo-label Generation (determining the background):

[0135] Invert the heatmap and calculate the background confidence map. .

[0136] Set strict background confidence thresholds .Will Regions with a median value greater than 0.75 are marked as defined background regions and used as negative cue pseudo-labels. This high threshold setting ensures the purity of negative samples, preventing the blurred edges of lesions from being misjudged as background.

[0137] Step S3.3: End-to-end network training and loss function assignment

[0138] Constructing a dual-channel composite loss function The prompt generation network is optimized. In this embodiment, Dice Loss is selected as the basic segmentation loss function. This is because it can effectively alleviate the class imbalance problem caused by the scarcity of edge pixels (foreground).

[0139] The loss function is defined as follows:

[0140]

[0141] The specific strategy for assigning weighting coefficients is as follows:

[0142] · Higher weights are given to positive boundary learning. This is because the accuracy of lesion boundaries directly determines the accuracy of clinical measurements, and the number of boundary pixels is far less than that of the background, so they need to be given special attention.

[0143] · Give appropriate weights to background suppression to ensure that the model does not produce too many false positive responses.

[0144] The specific training configuration is as follows: the prompt generation network is trained separately for 50 epochs, with an initial learning rate set to... The Adam optimizer is used. After training, the network can transform blurry attention maps into clear geometric cues (Point / Box or Mask) with structural information for use by subsequent modules.

[0145] This embodiment elaborates on step S4 in detail: Acquire the fundus image to be tested, use the fine-tuned bimodal encoder to calculate the similarity matrix between the image and the preset text features, and generate a lesion attention heatmap; input the heatmap into the prompt generation network trained in step S3, and output segmentation prompt points;

[0146] This step details how to use the trained model to process the fundus image under test and generate visual cue points that can be used for downstream segmentation tasks (such as inputting the SAM model).

[0147] Step S4.1: Extract bimodal features from the inference stage.

[0148] Acquire color images of the fundus of the eye to be tested In inference preprocessing, the image is... Size adjustment And standardize it.

[0149] The processed image is input into the visual branch of the fine-tuned bimodal encoder in step S2. After forward propagation, the local visual feature map output from the last Transformer layer is extracted. Based on the backbone configuration in step 2.1, the spatial resolution of this feature map is... Channel dimension .

[0150] Simultaneously, a specific prompt text was set as: "Peripapillary atrophy area with crescent or irregular shape around the optic disc" (or "Peripapillary atrophy area with crescent or irregular shape" in English). This text was input into a finely tuned text encoder to extract a global text feature vector that characterizes the core semantics of PPA lesions. .

[0151] Step S4.2: Calculate the image-text feature similarity matrix.

[0152] Calculate pixel-by-pixel similarity using the cosine similarity formula. The degree of matching between visual feature vectors and text feature vectors:

[0153]

[0154] The original resolution is obtained as Two-dimensional similarity matrix Positions in the matrix with values ​​close to 1.0 indicate that the region is not only visually significant but also highly consistent with the semantic description of a "shrunken region".

[0155] Step S 4.3: Generate a normalized attention heatmap.

[0156] To obtain pixel-level localization, a bilinear interpolation algorithm is first used to... matrix Upsampled to the original image resolution .

[0157] Subsequently, the values ​​were mapped to Min-Max normalization method. The interval is used to obtain the final attention heatmap of the lesion. .

[0158] In this step, we set a display threshold. It is only used to filter out low-confidence noise when displaying visualizations, but retains the complete floating-point values ​​when inputting into subsequent networks to prevent information loss.

[0159] Step S4.4: Automatic generation of point prompts

[0160] The above heat map Input the trained prompt generation network from step S3 to obtain dual-channel output. (resolution) To extract discrete geometric cue points from the probability map, this embodiment uses the following specific post-processing parameters:

[0161] Step S4.4.1 Extraction of positive cue points (for...) aisle).

[0162] First of all The graph performs non-maximum suppression (NMS). A spatial distance threshold is set. Pixels. That is, within a radius of 15 pixels centered on a high-response point, only the point with the largest value is retained to avoid the prompt points being too densely clustered in the same location.

[0163] Then perform Top-K filtering and set... The five points with the highest response values ​​are selected as positive feedback points. These five points are typically located at the geometric center and key boundaries of the lesion, simulating the action of a doctor clicking to diagnose the lesion.

[0164] Step S4.4.2 Extraction of negative cue points (for...) aisle).

[0165] Firstly, the same applies to The graph performs an NMS operation while maintaining the distance threshold. Pixel.

[0166] Then set Select the 5 background points with the highest confidence. These points are usually located in flat areas of the retina away from the optic disc, and are used to tell the segmentation model "this is definitely not a lesion".

[0167] Through the above steps, the algorithm eventually automatically outputs a set of prompts containing 10 coordinate points. (1 represents positive, 0 represents negative). This set of prompts can be directly used as input prompts for large models such as the Segment Anything Model (SAM), thereby achieving high-precision lesion segmentation without human intervention.

[0168] This embodiment details step S5: The fundus image to be tested and the segmentation prompt points are input into the segmentation basic model. The prompt points are used as position guides to guide the model to extract the boundary of the optic disc atrophy arc, thereby achieving automatic segmentation of the optic disc atrophy arc under weak text supervision.

[0169] This step details how to use the geometric cue points generated in step S4 to drive the Segmentation Base Model (SAM) to complete the final segmentation of the optic disc atrophy arc (PPA).

[0170] Step S5.1: Construct and assign multimodal input feature embeddings.

[0171] First, the pre-trained SAM-ViT-B (Base version) model is loaded as the base segmenter. Compared to the Huge version, the Base version has advantages in terms of memory usage and inference speed, and is sufficient to handle segmentation tasks of fundus medical images.

[0172] The image encoding in the basic segmenter specifically involves preprocessing the image from step S1... Fundus images are input to the SAM's image encoder. After encoding, the high-resolution image is mapped to low-resolution image feature embeddings. In this embodiment, The dimensions are specifically as follows .in, For the number of embedded channels, This represents the spatial resolution of the feature map (i.e., 1 / 8 of the original image).

[0173] The basic segmenter's cue encoding involves inputting the set of 5 positive and 5 negative cue points extracted in step S4 into the SAM's prompt encoder. Each point is represented by coordinates. It consists of category labels (foreground / background). The model uses learnable positional encodings to map these discrete coordinates into sparse cue feature embeddings. Specifically, the generated Dimensions (10 points, each mapped to a 256-dimensional vector), consistent with the channel dimension of the image feature embedding, to facilitate subsequent fusion.

[0174] Step S5.2: Feature fusion and mask decoding.

[0175] Embedding image features and cue feature embedding The inputs are shared with a lightweight mask decoder. The decoder internally contains two bidirectional Transformer layers. During decoding, self-attention is used to handle interactions between cues, and cross-attention is used to handle interactions between the image and the cues. After processing by the decoder, the model first outputs a resolution of [resolution value missing]. The low-resolution mask is then upsampled to the original resolution using bilinear interpolation. This yields a lesion probability map containing the predicted probability of each pixel.

[0176] Step S5.3: Generate binary segmentation mask and perform post-processing.

[0177] To obtain the final confirmed region, the probability map is processed with a Sigmoid activation function, mapping the values ​​to... Interval. Set probability segmentation threshold. Pixels with a probability value greater than 0.5 are identified as PPA lesion areas (pixel value set to 1); pixels with a probability value less than or equal to 0.5 are identified as background (pixel value set to 0). To further remove noise, connected component analysis is performed on the generated binary mask. A minimum area threshold is set. Pixels, removing isolated noise areas with an area of ​​less than 20 pixels.

[0178] The final preserved connected region is the predicted optic disc atrophy arc lesion region, and the output result shows a fine edge with smoothness and conformity to the anatomical structure.

[0179] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A weakly supervised segmentation method for optic disc atrophy arc based on a visual language model, characterized in that, Includes the following steps: S1: Construct a dual-modal encoder that includes a visual encoder and a text feature encoder; configure the visual encoder to extract multi-scale visual features from fundus images, and configure the text encoder to extract text semantic features of optic disc atrophy arc lesions, thus establishing the foundation for feature extraction; S2: The bimodal encoder is fine-tuned by comparative learning using fundus image-text dataset to achieve semantic alignment between visual features and text features in a unified feature space, thereby obtaining the fine-tuned bimodal encoder; S3: Construct an adaptive multi-channel cue generation network; use a fine-tuned bimodal encoder to generate self-supervised pseudo-labels based on lesion attention heatmaps, and train the network to enable it to map the coordinates of cue points from the heatmap, thus obtaining the trained cue generation network; S4: Acquire the fundus image of the eye to be tested, and use the fine-tuned dual-modal encoder to calculate the similarity matrix between the image and the preset text features to generate a lesion attention heatmap; Input the heatmap into the prompt generation network trained in step S3, and output segmentation prompt points; S5: Input the fundus image to be tested and the segmentation prompt points into the segmentation base model. Use the prompt points as position guides to guide the segmentation base model to extract the boundary of the optic disc atrophy arc, so as to achieve automatic segmentation of the optic disc atrophy arc under weak text supervision.

2. The method as described in claim 1, characterized in that: The method for constructing the visual encoder in S1 is as follows: A visual Transformer is used as the frozen backbone network; the input fundus image is... Divided into Each size is Image blocks; Each image patch is mapped using a linear projection layer. The dimensional embedding vector, after being added with position encoding, is input into the Transformer layer; and a visual adapter is inserted in parallel between every two Transformer encoding blocks of the visual Transformer to obtain the visual encoder.

3. The method as described in claim 1, characterized in that: The method for constructing the text feature encoder in S1 is as follows: A pre-trained large-scale language model is used as the frozen backbone network; the input text description information related to the visual disc atrophy arc (PPA) is segmented; each segmented word is mapped to a D-dimensional embedding vector through an embedding layer to form a text vector sequence, which is then input into the Transformer encoding layer of the backbone network; and a text adapter is inserted between several Transformer layers of the language model to obtain the text feature encoder.

4. The method as described in claim 1, characterized in that: The specific method for establishing and fine-tuning the cross-modal feature alignment mechanism in S2 is as follows: By constructing a comprehensive loss function Image features obtained in step S1 and text features Joint optimization training is performed, and the comprehensive loss function consists of global feature alignment loss, local feature alignment loss, cross-level consistency loss, and a regularization term. The specific calculation steps are as follows: Step S2.1: Construct the global feature alignment loss At the overall level, the similarity of positive sample image-text pairings is improved through contrastive learning, and the loss function is defined as: in, Batch size; For the first The global feature vector of an image; For the first The positive sample text feature vectors corresponding to each image; For any number within the batch One text feature vector; Represents the cosine similarity function; This refers to temperature hyperparameters. Step S2.2: Construct the local feature alignment loss At the local level, a multi-instance learning strategy is employed to perform weakly supervised matching between local image features and text vocabulary features. Since pixel-level annotation is not used, the loss function automatically selects the region with the highest response value in the image as a potential lesion area for optimization, defined as: in, A set of text terms describing PPA lesions; For position Visual feature vector at the location; as a word element The text feature vector; The image feature map with the highest similarity to the lesion text The loss function is a set of Top-K Regions in spatial locations; it guides the model to automatically focus on and locate lesion regions by maximizing the similarity between high-response regions and lesion text. Step S2.3: Construct cross-level consistency loss At the cross-layer constraint level, to minimize the difference between the aligned distributions of shallow and deep features, the loss function is defined as: in, The set of network layers participating in the alignment; For the first Similarity distribution between layer visual features and text features; This serves as a reference similarity distribution; The Kullback-Leibler divergence; For the first Layer weight coefficients; Step S2.4: Calculate the comprehensive loss function in, Let be the weight hyperparameter, satisfying ; For the adapter parameter set Norm regularization term.

5. The method as described in claim 1, characterized in that: The specific method for constructing and training the adaptive multi-channel prompt generation network in S3 is as follows: Step 3.1: Constructing the Network Structure The adaptive multi-channel cue generation network adopts an encoder-decoder structure, and the input is the single-channel attention heatmap used for training. This heatmap comes from step S2.

2. The output of the adaptive multi-channel cue generation network is a dual-channel cue probability map. The dual-channel prompt probability map includes a positive prompt probability map. Negative hint probability graph These correspond to the lesion boundary region and the high-confidence background region predicted by the model, respectively, and achieve collaborative learning of boundary recognition and background suppression within the same network structure. Step 3.2: Construct self-supervised pseudo-labels using image processing algorithms from attention heatmaps Automatically generate pseudo-tag sets Attention heatmap Calculate gradient magnitude plots using gradient operators such as Sobel or Scharr. and to Thresholding and morphological skeletonization are performed to refine the boundary contour, which is then used as a positive cue pseudo-label. ; heatmap Normalization and inversion are performed to obtain the background confidence map. ,right Thresholding is performed, and high-confidence regions are retained as negative cue pseudo-labels. ; Step 3.3: End-to-end network training to construct a dual-channel composite loss function containing positive and negative weights. Optimize the network: in, Selected from Dice loss or Focal Loss to alleviate the problem of foreground and background pixel imbalance; and These are the weighting coefficients.

6. The method as described in claim 1, characterized in that: The specific implementation method for generating the lesion attention heatmap and the cue points in S4 is as follows: Step 4.1: Extract bimodal features from the inference stage, and analyze the fundus image to be tested. The visual encoder, which is fine-tuned in step S2, is input to the dual-modal encoder to extract local visual feature maps. ,in The spatial resolution of the feature map is set; simultaneously, the preset target text prompt of the visual disc atrophy arc is input into the text encoder of the dual-modal encoder after fine-tuning in step S2 to obtain the corresponding global text feature vector. ; Step 4.2: Calculate the image-text feature similarity matrix and calculate the text feature vector. With the local visual feature map Features of various spatial locations The cosine similarity between them yields a two-dimensional similarity matrix. : Among them, the two-dimensional similarity matrix Each position in the matrix The value indicates the degree of correlation between the local area of ​​the image and the preset semantic meaning of "optic disc atrophy arc". The higher the value, the higher the probability that the area is a PPA lesion. Step 4.3: Generate a normalized attention heatmap, and then convert the two-dimensional similarity matrix... Upsampled to the same resolution as the input image The data was then normalized to obtain the final lesion attention heatmap. This heatmap visually reflects the spatial distribution probability of PPA lesions, and is used to achieve coarse localization of the lesion area. Step 4.4: Cue point generation, generating the attention heatmap of the image to be tested. Input the trained network to obtain dual-channel output. ; Positive cue probability diagram in dual-channel output Perform non-maximum suppression (NMS) and Top-K filtering to select the response with the highest value. Each point serves as a positive feedback point; in the negative feedback probability map... Perform nonmaximum suppression and Top-K filtering to select... A high-confidence background point is used as a negative cue point.

7. The method as described in claim 1, characterized in that: The specific implementation method of pixel-level fine segmentation using cue points in S5 is as follows: Step 5.1: Constructing Multimodal Input Feature Embeddings. The preprocessed fundus image from Step S1 is input into the image encoder of the segmentation base model SAM to map the high-resolution image into low-resolution image feature embeddings. Simultaneously, the positive cue point set and negative cue point set extracted in step S4 are input into the cue encoder of the segmentation base model; The spatial coordinate information of the cue points is mapped into sparse cue feature embeddings using learnable location encoding. Positive cue points are encoded as foreground features, and negative cue points are encoded as background features, in order to distinguish lesion and non-lesion areas in the feature space. Step 5.2: Feature Fusion and Mask Decoding embeds the image features. and cue feature embedding The features are jointly input into the mask decoder of the segmentation base model for feature fusion and interaction, and the model outputs a predicted lesion probability map. Step 5.3: Apply the Sigmoid activation function to the probability map for binary segmentation mask, and convert it into a binary mask by setting a probability threshold; the connected regions with a pixel value of 1 in the binary mask are the final predicted PPA lesion regions of the optic disc atrophy arc; output the lesion segmentation result with fine edge structure by locating the main body of the lesion through positive prompts and suppressing background noise through negative prompts.

Images