Semi-supervised spinal column segmentation method based on global-local semantic constraint visual language model

Abstract

The application provides a kind of semi-supervised spinal column segmentation method based on global-local semantic constraint visual language model. It includes: image, text encoder is used to extract spinal column MR slice image embedding and structure description text embedding respectively;Local level learns the semantic relationship of image block and text to measure the uncertainty of local view;Global level measures the wasserstein 2 distance between cross-modal distribution representation to regulate global semantic similarity;Based on the pre-training GLsc model, the text and unmarked slice are combined to generate prompt guide mask, and the semantic constraint is strengthened through feature supervision and region alignment loss;The prediction results of student and teacher model are fused to generate high-quality pseudo label, and the segmentation network is optimized with the help of supervised and consistency loss. The application introduces prompt guide mask into spinal column segmentation semi-supervised learning, enhances the cross-modal uncertainty perception ability of VLM through double-dimensional semantic constraint, effectively improves the pseudo label quality and segmentation precision, and provides an innovative technical paradigm for medical image semi-supervised segmentation.

Description

Technical Field

[0001] This invention relates to the technical field of medical image segmentation, and more specifically, to a semi-supervised spine segmentation method based on a global-local semantic constraint visual language model. Background Technology

[0002] In the medical field, automated spine segmentation (specifically, multi-class segmentation of vertebral bodies and intervertebral discs in spinal MRI images) is of great significance in orthopedic applications, indispensable for scenarios such as spinal disease diagnosis, surgical treatment planning, and spinal pathology identification. Although fully supervised segmentation methods have made progress with the help of large amounts of labeled data, multi-class pixel-level annotation of spinal images is time-consuming, laborious, and requires specialized knowledge. Unlabeled data, on the other hand, is relatively easy to obtain and inexpensive. Furthermore, semi-supervised learning methods can combine limited labeled data with any amount of unlabeled data, thus alleviating the problem of label scarcity and promoting the research and application of spine segmentation.

[0003] Existing semi-supervised learning (SSL) methods for spine segmentation mainly rely on consistency regularization and pseudo-labeling techniques. By constraining consistency and generating pseudo-labels among sub-networks, they have achieved some success in semi-supervised segmentation tasks. These methods typically depend on the complementarity between models to ensure pseudo-label quality, such as generating pseudo-labels and maintaining prediction consistency through a teacher-student model architecture. However, existing SSL methods still have some shortcomings. On the one hand, if the complementarity between models is insufficient, existing SSL methods may produce inconsistent predictions for the same sample, directly affecting the quality of pseudo-labels. On the other hand, while visual language models (VLMs) have the potential to improve pseudo-label quality through image-text semantic associations, they face the challenge of cross-modal alignment uncertainty when applied to spine segmentation. The same text may correspond to multiple images, and the same image may match multiple semantically ambiguous similar texts (such as synonyms). Traditional cross-modal uncertainty perception methods based on distribution modeling rely only on distribution representations, which loses semantic attributes and cannot fully capture complex semantic associations, making it difficult for VLMs to effectively adapt to semi-supervised spine segmentation tasks.

[0004] Therefore, we urgently need to develop a novel semi-supervised spine segmentation method based on visual language models to address the shortcomings of existing semi-supervised learning methods for spine segmentation. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0006] To address the shortcomings of existing technologies, this invention aims to provide a semi-supervised spine segmentation method based on a global-local semantic constraint visual language model. It introduces cue-guided masks into the semi-supervised learning of spine segmentation, leveraging the semantic prior knowledge of a pre-trained visual language model to improve the quality of pseudo-label generation. Simultaneously, it proposes a global-local semantic constraint visual language model (GLsc), which learns the semantic relationship between image patches and text through local semantic constraints to measure local uncertainty, and uses global semantic constraints to calculate the Wasserstein-2 distance across modal distributions to normalize global semantic similarity, thereby solving the cross-modal uncertainty problem of VLM and enhancing the robustness of image-text alignment.

[0007] To achieve the above objectives, the present invention provides a semi-supervised spine segmentation method based on a global-local semantic constraint visual language model. The method includes: Step S1: Processing spine MR slices from a dataset using an image encoder to obtain corresponding image embeddings, and processing text describing the spine structure using a text encoder to generate corresponding text embeddings, thereby constructing a feature-level association between the image and the text; wherein the image embedding includes: image patch tokens and an image CLS token; the text embedding includes: a text CLS token; Step S2: Extracting the image patch tokens and the text CLS token, and then calculating the image patch tokens and the text CLS token... The cosine similarity between the image and text pairs is calculated, and a corresponding cosine similarity matrix A is constructed based on this cosine similarity. The variance of the similarity between different image patch-text pairs is then calculated based on this variance, and a weighted similarity is calculated based on this variance. The local semantics of high-uncertainty image patches are then associated based on this weighted similarity to impose constraints on the local semantics of these high-uncertainty image patches, thereby improving the reliability of local image-text alignment. High-uncertainty image patches are those where the variance of the calculated similarity between different image patch-text pairs is greater than a preset variance value; low-uncertainty image patches are those where the variance of the calculated similarity between different image patch-text pairs is less than or equal to a preset variance value. Step S3: Extract the image CLS token and text CLS token, calculate the cosine similarity between the image CLS token and text CLS token, construct a corresponding cosine similarity matrix B based on this cosine similarity, and then, using distribution modeling, associate the image CLS token and text CLS token... The corresponding semantic features are mapped from the embedding space to the distribution space to generate image distributions respectively. I n and text distribution T n Then calculate the image distribution.I n With text distribution T n The Wasserstein-2 distance between the images is used to construct an uncertainty matrix C, which is then used to constrain the global semantic similarity of the images and text. Each element of the uncertainty matrix C reflects the alignment uncertainty of different image-text pairs at the global semantic level. Step S4: Input the unlabeled spine MR slices in the dataset into a pre-trained image encoder from a pre-trained GLsc to obtain the corresponding image embeddings. Step S5: Input the text prompts of a certain category from the labeled spine MR slices in the dataset into a pre-trained text encoder from a pre-trained GLsc to generate the corresponding text embeddings. Then, calculate the confidence score of the current input corresponding to a certain category based on the image embedding and text embedding. Finally, filter out high-confidence image regions based on the confidence score to generate a prompt guidance mask for that category. The high-confidence image region is the image region corresponding to when the calculated confidence score for that category is greater than a preset confidence score. Step S6: For the labeled spine MR slices in the dataset... Step S5 is repeated for text cues of other categories of slices to obtain cue guidance masks for each category in the dataset. Then, the cue guidance masks for each category in the dataset are stacked based on confidence scores to form the final cue guidance mask. Step S7: Based on the final cue guidance mask, feature supervision loss and region alignment loss are calculated to strengthen cross-modal semantic constraints. Step S8: The final cue guidance mask is applied to unlabeled spine MR slices in the dataset. The unlabeled spine MR slices after the cue guidance mask is applied are input into the student model to generate unlabeled predictions. Labeled spine MR slices in the dataset are also input into the student model to generate labeled predictions. Unlabeled spine MR slices in the dataset are input into the teacher model to generate pseudo-labels. The unlabeled predictions, labeled predictions, and pseudo-labels are then fused to generate merged pseudo-labels. The teacher model is updated using the exponential moving average of the student model. Step S9: The segmentation network is iteratively optimized based on supervised loss and consistency loss to achieve a high-quality final output of spine segmentation results. The segmentation network consists of a teacher model and a student model.

[0008] Preferably, in step S1, the image encoder processes the spine MR slices of the dataset to obtain the corresponding image embeddings. Specifically, the image encoder extracts multi-scale and multi-level image features from the spine MR slices of the dataset to generate the image patch tokens and image CLS tokens. The image patch tokens correspond to local image region features at different resolutions. The image CLS token is a feature identifier that integrates global image information. The text encoder processes the text describing the spine structure to generate the corresponding text embeddings. Specifically, the text encoder processes the text describing the spine structure through word embedding and positional encoding to output the text CLS token. The text CLS token is a comprehensive representation of the text semantics.

[0009] Preferably, in step S2, the variance of the similarity of different image block-text pairs is calculated based on the cosine similarity matrix A. Specifically, based on the cosine similarity matrix A, the similarity score between each image block token and the text CLStoken is calculated, and then the variance of the similarity of different image block-text pairs is calculated within the set of similarity scores of all image block-text pairs. When calculating the weighted similarity by combining the variance, a first weight is assigned to the similarity score of image block-text pairs with high uncertainty, and a second weight is assigned to the similarity score of image block-text pairs with low uncertainty. Among them, image block-text pairs with high uncertainty correspond to image block-text matching pairs composed of image blocks whose variance is greater than a preset variance value; image block-text pairs with low uncertainty correspond to image block-text matching pairs composed of image blocks whose variance is less than or equal to a preset variance value.

[0010] Preferably, in step S6, the prompt guidance masks of each category in the dataset are stacked based on the confidence score to form the final prompt guidance mask. Specifically, the prompt guidance masks of each category in the dataset are stacked in descending order of confidence score, and the prompt guidance masks of a single category are weighted and superimposed.

[0011] Preferably, step S7 specifically includes: step S71: embedding the image patch corresponding to the final prompt guidance mask and associating it with the text global CLS token to calculate the feature supervision loss, so as to measure the feature difference between the image patch embedding corresponding to the final prompt guidance mask and the text global CLS token; step S72: extracting the region image corresponding to the final prompt guidance mask and comparing the region image with the text global CLS token to calculate the region alignment loss, so as to strengthen the alignment between the semantics of the image region and the semantics of the text global token. Preferably, in step S8, a weighted average method is used to fuse the unlabeled prediction, labeled prediction, and pseudo-label to generate a merged pseudo-label.

[0012] The beneficial effects of this invention are:

[0013] (1) The semi-supervised spine segmentation method based on the global-local semantic constraint visual language model provided by the present invention introduces the prompt guidance mask into the semi-supervised learning of spine segmentation for the first time. It improves the generation quality of pseudo-labels by using the semantic prior knowledge of the pre-trained visual language model, and introduces a new technical idea for the semi-supervised learning of spine segmentation.

[0014] (2) The semi-supervised spine segmentation method based on the global-local semantic constraint visual language model provided by this invention proposes a global-local semantic constraint visual language model. This model can pre-train a VLM with uncertainty perception capability. It learns the semantic relationship between image patches and text through local semantic constraints to measure local uncertainty. It uses global semantic constraints to calculate the Wasserstein-2 distance of cross-modal distribution to normalize global semantic similarity. This solves the problem of cross-modal uncertainty of VLM, enhances the robustness of image-text alignment, and optimizes the performance of visual language model in cross-modal tasks from the perspective of model architecture and training mechanism.

[0015] Additional aspects and advantages of the invention will become apparent from the description which follows, or may be learned by practice of the invention. Attached Figure Description

[0016] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0017] Figure 1 A schematic flowchart of a semi-supervised spine segmentation method based on a global-local semantic constraint visual language model according to an embodiment of the present invention is shown.

[0018] Figure 2A schematic framework diagram of the pre-training process of a visual language model based on global-local semantic constraints according to an embodiment of the present invention is shown.

[0019] Figure 3 A schematic framework diagram of a cue-guided semi-supervised learning process for spine segmentation according to an embodiment of the present invention is shown. Detailed Implementation

[0020] To better understand the above-mentioned objects, features, and advantages of the present invention, such as Figures 1 to 3 As shown in the accompanying drawings and specific embodiments, the present invention will be further described in detail below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0022] Figure 1 A schematic flowchart of a semi-supervised spine segmentation method based on a global-local semantic constraint visual language model, according to an embodiment of the present invention, is shown. Figure 1 As shown, this semi-supervised spine segmentation method based on a global-local semantic constraint visual language model includes:

[0023] Step S1: Process the spine MR slices of the dataset using an image encoder to obtain the corresponding image embeddings, and process the text describing the spine structure using a text encoder to generate the corresponding text embeddings, so as to build the association between the image and the text at the feature level; wherein, the image embedding includes: image block tokens and image CLS token; the text embedding includes: text CLS token;

[0024] Step S2: Extract the image block tokens and text CLS token, calculate the cosine similarity between the image block tokens and the text CLS token, construct a corresponding cosine similarity matrix A based on the cosine similarity, calculate the variance of the similarity between different image block-text pairs based on the cosine similarity matrix A, calculate the weighted similarity based on the variance, and associate the local semantics of high-uncertainty image blocks based on the weighted similarity to impose constraints on the local semantics of high-uncertainty image blocks in the future, thereby improving the reliability of local image-text alignment; wherein, high-uncertainty image blocks are the image blocks corresponding to the calculated variance of the similarity between different image block-text pairs being greater than a preset variance value; low-uncertainty image blocks are the image blocks corresponding to the calculated variance of the similarity between different image block-text pairs being less than or equal to a preset variance value;

[0025] Here, the variance is calculated as the "similarity between different image patch-text pairs", which reflects the "fuzziness of semantic matching between image patches and text", corresponding to the local unit (image patch) of the image.

[0026] Step S3: Extract the image CLS token and text CLS token, then calculate the cosine similarity between the image CLS token and the text CLS token, and construct the corresponding cosine similarity matrix B based on the cosine similarity. Then, using distribution modeling techniques, map the semantic features corresponding to the image CLS token and text CLS token from the embedding space to the distribution space to generate image distributions respectively. I n and text distribution T n Then calculate the image distribution. I n With text distribution T n The Wasserstein-2 distance between the images and text is used to construct an uncertainty matrix C, and then a constraint is imposed on the global semantic similarity of the images and text based on the uncertainty matrix C; wherein, each element of the uncertainty matrix C reflects the alignment uncertainty of different image-text pairs at the global semantic level.

[0027] Step S4: Input the unlabeled spine MR slices in the dataset into the pre-trained image encoder from the pre-trained GLsc to obtain the corresponding image embeddings;

[0028] Step S5: Input the text prompts of a certain category of the labeled spinal MR slices in the dataset into the pre-trained text encoder from the pre-trained GLsc to generate the corresponding text embeddings. Then, calculate the confidence score of the current input corresponding to the certain category based on the image embeddings and text embeddings. Then, filter out the image regions with high confidence based on the confidence score to generate the prompt guidance mask for the certain category. The high confidence image regions are the image regions corresponding to the calculated confidence score of the certain category being greater than the preset confidence score.

[0029] Step S6: Repeat step S5 for text cues of other categories of labeled spinal MR slices in the dataset to obtain cue guidance masks for each category in the dataset. Then, stack the cue guidance masks for each category in the dataset based on the confidence score to form the final cue guidance mask.

[0030] Step S7: Based on the final cue guidance mask, calculate the feature supervision loss and region alignment loss to strengthen cross-modal semantic constraints;

[0031] Step S8: Apply the final cue guidance mask to the unlabeled spinal MR slices in the dataset, and input the unlabeled spinal MR slices after applying the cue guidance mask into the student model to generate unlabeled predictions. Also input the labeled spinal MR slices in the dataset into the student model to generate labeled predictions. Input the unlabeled spinal MR slices in the dataset into the teacher model to generate pseudo-labels. Then, fuse the unlabeled predictions, labeled predictions, and pseudo-labels to generate merged pseudo-labels. The teacher model is updated by the student model using an exponential moving average.

[0032] Step S9: Iteratively optimize the segmentation network based on supervised loss and consistency loss to achieve a high-quality spine segmentation result; wherein the segmentation network consists of a teacher model and a student model.

[0033] In this embodiment, as Figure 1 As shown, the process begins with multimodal feature encoding in step S1, which involves extracting image and text features. Specifically, an image encoder processes spinal magnetic resonance (MR) slices to obtain image embeddings, achieving the goal of accurately capturing anatomical structural features in the image. A text encoder then processes the relevant text (i.e., text describing the spinal structure) to generate text embeddings, transforming the semantic meaning of the text into a machine-understandable feature form. These two steps establish a preliminary basic association between the image and text at the feature level.

[0034] Furthermore, a two-dimensional semantic constraint optimization is performed, optimizing image-text alignment from both local and global dimensions. Local semantic constraints focus on micro-level associations. Specifically, in step S2, image patch tokens are extracted from the spinal MR slices, each corresponding to a different local region within the slice. Text CLS tokens are extracted from the text describing spinal semantics (i.e., text describing spinal structure), serving as key representations of the text semantics. The cosine similarity between the image patch tokens and the text CLS tokens is then calculated to construct a similarity matrix A. This quantifies the degree of association between each image patch and the text semantics, clarifying the initial matching status between local image regions and text semantics. Based on this similarity matrix A, the variance of the similarity between different image patch-text pairs is calculated. This variance reflects the dispersion of local semantic associations; a larger variance indicates a more unstable correspondence between the image and text semantics in that local region, with higher uncertainty. Then, by combining the calculated variance to calculate the weighted similarity, the local semantic associations of high-uncertainty image patches (large variance) are determined, so as to impose constraints on subsequent model learning, making the model pay more attention to local regions with stable semantic associations and low uncertainty, thereby improving the reliability of local image-text alignment.

[0035] The global semantic constraint is addressed from a macroscopic perspective. Specifically, in step S3, key feature tokens representing global semantics are extracted from the spinal MR slices and the text describing the spinal structure. Based on the extracted image CLS tokens and text CLS tokens, their cosine similarity is calculated to construct a cosine similarity matrix B. Then, using distribution modeling techniques, the semantic features corresponding to the image CLS tokens and text CLS tokens are mapped from the embedding space to the distribution space to generate image distributions. I n and text distribution T n In the distribution space, the image distribution is then calculated. I n With text distribution T n The Wasserstein-2 (W2) distance between the images and text is calculated. An uncertainty matrix C is then constructed based on the calculated W2 distance; each element of this uncertainty matrix C reflects the alignment uncertainty of different image-text pairs at the global semantic level. Constraints are then imposed on the global semantic similarity of the images and text based on this uncertainty matrix C.

[0036] Furthermore, steps S4, S5, and S6 utilize a pre-trained Global-Local Semantic Constraint Visual Language Model (GLsc) to generate a cue guidance mask to improve the quality of pseudo-labels. Specifically, unlabeled spinal MR slices and corresponding text cuees (e.g., "lumbar vertebral body slices") from the dataset are used as input. The unlabeled MR slices are processed by a pre-trained image encoder to extract features, generating image CLS tokens and image patch tokens. The text cuees are encoded by a pre-trained text encoder to obtain text embeddings for the corresponding categories, providing semantic guidance for mask generation. The Global-Local Semantic Constraint Visual Language Model is referred to as GLsc. Then, using the text embeddings and image features, the confidence score of the current input's corresponding category is calculated, and high-confidence image regions are selected to generate a "cue guidance mask for that category." This operation is repeated for all categories in the dataset to obtain "masks for each category in the dataset," which are then stacked according to confidence to form the final cue guidance mask, achieving accurate labeling of the spinal target region.

[0037] Furthermore, in step S7, based on the final cue guidance mask, feature supervision loss and region alignment loss are calculated to achieve cross-modal alignment constraints. Specifically, feature and region constraints are performed simultaneously in this process: on the one hand, by embedding the image patch corresponding to the final cue guidance mask and associating it with the text-global CLS token to calculate the feature supervision loss, the model learns more accurate image-text feature associations by constraining the consistency between the "cue guidance image embedding" and the text semantics; on the other hand, by extracting the region image corresponding to the final cue guidance mask and comparing the region image with the text-global CLS token to calculate the region alignment loss. This forces the image region semantics to align with the text-global semantics, ensuring that the "intervertebral disc region" mask accurately corresponds to the actual intervertebral disc location in the image.

[0038] Furthermore, step S8 prompts a mask applied to the unlabeled MR slice, and combined with the predictions from the student model, generates "unlabeled predictions"; simultaneously, labeled data is processed by the student model to obtain "labeled predictions." These two are then fused with the output of the teacher model (0) to generate "merged pseudo-labels," thereby improving the quality of the pseudo-labels. The weights of the teacher model output are dynamically adjusted based on the consistency metric between the student and teacher models; when the consistency between the student and teacher models is high, the weights of the teacher model output increase, and vice versa.

[0039] Finally, in step S9, the segmentation network is continuously iteratively optimized in the semi-supervised learning process based on supervised loss and consistency loss. The model continuously learns and adjusts its parameters, ultimately enabling it to accurately identify various structures in spine images, thus achieving a high-quality spine segmentation output. This concludes the semi-supervised spine segmentation method based on a global-local semantic constraint visual language model in this embodiment.

[0040] In one embodiment of the present invention, in step S1, the image encoder processes the spinal MR slices of the dataset to obtain the corresponding image embedding. Specifically, the image encoder extracts multi-scale and multi-level image features from the spinal MR slices of the dataset to generate the image patch tokens and the image CLS token. The image patch tokens correspond to local image region features at different resolutions. The image CLS token is a feature identifier that integrates global image information. The text encoder processes the text describing the spinal structure to generate the corresponding text embedding. Specifically, the text encoder processes the text describing the spinal structure through word embedding and positional encoding to output the text CLS token. The text CLS token is a comprehensive representation of the text semantics.

[0041] In one embodiment of the present invention, in step S2, the variance of the similarity of different image block-text pairs is calculated based on the cosine similarity matrix A. Specifically, based on the cosine similarity matrix A, the similarity score between each image block token and the text CLS token is calculated, and the variance of the similarity of different image block-text pairs is calculated within the set of similarity scores of all image block-text pairs. When calculating the weighted similarity by combining the variance, a first weight is assigned to the similarity score of image block-text pairs with high uncertainty, and a second weight is assigned to the similarity score of image block-text pairs with low uncertainty. Among them, image block-text pairs with high uncertainty correspond to image block-text matching pairs composed of image blocks whose variance is greater than a preset variance value; image block-text pairs with low uncertainty correspond to image block-text matching pairs composed of image blocks whose variance is less than or equal to a preset variance value.

[0042] In one embodiment of the present invention, in step S6, the prompt guidance masks of each category in the dataset are stacked based on the confidence score to form the final prompt guidance mask. Specifically, the prompt guidance masks of each category in the dataset are stacked in descending order of confidence score, and the prompt guidance masks of a single category are weighted and superimposed.

[0043] In one embodiment of the present invention, step S7 specifically includes: step S71: embedding the image patch corresponding to the final prompt guidance mask and associating it with the text global CLS token to calculate feature supervision loss, so as to measure the feature difference between the image patch embedding corresponding to the final prompt guidance mask and the text global CLS token; step S72: extracting the region image corresponding to the final prompt guidance mask and comparing the region image with the text global CLS token to calculate region alignment loss, so as to strengthen the alignment between image region semantics and text global semantics. In one embodiment of the present invention, in step S8, a weighted average method is used to fuse the unlabeled prediction, labeled prediction, and pseudo-label to generate a merged pseudo-label.

[0044] Another embodiment of the present invention provides a semi-supervised spine segmentation system based on a global-local semantic constraint visual language model. This system includes: a multimodal feature encoding module configured to extract features from images and text and construct their association; inputting MR slices of the spine and text describing the spine structure; calling an image encoder to extract image embeddings (including image patch tokens and image CLS tokens) to accurately capture anatomical structural features; a text encoder processing the text to generate text embeddings (including text CLS tokens) to realize the conversion of text semantics into machine-understandable features, initially establishing a basic association at the image-text feature level; and a semantic constraint optimization module configured to optimize image-text alignment from both local and global dimensions. Local semantic constraints focus on micro-level semantic associations, extracting image patch tokens and text CLS tokens, calculating cosine similarity to construct a matrix, and quantifying the local semantic matching degree; then, local semantic uncertainty is measured through variance calculation, and weighted similarity is generated based on variance to impose constraints on high-uncertainty associations, guiding the model to focus on stable semantic regions and improving local alignment reliability; global semantic constraints are oriented towards macro-level semantic distribution, extracting global CLS tokens from images and text. The system calculates cosine similarity to construct a matrix; it maps semantic features to a distribution space through distribution modeling, generating image and text distributions; it calculates the W2 distance between the two to measure the distribution difference, constructs an uncertainty matrix, and imposes constraints on global semantic similarity to enhance the robustness of cross-modal semantic alignment; the cue guidance mask generation module is configured to rely on pre-trained GLsc, taking unlabeled MR slices and corresponding category text cue as input, and reusing the pre-trained encoder of the encoding module. It calculates category confidence scores using text embeddings and image features, selects high-confidence regions to generate single-category cue guidance masks; it traverses all categories in the dataset, stacks masks to obtain the final version, accurately labels the spine target region, and provides a foundation for pseudo-label optimization; the cross-modal alignment constraint module is configured to execute synchronously with mask generation, strengthens semantic alignment through two types of loss functions, associates the image patch embeddings of the cue guidance mask with the global CLStoken of the text, calculates feature supervision loss, constrains the consistency of image-text feature association, and allows the model to learn more accurate semantic matching. The image corresponding to the mask is extracted and compared with the global and regional CLS tokens of the text. The region alignment loss is calculated to force the semantics of the image region to match the text description. The pseudo-label enhancement module is configured to integrate multi-source prediction results to optimize the quality of pseudo-labels. The cue-guided mask is applied to unlabeled slices and combined with the student model output to generate "unlabeled predictions"; labeled data is used by the student model to generate "labeled predictions". The two types of predictions are fused with the output of the teacher model (updated by the student model EMA) to generate "merged pseudo-labels", which solves the problem of single-source prediction bias and provides high-quality supervision signals for semi-supervised learning. The segmentation network iterative optimization module is configured to drive model training using multiple loss functions.The segmentation network is iteratively optimized within a semi-supervised framework using supervised loss and consistency loss methods. The model continuously adjusts its parameters, gradually improving the accuracy of spinal structure recognition, ultimately achieving the goal of high-quality spinal segmentation.

[0045] Another embodiment of the present invention provides a medium having a program stored thereon, which, when executed by a processor, implements the steps in the semi-supervised spine segmentation method based on a global-local semantic constraint visual language model as described in any of the above embodiments of the present invention.

[0046] Another embodiment of the present invention provides an apparatus including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the semi-supervised spine segmentation method based on a global-local semantic constraint visual language model as described in any of the above embodiments of the present invention.

[0047] The technical solution of the present invention will be illustrated below with a specific embodiment. This specific embodiment of the semi-supervised spine segmentation method based on a global-local semantic constraint visual language model is implemented through the following steps:

[0048] (1) Step 1: Obtain image and text features. Use an image encoder to process the spinal magnetic resonance (MR) slices to obtain image embeddings; use a text encoder to process the relevant text (i.e., the text describing the spinal structure) to generate text embeddings.

[0049] The image embedding includes image block tokens and image CLS tokens; the text embedding includes text CLS tokens; a single image can be divided into multiple image blocks and has multiple local features, so tokens are in plural form; however, a single image or a single text has only one global feature, so CLS tokens are in singular form.

[0050] (2) Step 2: VLM pre-training using Global-Local Semantic Constraints (GLSC). GLSC adaptively models the uncertainty in each patch text correspondence to achieve local semantic understanding and measures the distribution-based uncertainty representation to achieve semantic similarity constraints, thereby improving the robustness of cross-modal alignment.

[0051] Step 2 specifically includes:

[0052] Step 2.1: Apply Local Semantic Constraints. The goal of local semantic constraints is to guide the model to adaptively focus on diverse semantic information by measuring the uncertainty between each image patch and the text. The similarity of mismatched image patch-text pairs is treated as a probability distribution, the variance of which... The uncertainty weights are characterized by the image patch-text contrastive learning. Among them, variance... The expression is:

[0053]

[0054] In equation (1), k It is the first image k One patch; The contrastive learning loss applied between the k-th patch and the text is used. The expression for the loss due to local semantic constraints is:

[0055]

[0056] In equation (2), This is considered local uncertainty. Smaller This corresponds to lower uncertainty, thus attracting greater attention to the corresponding patch-text loss.

[0057] Step 2.2: Perform global semantic constraints. The goal of global semantic constraints is to constrain global semantic similarity based on the corresponding level of uncertainty. It embeds the semantics of images and text using a Gaussian distribution. Modeled as a distributed representation The distance represented by the distribution is formulated as the level of uncertainty.

[0058] Given N Image-text pairs , n ∈{1,2,..., N Semantic distance is defined as:

[0059]

[0060] In equation (3), It is the first n Semantic embeddings of image-text pairs. A Gaussian distribution is used to model the semantic embeddings as a distributed representation. 2-Wasserstein is used to measure the difference between image-text distribution representations. The 2-Wasserstein distance is defined as:

[0061] (4)

[0062] In equation (4), Represents the mean vector. It is the standard deviation vector. The level of uncertainty is given by the following formula:

[0063]

[0064] In equation (5), a It is a direct proportionality factor. bThis is the deviation value. Uncertainty level. Semantic differences The ratio is defined as relative uncertainty. :

[0065]

[0066] In equation (6), It is a positive parameter used to control the strength of the constraint.

[0067] To avoid semantic uncertainty in image-text alignment, cosine similarity between images and text is used. Through relative uncertainty Constraints are applied to obtain uncertain cosine similarity. Optimization is performed using the InfoNCE loss function. The global semantic constraint loss is expressed as:

[0068]

[0069]

[0070] In equation (8), It is a learnable temperature hyperparameter. T + Indicates and I 1. Matching the main text, I + Is with T 1. Matching positive image.

[0071] (3) Step 3: Semi-supervised learning for pseudo-label quality improvement. Our cue-guided semi-supervised learning integrates semantic priors from pre-trained GLsc into semi-supervised spine segmentation, improving pseudo-label quality through guided masks.

[0072] Step 3.1: Cue-Guiding Mask Generation. The spine image embedding is merged with the text cue for each category to generate a mask corresponding to the text cue for each category. The masks for each category are stacked based on their respective confidence scores to form the final cue-guiding mask. Specifically, this is achieved through a decoder. Image patchtokens encoded by the image encoder Decoding into dense image embedding The mask for each category is... and The dot product between them is generated (text embeddings for each category).

[0073]

[0074] In equation (9), This refers to the softmax function. Operation based on each Similarity score right Stacking is performed. To obtain reliable cue-guided masks, we introduce region alignment loss and feature supervision loss.

[0075] Step 3.1.1: Calculate the region alignment loss. To directly learn region-text alignment for better semantic correspondence, extract region image embeddings from the cue-guided region images. Embedded with the overall text Alignment. The alignment loss for this region is defined as:

[0076]

[0077] Step 3.1.2: Calculate the feature-supervised loss. To prevent the model from generating masks for regions not described in the text, a feature-supervised loss is introduced to constrain the negative masks obtained from irrelevant text. Its expression is:

[0078]

[0079] In equation (11), The image embedding features indicate the prompts and guidance.

[0080] Step 3.2: Generating pseudo-tags for teacher-student network guidance. Teacher network From the student network via the Exponential Moving Average (EMA) Update the parameters; For labeled images Compared with unlabeled images Make predictions; Then generate pseudo-labels for the unlabeled images. .

[0081]

[0082] In equation (12), This represents the prediction results of the labeled data. This indicates the prediction result for unlabeled data.

[0083] (4) Step 4: Training a cue-guided semi-supervised learning paradigm. For labeled images, in the real label... And labeled prediction Calculate the monitoring loss between For unlabeled images In the unlabeled prediction results Calculate the semi-supervised loss between the merged pseudo-labels and the pseudo-labels. The fused pseudo-labels are a combination of pseudo-labels generated by the teacher network and text-based guidance masks; simultaneously, in the unlabeled prediction results... With text prompt guide mask Calculated between. and Defined as:

[0084]

[0085]

[0086]

[0087]

[0088] In equations (13), (14), (15), and (16) above, This represents the cross-entropy loss function. Representing the The first of the masks 1 pixel. These represent the batch size of labeled and unlabeled images, respectively. W and H represent the width and height of the image.

[0089] The overall loss function is The segmentation network is optimized using the overall loss function.

[0090] The overall inventive concept of this invention is described below with reference to a specific embodiment: For the semi-supervised learning task of spinal segmentation, a technical solution is proposed that integrates a visual language model (VLM) with semantic constraints and prompting guidance. For example... Figure 1 and Figure 2 As shown, firstly, an image encoder processes spinal magnetic resonance (MR) slices to obtain image embeddings, and a text encoder processes the spinal structure description text to generate text embeddings, constructing a basic association at the image-text feature level. Next, semantic constraint learning is performed from local and global dimensions. Local semantic constraints extract image patch tokens and text CLS tokens, calculate the cosine similarity matrix, variance, and weighted similarity to constrain the uncertainty of local semantic associations and improve local alignment reliability. Global semantic constraints extract global CLS tokens from the image and text, construct the cosine similarity matrix, perform distribution modeling, calculate Wasserstein-2 distance, and construct the uncertainty matrix to constrain global semantic similarity. Finally, a pre-trained global-local semantic constraint VLM (GLsc) is used, such as... Figure 3As shown, R slices and corresponding category text prompts are used as inputs. After feature extraction and confidence score calculation, prompt guidance masks for each category are generated and stacked. Simultaneously, feature supervision loss and region alignment loss are used to strengthen feature and region constraints. Finally, the prompt guidance masks are applied to unlabeled MR slices and combined with student model predictions. These predictions are then fused with labeled data predictions and teacher model outputs to generate merged pseudo-labels. The segmentation network is iteratively optimized using supervised loss and consistency loss to achieve high-quality spine segmentation. This solves the problems of label scarcity and cross-modal alignment difficulties in existing methods, improving the accuracy and robustness of spine segmentation.

[0091] In summary, this invention proposes a semi-supervised spine segmentation method based on a global-local semantically constrained visual language model. It is the first to introduce cue-guided masks into semi-supervised spine segmentation learning, leveraging the rich semantic prior knowledge inherent in a pre-trained visual language model (VLM) to deeply mine the semantic relationships between spinal medical images and text descriptions, accurately guiding the pseudo-label generation process. By enabling the model to understand the guidance of textual semantics such as "vertebral body morphology" and "intervertebral disc location" on image segmentation, it effectively improves the quality of pseudo-labels in terms of anatomical structure correspondence and semantic consistency. This opens up a new technical path for spine segmentation semi-supervised learning tasks that have long been limited by scarce labels and insufficient semantic utilization, integrating multimodal semantic guidance.

[0092] This invention proposes a novel visual language model with two-dimensional semantic constraints, endowing the model with cross-modal uncertainty perception capabilities through innovative architecture and training mechanisms. The model constructs a local-global semantic constraint collaborative system during the pre-training phase.

[0093] This invention proposes a local semantic constraint module that focuses on the precise mapping between microscopic regions of an image and text semantics. By extracting image patch tokens and text CLS tokens, a semantic similarity matrix is ​​calculated and combined with variance analysis to quantify the uncertainty of the semantic association between each local region of the image and the text. Dynamic weight constraints are applied to local regions with high variance (i.e., unstable semantic correspondence), guiding the model to prioritize learning image patches with clear anatomical structures and stable semantic associations (such as vertebral body edges, intervertebral disc spaces, and other feature regions), thereby improving the reliability of local semantic alignment.

[0094] This invention proposes a global semantic constraint module to optimize cross-modal associations at a macro-distribution level. By extracting global CLS tokens from images and text, semantic features are mapped to the distribution space, and the Wasserstein-2 (W2) distance is calculated to accurately measure the degree of difference in semantic distribution between images and text. An uncertainty matrix is ​​constructed based on the W2 distance to apply regularization constraints to image-text pairs with large deviations in semantic distribution, suppressing semantic misalignment caused by cross-modal ambiguity such as "one word in multiple images" and "one image with multiple meanings," and strengthening the consistent learning of global semantic similarity.

[0095] The innovative points of this invention are illustrated below with reference to a specific embodiment:

[0096] (1) A novel pre-trained global-local semantic constraint (GLsc) visual language model with uncertainty perception capability was developed to solve the cross-modal alignment problem through global and local dual-dimensional constraints. For the first time, global semantic constraints were achieved by calculating the difference between image distribution features and text distribution features to constrain the cosine similarity between image features and text features. For the first time, local semantic constraints were achieved by constraining the cosine similarity between each image patch and global text features, as well as the cosine similarity between each word feature and global image features.

[0097] The innovations mentioned above differ from existing technologies as follows: In the field of medical imaging, such as in spinal segmentation, images contain rich and unique semantic attributes, such as the specific shape of the vertebrae and the relative positional relationship between the intervertebral discs and surrounding tissues. However, existing visual language models, when processing cross-modal information, often rely on distribution-based modeling methods that only focus on the statistical distribution characteristics of features. In calculating the differences in feature distribution between images and text to achieve cross-modal alignment, these crucial semantic attributes are ignored. This fails to establish a clear correspondence between local semantic units in the image and the text description, leading to a biased understanding of the spinal anatomy.

[0098] Our proposed visual language model, pre-trained with GLsc, estimates the uncertainty relationship between local semantics and text in images and measures the distance between cross-modal distributions through local and global semantic constraints, respectively. This achieves more accurate cross-modal alignment and effectively addresses the shortcomings of existing methods in semantic association.

[0099] (2) A novel semi-supervised learning framework was constructed, and for the first time, prompt guidance masks were transferred to semi-supervised learning for spine segmentation. The quality of pseudo-labels in semi-supervised training was improved by utilizing the GLsc semantic prior of the pre-trained visual language model.

[0100] The differences between the above-mentioned innovations and existing technologies are as follows: Existing semi-supervised learning (SSL) methods are mainly based on consistency regularization and pseudo-label techniques. Although they show good performance in semi-supervised segmentation, they have significant shortcomings: They rely on consistency constraints or pseudo-label generation between sub-networks. However, when the models have similar architectures and similar training strategies, resulting in insufficient complementarity, feature extraction homogenization problems are likely to occur. When the models are not complementary enough, they will produce contradictory predictions for the same unlabeled sample, which will pollute the quality of pseudo-labels, mislead model training, and reduce segmentation accuracy.

[0101] Our method is the first to transfer cue-guided masks to semi-supervised learning, using the semantic prior knowledge of the pre-trained visual language model GLsc to generate pseudo-labels. This upgrades the pseudo-label generation for spine segmentation from "inter-model prediction consistency driven" to "joint semantic drive of text semantics and image features". By optimizing the semantic decision boundary through text cues, we fundamentally improve the quality and reliability of pseudo-labels.

[0102] (3) For the first time, the visual language model was applied to the spinal segmentation task. It fully utilizes the complementary information of text-image multimodal modes to construct a cross-modal semantic alignment and constraint system, providing a new paradigm for multimodal knowledge injection in the field of medical image segmentation.

[0103] The differences between the above-mentioned innovations and existing technologies are as follows: Existing medical image segmentation technologies are mostly limited to single modality or simple model collaboration, making it difficult to fully utilize the complementary information of multimodal data.

[0104] Our newly constructed framework deeply integrates visual language models and semi-supervised learning, making full use of multimodal complementary information to conduct cross-modal deep interaction between text semantics and image features, providing a new technical path for medical image segmentation.

[0105] The above three points constitute the core supporting innovations of this invention. In summary, the semi-supervised spine segmentation method based on a global-local semantic constraint visual language model provided by this invention has unparalleled technical advantages.

[0106] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A semi-supervised spine segmentation method based on a global-local semantic constraint visual language model, characterized in that, include: Step S1: Process the spine MR slices of the dataset using an image encoder to obtain the corresponding image embeddings, and process the text describing the spine structure using a text encoder to generate the corresponding text embeddings, so as to build the association between the image and the text at the feature level; wherein, the image embedding includes: image patch tokens and image CLS tokens; the text embedding includes: text CLS tokens; Step S2: Extract the image block tokens and text CLS token, calculate the cosine similarity between the image block tokens and the text CLS token, construct the corresponding cosine similarity matrix A based on the cosine similarity, calculate the variance of the similarity between different image block-text pairs based on the cosine similarity matrix A, calculate the weighted similarity based on the variance, and associate the local semantics of high-uncertainty image blocks based on the weighted similarity to impose constraints on the local semantics of the high-uncertainty image blocks in the future, thereby improving the reliability of local image-text alignment; wherein, a high-uncertainty image block is the image block corresponding to the calculated variance value of the similarity between different image block-text pairs being greater than a preset variance value; a low-uncertainty image block is the image block corresponding to the calculated variance value of the similarity between different image block-text pairs being less than or equal to a preset variance value; Step S3: Extract the image CLS token and text CLS token, then calculate the cosine similarity between the image CLS token and the text CLS token, and construct the corresponding cosine similarity matrix B based on the cosine similarity. Then, using distribution modeling techniques, map the semantic features corresponding to the image CLS token and text CLS token from the embedding space to the distribution space to generate image distributions respectively. I n and text distribution T n Then calculate the image distribution. I n With text distribution T n The Wasserstein-2 distance between the images and text is used to construct an uncertainty matrix C, and then a constraint is imposed on the global semantic similarity of the images and text based on the uncertainty matrix C; wherein, each element of the uncertainty matrix C reflects the alignment uncertainty of different image-text pairs at the global semantic level. Step S4: Input the unlabeled spine MR slices in the dataset into the pre-trained image encoder from the pre-trained GLsc to obtain the corresponding image embeddings; Step S5: Input the text prompts of a certain category of the labeled spinal MR slices in the dataset into the pre-trained text encoder from the pre-trained GLsc to generate the corresponding text embeddings. Then, calculate the confidence score of the current input corresponding to the certain category based on the image embeddings and text embeddings. Then, filter out the image regions with high confidence based on the confidence score to generate the prompt guidance mask for the certain category. The high confidence image regions are the image regions corresponding to the calculated confidence score of the certain category being greater than the preset confidence score. Step S6: Repeat step S5 for text cues of other categories of labeled spinal MR slices in the dataset to obtain cue guidance masks for each category in the dataset. Then, stack the cue guidance masks for each category in the dataset based on the confidence score to form the final cue guidance mask. Step S7: Based on the final cue guidance mask, calculate the feature supervision loss and region alignment loss to strengthen cross-modal semantic constraints; Step S8: Apply the final cue guidance mask to the unlabeled spinal MR slices in the dataset, and input the unlabeled spinal MR slices after applying the cue guidance mask into the student model to generate unlabeled predictions. Also input the labeled spinal MR slices in the dataset into the student model to generate labeled predictions. Input the unlabeled spinal MR slices in the dataset into the teacher model to generate pseudo-labels. Then, fuse the unlabeled predictions, labeled predictions, and pseudo-labels to generate merged pseudo-labels. The teacher model is updated by the student model using an exponential moving average. Step S9: Iteratively optimize the segmentation network based on supervised loss and consistency loss to achieve a high-quality spine segmentation result; wherein the segmentation network consists of a teacher model and a student model.

2. The semi-supervised spine segmentation method based on a global-local semantic constraint visual language model according to claim 1, characterized in that, In step S1, the spine MR slices of the dataset are processed using an image encoder to obtain the corresponding image embeddings. Specifically, the image encoder extracts multi-scale, multi-level image features from the spine MR slices of the dataset to generate the image patch tokens and the image CLS token. The image patch tokens correspond to local image region features at different resolutions; the image CLS token is a feature identifier that integrates global image information. The text describing the spinal structure is processed using a text encoder to generate a corresponding text embedding. Specifically, the text encoder processes the text describing the spinal structure through word embedding and positional encoding to output a text CLS token. The text CLS token is a comprehensive representation of the text semantics.

3. The semi-supervised spine segmentation method based on a global-local semantic constraint visual language model according to claim 1, characterized in that, In step S2, the variance of the similarity between different image blocks and text pairs is calculated based on the cosine similarity matrix A. Specifically, based on the cosine similarity matrix A, the similarity score between each image block token and the text CLS token is calculated, and then the variance of the similarity between different image blocks and text pairs is calculated within the set of similarity scores for all image blocks and text pairs. as well as When calculating weighted similarity by combining variance, the image block-text pair similarity score with high uncertainty is assigned a first weight, and the image block-text pair similarity score with low uncertainty is assigned a second weight. Among them, the image block-text pair with high uncertainty is the image block-text matching pair formed by image blocks with variance greater than a preset variance value; the image block-text pair with low uncertainty is the image block-text matching pair formed by image blocks with variance less than or equal to a preset variance value.

4. The semi-supervised spine segmentation method based on a global-local semantic constraint visual language model according to claim 1, characterized in that, In step S6, the cue guidance masks of each category in the dataset are stacked based on the confidence score to form the final cue guidance mask. Specifically, the cue guidance masks of each category in the dataset are stacked in descending order of confidence score, and the cue guidance masks of a single category are weighted and superimposed.

5. The semi-supervised spine segmentation method based on a global-local semantic constraint visual language model according to claim 1, characterized in that, Step S7 specifically includes: Step S71: Embed the image patch corresponding to the final prompt guidance mask and associate it with the text global CLS token to calculate the feature supervision loss, so as to measure the feature difference between the image patch embedding corresponding to the final prompt guidance mask and the text global CLS token; Step S72: Extract the region image corresponding to the final prompt guidance mask, and compare the region image with the text global CLS token to calculate the region alignment loss, so as to enhance the alignment between the image region semantics and the text global semantics.

6. The semi-supervised spine segmentation method based on a global-local semantically constrained visual language model according to any one of claims 1 to 5, characterized in that, In step S8, a weighted average is used to fuse the unlabeled predictions, labeled predictions, and pseudo-labels to generate merged pseudo-labels.

Images