Medical image segmentation method and system based on concept guidance and cross-modal alignment

Abstract

The application provides a medical image segmentation method and system based on concept guidance and cross-modal alignment, and belongs to the field of medical image processing. The method comprises the following steps: obtaining a medical image to be segmented and its corresponding clinical text description; generating clinical knowledge concepts related to the target disease by using a large language model, and constructing a concept set through clinical review; inputting the medical image, the clinical text description and the concept set into a trained concept-guided segmentation model to extract visual features, text features and concept labels; generating concept features aligned with the visual features through a concept-visual alignment module; dynamically adjusting the normalization process of the visual features through a concept modulation decoder, combining the features through multi-head cross attention, and outputting the final image segmentation result by using a segmentation head. The application effectively solves the problems of lack of effective clinical prior guidance in existing medical image segmentation, poor cross-modal feature alignment, and insufficient lesion segmentation accuracy.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a medical image segmentation method and system based on concept-guided and cross-modal alignment. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Chest imaging, particularly chest X-ray (CXR) and computed tomography (CT), plays a crucial role in the clinical diagnosis and disease monitoring of lung diseases. Accurately segmenting lesion regions from these medical images is a key prerequisite for quantitative analysis, treatment planning, and prognostic assessment. In recent years, deep learning-based medical image segmentation techniques have made significant progress. However, due to the diverse morphologies of lung lesions, their weak contrast with surrounding normal tissue, and their frequent embedding within complex anatomical backgrounds, achieving high-precision automated segmentation remains a highly challenging task.

[0004] To improve segmentation performance, researchers have begun exploring multimodal learning methods aimed at combining medical images with their accompanying clinical text descriptions. The text descriptions provide high-level semantic cues that complement visual features. Examples include visual-language fusion using local text annotation and a CNN-Transformer hybrid architecture; and the existing GuideDecoder model performing hierarchical cross-modal feature fusion throughout the decoder.

[0005] While the methods described above improve segmentation results to some extent, they still have significant limitations. First, available text supervision is typically coarse-grained, focusing on a rough description of lesion location while lacking task-level attributes closely related to precise lesion delineation. This coarse-grained semantic guidance struggles to handle complex lesions with heterogeneous morphology and blurred edges. Second, because natural language descriptions themselves lack explicit spatial localization information, the alignment between textual cues and image regions is often weakly supervised and ambiguous. This alignment inaccuracy limits the model's ability to effectively map semantic prior knowledge to pixel-level segmentation tasks, especially when lesions are small or have extremely low contrast, leading to a significant drop in segmentation accuracy. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a medical image segmentation method and system based on concept guidance and cross-modal alignment, aiming to solve the problems of insufficient semantic guidance and weak alignment of text and image features in existing medical image segmentation methods.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a medical image segmentation method based on concept-guided and cross-modal alignment; Medical image segmentation methods based on concept-guided and cross-modal alignment include: Obtain the medical image to be segmented and its corresponding clinical text description; Large language models are used to generate clinical knowledge concepts related to the target disease, and a concept set is constructed after clinical review. The medical images, clinical text descriptions, and concept sets are input into a trained concept-guided segmentation model, which outputs image segmentation results. Among them, the trained concept-guided segmentation model extracts multi-scale visual features of medical images through a visual encoder, extracts text features of clinical text descriptions through a text encoder, and extracts structured concept tags from the concept set using a concept encoder. The conceptual features and the patch-level visual features output by the visual encoder are input into the concept-visual alignment module to generate conceptual features aligned with the visual features; The aligned conceptual features, text features, and multi-scale visual features are input into the conceptual modulation decoder. The normalization process of the visual features is dynamically adjusted through modulation parameters. The modulated visual features are combined with the text features through multi-head cross attention. The final image segmentation result is output using the segmentation head.

[0008] As a further technical solution, a large language model is used to generate clinical knowledge concepts related to the target disease, and a concept set is constructed after clinical review, including: The large language model is guided by prompt words to generate conceptual phrases that capture the morphology, edge clarity, internal texture, and spatial distribution attributes of lesions. By scoring and filtering the generated concept phrases, low-scoring concepts are eliminated and prompts are optimized and regenerated until a high-quality concept set is obtained.

[0009] As a further technical solution, the conceptual features and the patch-level visual features output by the visual encoder are input into the concept-visual alignment module to generate conceptual features aligned with the visual features, including: Calculate the cosine similarity between each concept tag and all visual patches; The cosine similarity is normalized at the patch dimension to obtain the attention weight; The attention weights are used to perform a weighted summation of all visual patches to synthesize the conceptual conditional visual evidence corresponding to each conceptual tag; By introducing a bidirectional contrastive loss function, we can bring the matching concepts closer together – visual pairs – and push away mismatched pairs, thus obtaining conceptual features that are aligned with visual features.

[0010] As a further technical solution, the process of generating the final image segmentation result using a concept modulation decoder includes: Linear projection is performed on the text features and concept features of the current decoding stage to match the number of channels with the visual features of the current stage; Global average pooling and global max pooling are performed on the channel-aligned conceptual features, and the pooling results are concatenated and merged to form a unified global conceptual description. Channel scaling factor, bias term, and residual scaling factor are generated from the global concept description using a regression network; Using the channel scaling factor and bias term, the visual features of the current stage are subjected to layer normalization and adaptive modulation. The modulated visual features are used as queries, and the text features are used as keys and values. Multi-head cross-attention calculation is performed, and the attention calculation results are added to the original visual features after being weighted by the residual scaling factor to output the decoded fused features. The decoded and fused features are input into the segmentation head, and the final image segmentation result is output.

[0011] As a further technical solution, the channel scaling factor and bias term are used to perform layer normalization and adaptive modulation on the visual features of the current stage, as shown below:

[0012] in, The modulated visual features of the i-th layer; For layer normalization operation; For the visual features of the i-th layer; The product of Hadamard; This is the channel scaling factor; This is a bias term.

[0013] As a further technical solution, the concept-guided segmentation model is optimized using a composite objective function, which includes cross-entropy loss, Dice loss, and concept-visual alignment loss.

[0014] A second aspect of the present invention provides a medical image segmentation system based on concept guidance and cross-modal alignment.

[0015] A concept-guided and cross-modal alignment-based medical image segmentation system includes: The data acquisition module is configured to acquire the medical image to be segmented and its corresponding clinical text description; The concept set construction module is configured to: generate clinical knowledge concepts related to the target disease using a large language model, and construct a concept set after clinical review; The image segmentation module is configured to: input the medical image, clinical text description, and concept set into a trained concept-guided segmentation model, and output the image segmentation result; Among them, the trained concept-guided segmentation model extracts multi-scale visual features of medical images through a visual encoder, extracts text features of clinical text descriptions through a text encoder, and extracts structured concept tags from the concept set using a concept encoder. The conceptual features and the patch-level visual features output by the visual encoder are input into the concept-visual alignment module to generate conceptual features aligned with the visual features; The aligned conceptual features, text features, and multi-scale visual features are input into the conceptual modulation decoder. The normalization process of the visual features is dynamically adjusted through modulation parameters. The modulated visual features are combined with the text features through multi-head cross attention. The final image segmentation result is output using the segmentation head.

[0016] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of a concept-guided and cross-modal alignment-based medical image segmentation method as described in the first aspect of the present invention.

[0017] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a concept-guided and cross-modal alignment-based medical image segmentation method as described in the first aspect of the present invention.

[0018] The fifth aspect of the present invention provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the medical image segmentation method based on concept guidance and cross-modal alignment described in the first aspect of the present invention.

[0019] The above one or more technical solutions have the following beneficial effects: (1) To address the shortcomings of existing technologies, such as coarse-grained text supervision and a lack of detailed descriptions of lesion attributes, this invention introduces a structured concept set generated by a large language model and verified by clinicians. These concepts cover key attributes such as lesion morphology, edge sharpness, internal texture, and spatial distribution, providing richer and more accurate semantic priors for segmentation tasks than traditional coarse-grained location descriptions. By explicitly integrating these clinical concepts into the segmentation framework, the model can fully utilize expert knowledge in the field of radiology to effectively address challenging scenarios such as diverse lesion morphologies, blurred edges, and weak contrast, significantly improving the segmentation accuracy for complex and small-scale lesions.

[0020] (2) This invention constructs a concept-guided segmentation model (CGSM). Utilizing the concept-visual alignment module within the model, an explicit correspondence is established between each clinical concept and a local image patch through a similarity-based attention mechanism. A bidirectional contrastive learning strategy is employed to optimize the semantic consistency between concept representations and visual features. This effectively solves the cross-modal alignment ambiguity problem caused by the lack of spatial localization in existing text descriptions, ensuring that each concept is anchored to the most relevant image region. Furthermore, the close-and-slide mechanism of the contrastive loss function enhances the discriminability between different concepts and avoids feature collapse.

[0021] The concept modulation decoder breaks through the traditional fusion method of feature splicing or simple addition. It uses aligned clinical concepts as dynamic modulation signals to adaptively regulate the normalization process of visual features and the intensity of cross-modal interaction. It can regress channel scaling factors, bias terms, and residual scaling coefficients from the global concept description, achieving dynamic scaling and offsetting of visual features. The residual coefficients control the injection intensity of concept-guided text-visual fusion features. This enables the model to adaptively highlight task-relevant features and suppress background interference based on the prior clinical concepts corresponding to the input image, demonstrating excellent performance in lesion boundary delineation and detection of small lesions.

[0022] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] Figure 1 This is a flowchart of the method in the first embodiment.

[0025] Figure 2 This is a schematic diagram showing the comparison results of the method of the present invention in the first embodiment and the current medical image segmentation method on different datasets. Detailed Implementation

[0026] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0028] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0029] Example 1 This embodiment discloses a medical image segmentation method based on concept guidance and cross-modal alignment. By introducing radiological concepts generated by a large language model and clinically reviewed, and combining concept-visual alignment and concept modulation decoding mechanism, it achieves accurate matching and adaptive fusion of text concepts and visual features, improving the accuracy of lung lesion segmentation, especially the performance of lesion boundary delineation and small-scale lesion detection.

[0030] Specifically, such as Figure 1 As shown, a medical image segmentation method based on concept-guided and cross-modal alignment includes: Step S1: Obtain the medical image to be segmented and its corresponding clinical text description.

[0031] The medical images to be segmented are selected from chest X-ray (CXR) or lung computed tomography (CT) images, all of which are obtained from clinically standardized acquisition equipment. After acquisition, they are uniformly converted into standard formats, and invalid images with artifacts, blurring, or abnormal acquisition parameters are removed. The valid images are normalized to ensure that the pixel value grayscale range is consistent with the pre-training data of the visual encoder, thus completing the standardized preprocessing of medical images.

[0032] The corresponding clinical text description is a structured clinical radiology report paired one-to-one with medical images. Effective semantic information related to lung lesions is extracted from the report, and irrelevant content such as patient basic information and examination orders is removed. Core descriptive content such as lesion location, shape, texture, and distribution is retained. The retained text content is preprocessed with natural language processing, including word segmentation and stop word removal. The text sequence length is uniformly truncated or padded with zeros to a fixed length to generate standardized clinical text description data that meets the input requirements of the text encoder.

[0033] Finally, the preprocessed medical images and clinical text descriptions are precisely paired using unique image identifiers to form an image-text pair dataset, which is then stored in batches to a designated data path to ensure data synchronization and correspondence when called by subsequent models.

[0034] Step S2: Use a large language model to generate clinical knowledge concepts related to the target disease, and construct a concept set after clinical review.

[0035] In this embodiment, Gemini-3 is selected as the basic large language model. Based on the radiological diagnostic characteristics of lung target diseases, structured prompt words are designed. The model focuses on analyzing the core dimensions such as lesion distribution characteristics, affected lung area, texture density, and edge morphology. The model is required to extract key visual attributes based on the analysis results and generate concise and standardized conceptual phrases to ensure that the generated content accurately captures the four core attributes of lesion morphology, edge clarity, internal texture, and spatial distribution.

[0036] The designed prompt words are input into the large language model. Based on the radiological diagnosis logic, the model outputs a list of candidate concept phrases for the target disease. The phrases are all radiological professional terms, such as ground-glass opacity, consolidation, blurred margins, nodular opacity, distribution in the lower lobes of both lungs, and stony road sign, covering different manifestations of the four core attributes of the lesion, forming an initial candidate concept set.

[0037] Finally, clinicians independently and blindly reviewed the initial candidate conceptual phrases. The scoring dimensions included the phrase's match with the target disease lesion features, radiological professionalism, and its guiding value for the segmentation task. The average score for each conceptual phrase was used as the final score. A scoring threshold of 6 points was set, and low-scoring conceptual phrases with an average score below 6 were removed. These phrases were often vaguely described, had low relevance to lesion delineation, or did not contain typical features of the target disease.

[0038] For the low-scoring concept phrases that were removed, the reasons for their generation deviations were analyzed, and the initial prompts were optimized accordingly. This included refining the lesion attribute description requirements, adding examples of radiological terminology, and clarifying the phrase output format to reduce the ambiguity and bias in the model's generation. The optimized prompts were then input back into the large language model to regenerate candidate concept phrases. The clinical expert scoring and screening steps were repeated until the average score of all candidate concept phrases was ≥6 points, and the concept phrases fully covered the four core attributes of lesion morphology, edge clarity, internal texture, and spatial distribution, with no duplication or redundancy. Finally, the approved concept phrases were organized, standardized, and named to form a structured set of clinical knowledge concepts.

[0039] Step S3: Input the medical image, clinical text description, and concept set into the trained concept-guided segmentation model (CGSM) and output the image segmentation result.

[0040] Step S31, for the input image to be segmented In this embodiment, ConvNeXt-Tiny is used as the visual encoder to extract multi-scale visual features from a four-level feature pyramid. ,in For the aforementioned visual features, the focus is on extracting the patch-level visual features from the last layer's output. This lays the foundation for the subsequent concept of visual alignment.

[0041] Paired clinical text descriptions are input into a frozen text encoder, CXR-BERT, for language encoding. For paired texts, contextual labeling features are extracted directly using a fully frozen CXR-BERT. ,in Indicates the label length. For concept phrases, the same frozen CXR-BERT is reused as the backbone network, and a trainable projection head concept encoder is further added to extract structured concept label features adapted to the segmentation task, so that the concept representation is adapted to the segmentation task. Formally, it is defined as:

[0042] in, Features for labeling concepts; For trainable projection heads; Represents the tokenized concept phrases, each and The number of concept tags.

[0043] Step S32: In order to anchor concept tags to visual features and highlight task-related concepts during decoding, a concept-visual alignment module (CVAM) is introduced in this embodiment, which aims to explicitly align concept tags with patch-level features of the image.

[0044] Specifically, given the patch features of the last layer of the visual encoder and concept tag embedding CVAM establishes a marker-patch correspondence through a similarity-based attention mechanism, aiming to identify the most relevant visual region for each clinical concept. Specifically, it first calculates the cosine similarity between each concept marker and all visual patches, then normalizes it along the patch dimension to obtain attention weights.

[0045]

[0046] in, Cosine similarity between concept tags and visual patches; The normalized cosine similarity; It is a temperature parameter.

[0047] Ultimately, the generated conceptual conditional visual evidence is synthesized as a weighted sum of all patches:

[0048] in, For conceptual conditional visual features; This represents the number of visual patches.

[0049] While this soft attention anchoring mechanism can capture local evidence, it is susceptible to feature collapse, meaning different concept labels may focus on similar redundant visual regions. To enhance the discriminative representation of the labels, this embodiment also introduces a bidirectional contrastive loss designed based on the InfoNCE formula. The objective function aims to bring the concept of visual pair closer together. Simultaneously push out mismatched pairs:

[0050] in, This is a two-way contrastive loss function; The number of concept tags; and These are the loss from concept-conditional visual features to concept features, and the inverse contrast loss, respectively. They can be represented as:

[0051]

[0052] Finally, by minimizing The model can prompt each set of visual evidence Its corresponding clinical concept To maintain consistency and enhance the semantic consistency between concept tags and visual features, the output concept features C are aligned using a bidirectional contrastive loss function.

[0053] Step S33: Based on the aligned concept features, a concept modulation decoder (CM-Decoder) is introduced to integrate visual features through concept conditional normalization and cross-modal interaction. Text features The conceptual feature of alignment, C.

[0054] In each decoding stage, the text and concept embeddings are first linearly projected to match visual features. The channel dimension. To obtain a compact global description from the concept tag set, a linear projection is performed on the aligned concept features C to match the visual features. By determining the channel dimensions, conceptual features consistent with the visual feature channel dimensions are obtained. Subsequently, the conceptual features were analyzed. Perform global pooling operations, specifically global average pooling (GAP) and global max pooling (GMP):

[0055]

[0056] in, These are the conceptual features after global average pooling; These are the conceptual features resulting from global max pooling. For channel dimensions and visual features Matching conceptual features; and It is a learnable projection matrix. Capture global context statistics, and This emphasizes and activates the most prominent concepts.

[0057] The two are further integrated to form a unified global conceptual description:

[0058] in, This is a description of a global concept. This indicates a concatenation operation. This aggregation method captures both global contextual statistics and the most salient concept representations.

[0059] Furthermore, instead of using fixed learnable scaling and bias parameters in layer normalization, this embodiment introduces a conceptual representation. The concept of driving the modulation strategy is used. Specifically, three modulation parameters are controlled by a regression network based on linear layers. The regression yielded that its output was subsequently split into channel scaling factors. Bias terms and residual scaling factor These parameters dynamically control how visual features are normalized and how they are integrated with the textual context.

[0060] For the Layer decoder, visual features First, it is normalized and then adaptively modulated:

[0061] in, The modulated visual features of the i-th layer; Representation layer normalization; This represents the Hadamard product. Unlike standard normalization where scaling and bias parameters are fixed after training, the modulation strategy described above allows these parameters to be adjusted according to... Changes occur, thereby injecting clinically relevant prior knowledge into the feature space.

[0062] The modulated features were then combined with the text representation via multi-head cross-attention (MHCA):

[0063] in, This indicates a multi-headed cross-attention operation. Represents text embedding, This is the output of the i-th layer decoder. Residual coefficients. Further modulating the contribution of cross-modal interactions enables adaptive control of concept guidance strength. Through concept modulation normalization and adaptive residual scaling, the decoder can dynamically balance visual evidence and semantic priors, thereby achieving more robust and semantically consistent segmentation.

[0064] After multi-level decoding, the final decoded fusion feature, which integrates visual, textual, and conceptual information, is obtained. This final decoded fusion feature, output from the concept modulation decoder, is input into the model's segmentation head. After pixel-level classification and feature mapping, a segmentation mask map with the same resolution as the input medical image is generated. Different pixel values ​​in the mask map correspond to normal lung tissue and lesion areas, respectively, achieving accurate delineation of lung lesions. The final output image segmentation result can be directly connected to a clinical imaging diagnostic system for visualization and subsequent analysis.

[0065] For the constructed concept-guided segmentation model, a composite objective function is also used for optimization. This function performs three optimization tasks: pixel-level accuracy, region-level consistency, and concept-visual alignment. The total loss function... The formula is as follows:

[0066] in, , and It is a weighting coefficient that balances the contributions of each entity. and Let represent the cross-entropy loss and Dice loss, respectively, which supervise pixel-level classification accuracy and the consistency of region overlap. In addition, ... Enhance the semantic alignment between conceptual representations and visual features, thereby facilitating segmentation with conceptual awareness.

[0067] Finally, to verify the performance of the concept-guided segmentation model constructed in this invention, the method was evaluated on two public datasets: QaTa-COV19 and MosMedData+. The QaTa-COV19 dataset contains 9258 COVID-19 chest X-ray (CXR) images with expert-level pixel-labeled lesion annotations and predefined training / validation / test splits (5716 / 1429 / 2113 images, respectively). The MosMedData+ dataset consists of 2729 CT slices (split ratio of 2183 / 273 / 273), each slice with a lesion mask. In addition, both datasets provide paired structured medical reports to support multimodal segmentation tasks.

[0068] like Figure 2 As shown, the proposed CGSM was compared with current state-of-the-art medical image segmentation methods on the QaTa-COV19 and MosMedData+ datasets. The proposed method consistently outperforms existing methods on all metrics across both datasets. Specifically, compared to the current method MMI-UNet, CGSM improves the Dice score by 0.71% on the QaTa-COV19 dataset and by 0.66% on the MosMedData+ dataset.

[0069] Example 2 This embodiment discloses a medical image segmentation system based on concept guidance and cross-modal alignment; A concept-guided and cross-modal alignment-based medical image segmentation system includes: The data acquisition module is configured to acquire the medical image to be segmented and its corresponding clinical text description; The concept set construction module is configured to: generate clinical knowledge concepts related to the target disease using a large language model, and construct a concept set after clinical review; The image segmentation module is configured to: input the medical image, clinical text description, and concept set into a trained concept-guided segmentation model, and output the image segmentation result; Among them, the trained concept-guided segmentation model extracts multi-scale visual features of medical images through a visual encoder, extracts text features of clinical text descriptions through a text encoder, and extracts structured concept tags from the concept set using a concept encoder. The conceptual features and the patch-level visual features output by the visual encoder are input into the concept-visual alignment module to generate conceptual features aligned with the visual features; The aligned conceptual features, text features, and multi-scale visual features are input into the conceptual modulation decoder. The normalization process of the visual features is dynamically adjusted through modulation parameters. The modulated visual features are combined with the text features through multi-head cross attention. The final image segmentation result is output using the segmentation head.

[0070] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0071] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of a concept-guided and cross-modal alignment-based medical image segmentation method as described in Example 1.

[0072] Example 4 The purpose of this embodiment is to provide an electronic device.

[0073] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a concept-guided and cross-modal alignment-based medical image segmentation method as described in Embodiment 1.

[0074] Example 5 Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in a concept-guided and cross-modal alignment-based medical image segmentation method as described in Embodiment 1.

[0075] The steps and methods involved in the apparatuses of Embodiments 2, 3, 4, and 5 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0076] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0077] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A medical image segmentation method based on concept-guided and cross-modal alignment, characterized in that, include: Obtain the medical image to be segmented and its corresponding clinical text description; Large language models are used to generate clinical knowledge concepts related to the target disease, and a concept set is constructed after clinical review. The medical images, clinical text descriptions, and concept sets are input into a trained concept-guided segmentation model, which outputs image segmentation results. Among them, the trained concept-guided segmentation model extracts multi-scale visual features of medical images through a visual encoder, extracts text features of clinical text descriptions through a text encoder, and extracts structured concept tags from the concept set using a concept encoder. The conceptual features and the patch-level visual features output by the visual encoder are input into the concept-visual alignment module to generate conceptual features aligned with the visual features; The aligned conceptual features, text features, and multi-scale visual features are input into the conceptual modulation decoder. The normalization process of the visual features is dynamically adjusted through modulation parameters, and the modulated visual features are combined with the text features through multi-head cross-attention. The final image segmentation result is output using the segmentation head. Specifically, the text features and conceptual features in the current decoding stage are linearly projected to match the number of channels with the visual features in the current stage. Global average pooling and global max pooling are performed on the channel-aligned conceptual features, and the pooling results are concatenated and fused to form a unified global conceptual description. Channel scaling factors, bias terms, and residual scaling coefficients are generated from the global conceptual description through a regression network. The channel scaling factors and bias terms are used to perform layer normalization and adaptive modulation on the visual features in the current stage. The modulated visual features are used as queries, and the text features are used as keys and values. Multi-head cross-attention is calculated, and the attention calculation result is weighted with the original visual features by the residual scaling coefficient and added to output the decoded fusion feature. The decoded fusion feature is input into the segmentation head to output the final image segmentation result.

2. The medical image segmentation method based on concept-guided and cross-modal alignment as described in claim 1, characterized in that, Large language models are used to generate clinical knowledge concepts related to the target disease, and a concept set is constructed after clinical review, including: The large language model is guided by prompt words to generate conceptual phrases that capture the morphology, edge clarity, internal texture, and spatial distribution attributes of lesions. By scoring and filtering the generated concept phrases, low-scoring concepts are eliminated and prompts are optimized and regenerated until a high-quality concept set is obtained.

3. The medical image segmentation method based on concept-guided and cross-modal alignment as described in claim 1, characterized in that, The conceptual features and the patch-level visual features output by the visual encoder are input into the concept-visual alignment module to generate conceptual features aligned with the visual features, including: Calculate the cosine similarity between each concept tag and all visual patches; The cosine similarity is normalized at the patch dimension to obtain the attention weight; The attention weights are used to perform a weighted summation of all visual patches to synthesize the conceptual conditional visual evidence corresponding to each conceptual tag; By introducing a bidirectional contrastive loss function, we can bring the matching concepts closer together – visual pairs – and push away mismatched pairs, thus obtaining conceptual features that are aligned with visual features.

4. The medical image segmentation method based on concept-guided and cross-modal alignment as described in claim 1, characterized in that, Using the aforementioned channel scaling factor and bias term, the visual features of the current stage are subjected to layer normalization and adaptive modulation, as shown below: in, The modulated visual features of the i-th layer; For layer normalization operation; For the visual features of the i-th layer; The product of Hadamard; This is the channel scaling factor; This is a bias term.

5. The medical image segmentation method based on concept-guided and cross-modal alignment as described in claim 1, characterized in that, The concept-guided segmentation model is optimized using a composite objective function, which includes cross-entropy loss, Dice loss, and concept-visual alignment loss.

6. A medical image segmentation system based on concept-guided and cross-modal alignment, characterized in that, include: The data acquisition module is configured to acquire the medical image to be segmented and its corresponding clinical text description; The concept set construction module is configured to: generate clinical knowledge concepts related to the target disease using a large language model, and construct a concept set after clinical review; The image segmentation module is configured to: input the medical image, clinical text description, and concept set into a trained concept-guided segmentation model, and output the image segmentation result; Among them, the trained concept-guided segmentation model extracts multi-scale visual features of medical images through a visual encoder, extracts text features of clinical text descriptions through a text encoder, and extracts structured concept tags from the concept set using a concept encoder. The conceptual features and the patch-level visual features output by the visual encoder are input into the concept-visual alignment module to generate conceptual features aligned with the visual features; The aligned conceptual features, text features, and multi-scale visual features are input into the conceptual modulation decoder. The normalization process of the visual features is dynamically adjusted through modulation parameters, and the modulated visual features are combined with the text features through multi-head cross-attention. The final image segmentation result is output using the segmentation head. Specifically, the text features and conceptual features in the current decoding stage are linearly projected to match the number of channels with the visual features in the current stage. Global average pooling and global max pooling are performed on the channel-aligned conceptual features, and the pooling results are concatenated and fused to form a unified global conceptual description. Channel scaling factors, bias terms, and residual scaling coefficients are generated from the global conceptual description through a regression network. The channel scaling factors and bias terms are used to perform layer normalization and adaptive modulation on the visual features in the current stage. The modulated visual features are used as queries, and the text features are used as keys and values. Multi-head cross-attention is calculated, and the attention calculation result is weighted with the original visual features by the residual scaling coefficient and added to output the decoded fusion feature. The decoded fusion feature is input into the segmentation head to output the final image segmentation result.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the medical image segmentation method based on concept guidance and cross-modal alignment as described in any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the medical image segmentation method based on concept guidance and cross-modal alignment as described in any one of claims 1-5.

9. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instructions implement the steps in the concept-guided and cross-modal alignment-based medical image segmentation method as described in any one of claims 1-5.

Images