Automatic lung nodule labeling method based on multimodal semantic affinity iterative optimization

Abstract

The embodiment of the present application provides a kind of lung nodule automatic labeling method based on multimodal semantic affinity iterative optimization, belong to data processing technical field, specifically include: obtaining lung CT image and text report;Text report is carried out structured semantic analysis;Lung CT image is carried out visual feature extraction;Based on key semantic information, semantic guided processing is carried out to deep visual feature representation, and semantic enhanced visual guided feature is generated;Based on semantic enhanced visual guided feature, call segmentation base model, and generate multiple candidate segmentation masks by introducing active variation mechanism;For each candidate segmentation mask, the semantic matching degree score of the candidate segmentation mask is calculated;The semantic matching degree score of all candidate segmentation masks is compared, and the candidate segmentation mask with the highest score is selected as the final lung nodule labeling result output.The scheme of the present application improves the labeling accuracy, adaptability and interpretability.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an automatic annotation method for lung nodules based on iterative optimization of multimodal semantic affinity. Background Technology

[0002] Currently, accurate annotation of lung nodules is a crucial step in the early auxiliary diagnosis of lung cancer. To improve efficiency, automatic annotation methods based on deep learning have emerged. Existing methods can be divided into two categories: one is a visual model that relies solely on CT images, whose segmentation results cannot be semantically aligned with the text reports describing nodule characteristics; the other is a multimodal method that attempts to integrate images and text, often using feature concatenation or attention mechanisms for one-time prediction, lacking quantitative evaluation and iterative optimization mechanisms for fine-grained semantic consistency between segmentation results and text descriptions.

[0003] It is evident that there is an urgent need for an automatic lung nodule annotation method based on multimodal semantic affinity iterative optimization that can utilize cross-modal semantic matching for iterative optimization, thereby automatically generating a high degree of consistency with the text description. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an automatic lung nodule annotation method based on multimodal semantic affinity iterative optimization, which at least partially solves the problems of poor annotation efficiency and accuracy in the prior art.

[0005] This invention provides an automatic lung nodule annotation method based on multimodal semantic affinity iterative optimization, comprising:

[0006] Step 1: Obtain the lung CT images to be annotated and the corresponding text reports describing the features of the lung nodules;

[0007] Step 2: Perform structured semantic parsing on the text report to extract key semantic information about nodule location and morphological features.

[0008] Step 3: Extract visual features from lung CT images to obtain depth visual feature representation;

[0009] Step 4: Based on key semantic information, perform semantic guidance processing on the deep visual feature representation to generate semantically enhanced visual guidance features;

[0010] Step 5: Based on the semantically enhanced visual guidance features, call the basic segmentation model and generate multiple candidate segmentation masks by introducing an active mutation mechanism;

[0011] Step 6: For each candidate segmentation mask, use a pre-trained cross-modal semantic matching model to calculate the cosine similarity between the image region features corresponding to the candidate segmentation mask and the text description features corresponding to the text report, and use it as the semantic matching score of the candidate segmentation mask.

[0012] Step 7: Compare the semantic matching scores of all candidate segmentation masks, and select the candidate segmentation mask with the highest score as the final lung nodule annotation result output.

[0013] According to a specific implementation of an embodiment of the present invention, step 2 specifically includes:

[0014] Based on the parsing function, the text report is parsed into a structured semantic information set S consisting of K triples as key semantic information using a preset parsing method. The triples are... ,in, Indicates the medical entity being described. This represents the attribute dimensions of a medical entity. This represents the specific descriptive value of the attribute dimension. This includes location, size, density, shape, and edge features.

[0015] According to a specific implementation of an embodiment of the present invention, the preset parsing method includes an implicit parsing method based on a large language model and an explicit parsing method based on rules and knowledge base;

[0016] The implicit parsing method based on a large language model includes:

[0017] After embedding the text report into a predefined instruction prompt template, input it into a large language model to obtain a JSON output containing triples, and then convert the JSON output into a structured semantic information set S.

[0018] The explicit parsing method based on rules and knowledge base includes:

[0019] The text report is segmented and subjected to dependency parsing. Pattern matching is performed based on a predefined medical description dictionary to identify attribute values. As specific descriptive values ​​for attribute dimensions, and determined based on syntactic dependency relations. Modified medical entities , to form a triplet.

[0020] According to a specific implementation of an embodiment of the present invention, the active mutation mechanism includes at least one of the following strategies:

[0021] The cue variation strategy is as follows: the initial cue information is determined based on visual guidance features, and multiple different cue information are generated by randomly perturbing the position and / or scale parameters of the initial cue information and input into the segmentation base model respectively.

[0022] Confidence threshold variation strategy: Fix the prompt information of the input segmentation base model, and binarize the probability map output by the model by setting a set of different thresholds to generate multiple different candidate segmentation masks;

[0023] Strategy for utilizing internal randomness within the model: By setting different random seeds for multiple calls to the basic segmentation model, multiple different candidate segmentation masks can be obtained.

[0024] According to a specific implementation of an embodiment of the present invention, the cue mutation strategy specifically includes:

[0025] Initial bounding box cues are derived from visually guided features:

[0026] ;

[0027] in, and These represent the x and y coordinates of the top-left vertex of the bounding box in the image, respectively. and These represent the width and height of the bounding box, respectively.

[0028] Through formula Generate the first A variation bounding box prompt ,in, and For random position offsets within a preset range, and The random scale scaling factor is within a preset range;

[0029] Prompt each variant bounding box Each of these is used as input to call the basic segmentation model to generate the corresponding candidate segmentation mask.

[0030] According to a specific implementation of an embodiment of the present invention, step 6 specifically includes:

[0031] Step 6.1, for the i-th candidate segmentation mask ,according to The pixel coordinates marked as foreground are used to crop the corresponding image region from the lung CT image. ;

[0032] Step 6.2, divide the image region The image encoder of the cross-modal semantic matching model is input to obtain the image feature embedding vector. ;

[0033] Step 6.3: Input the text report into the text encoder of the cross-modal semantic matching model to obtain the text feature embedding vector. ;

[0034] Step 6.4, calculate the image feature embedding vector. With text feature embedding vector The cosine similarity between them is used as a candidate segmentation mask. semantic matching score .

[0035] According to a specific implementation of an embodiment of the present invention, the cross-modal semantic matching model is a visual-language dual encoder model pre-trained through contrastive learning on a large-scale image-text pair dataset.

[0036] The automatic lung nodule annotation scheme based on multimodal semantic affinity iterative optimization in this embodiment of the invention includes: Step 1, acquiring the lung CT image to be annotated and the corresponding text report describing the features of the lung nodules; Step 2, performing structured semantic parsing on the text report to extract key semantic information on the location and morphological features of the nodules contained therein; Step 3, extracting visual features from the lung CT image to obtain a depth visual feature representation; Step 4, performing semantic guidance processing on the depth visual feature representation based on the key semantic information to generate semantically enhanced visual guidance features; Step 5, calling the basic segmentation model based on the semantically enhanced visual guidance features and generating multiple candidate segmentation masks by introducing an active mutation mechanism; Step 6, for each candidate segmentation mask, using a pre-trained cross-modal semantic matching model, calculating the cosine similarity between the image region features corresponding to the candidate segmentation mask and the text description features corresponding to the text report, as the semantic matching score of the candidate segmentation mask; Step 7, comparing the semantic matching scores of all candidate segmentation masks and selecting the candidate segmentation mask with the highest score as the final lung nodule annotation result output.

[0037] The beneficial effects of the embodiments of the present invention are as follows: Through the solution of the present invention, a closed-loop optimization process of "generation-sorting" is constructed. Multiple candidate segmentation results are quantitatively evaluated and dynamically selected through a cross-modal semantic matching model, thereby achieving high-precision semantic alignment between the annotation results and the medical text description, and improving the adaptability and interpretability of the annotation process. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating an automatic lung nodule annotation method based on multimodal semantic affinity iterative optimization, provided as an embodiment of the present invention. Detailed Implementation

[0040] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0041] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0042] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this invention, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0043] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0044] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0045] Accurate labeling of lung nodules is a crucial step in the early auxiliary diagnosis of lung cancer. Current technologies heavily rely on manual delineation by radiologists, a method that is inefficient and suffers from subjective differences and poor consistency.

[0046] To improve efficiency, automatic annotation methods based on deep learning have emerged. Existing methods can be divided into two categories: one is a visual model that relies solely on CT images, whose segmentation results cannot be semantically aligned with the text reports describing nodule characteristics; the other is a multimodal method that attempts to integrate images and text, which often uses feature stitching or attention mechanisms for one-time prediction, lacking a quantitative evaluation and iterative optimization mechanism for fine-grained semantic consistency between segmentation results and text descriptions.

[0047] Specifically, existing technologies suffer from the following shortcomings: First, existing visual models or simple multimodal fusion methods struggle to transform complex textual semantics such as "spiculated" or "lobed" into precise constraints on pixel-level segmentation boundaries. Second, while general segmentation models, such as the Segment Anything Model, can generate candidate regions in medical scenarios, their outputs are singular or lack semantic guidance, failing to adaptively select the most suitable result from multiple possibilities. Third, the capabilities of visual-language models, such as the CLIP model, have not been effectively utilized in multimodal medical annotation tasks and have not yet been used as a "semantic arbitrator" for real-time evaluation and optimization of segmentation candidates.

[0048] Therefore, existing technologies struggle to achieve highly semantically consistent and adaptive automatic annotation of lung nodules. There is an urgent need for a pixel-level annotation scheme that can utilize cross-modal semantic matching for iterative optimization, thereby automatically generating annotations that highly match the text description.

[0049] This invention provides an automatic lung nodule annotation method based on multimodal semantic affinity iterative optimization, which can be applied to the lung nodule annotation process in medical scenarios.

[0050] See Figure 1 This is a flowchart illustrating an automatic lung nodule annotation method based on multimodal semantic affinity iterative optimization, as provided in an embodiment of the present invention. Figure 1 As shown, the method mainly includes the following steps:

[0051] Step 1: Obtain the lung CT images to be annotated and the corresponding text reports describing the features of the lung nodules;

[0052] Step 2: Perform structured semantic parsing on the text report to extract key semantic information about nodule location and morphological features.

[0053] In practice, this step aims to transform free-text medical reports into structured semantic instructions that machines can precisely execute. Unlike traditional methods that only extract keywords or classify text, this step achieves the transformation from unstructured reports to computable semantic triples. It provides precise input constraints for subsequent visual feature guidance and semantic evaluation. This step is fundamental to establishing fine-grained semantic alignment between text and images, and can be formally defined as a mapping function from the original text string to a structured semantic set.

[0054] ;

[0055] in, It is an analytic function, and its output is... For a by A set consisting of triplets. Each triplet Include:

[0056] main body : indicates the medical entity being described, which in this invention is typically a "nodule" or "lesion".

[0057] relation : Represents the attribute dimension of the entity, the set of relations Key descriptive dimensions in medical image reports, including but not limited to location, size, density, shape, and edge features.

[0058] Attribute value : Represents a specific description of the corresponding dimension, whose value space is determined by various relations. The corresponding medical descriptive dictionary Dri defines it.

[0059] Specifically, the analytic function is implemented. First, a domain knowledge base containing medical anatomical locations, lesion morphology terminology, and imaging features should be constructed or invoked. Then, one of the following two optional approaches can be used:

[0060] Implicit parsing based on the Large Language Model API: This method guides the Large Language Model to complete parsing by constructing prompt templates. This process can be formally represented as:

[0061] ;

[0062] in, This is a prompt string that embeds the raw text after a specific instruction (such as "extract as an array of triples in JSON format"). For large language model calls, This is a post-processing function that converts the JSON string output by the model into a set of triples.

[0063] Explicit parsing based on rules and knowledge bases: relies on a predefined medical knowledge base and natural language processing tools (such as dependency parsers). This process can be broken down into:

[0064] ;

[0065] ;

[0066] in, Perform word segmentation and dependency analysis on the text. Functions based on dictionary Perform pattern matching to identify attribute values , Functions based on syntactic dependency relations Determine the entity modified by this attribute value. .

[0067] For example, given the input text "Ground-glass nodules were observed in the upper lobe of the left lung, with spiculated edges," applying the above parsing function yields a set of structured semantic information: ;

[0068] Finally, all parsed triples are integrated and standardized to form a structured semantic representation that can be used to drive subsequent models.

[0069] Step 3: Extract visual features from lung CT images to obtain depth visual feature representation;

[0070] Step 4: Based on key semantic information, perform semantic guidance processing on the deep visual feature representation to generate semantically enhanced visual guidance features;

[0071] In practice, based on the key semantic information extracted in step 2, semantic guidance processing is performed on the deep visual feature representation obtained in step 3. This step aims to achieve early fusion of cross-modal information, injecting textual semantics as prior knowledge into the visual processing flow, so that visual features focus on regions and attributes related to the text description, thereby generating visual guidance features with clear semantic orientation.

[0072] Step 5: Based on the semantically enhanced visual guidance features, call the basic segmentation model and generate multiple candidate segmentation masks by introducing an active mutation mechanism;

[0073] In practical implementation, compared to the standard single-call of the segmentation base model (such as the Segment Anything Model), an "active variation" mechanism is introduced to transform it into a generator of diverse solutions. Specifically, after obtaining semantically enhanced visual guidance features, multiple calls to the segmentation base model are made by systematically changing the model's input cues (such as perturbing the initial bounding box coordinates) or adjusting internal parameters (such as the mask binarization threshold) to generate multiple candidate segmentation masks. This step is one of the key aspects of this invention, and its core lies in introducing the "active variation" mechanism. This mechanism aims to actively explore the solution space by performing multiple differentiated segmentation inference attempts on the same semantic target, thereby generating a diverse set of candidate masks. This provides a sufficient decision-making basis for subsequent optimization selection based on semantic affinity.

[0074] This process can be formally represented as:

[0075] ;

[0076] in, This is the basic model for segmentation. These are semantically enhanced visual guidance features input into the segmentation model. Representing the The specific input prompts or internal parameter variations used during the next inference call, where N is the preset number of candidate segmentation masks generated.

[0077] In practice, the underlying segmentation model is a prompt-based general image segmentation model, such as the Segment Anything Model or its medical image-optimized version, MedSAM. The key to generating multiple candidates lies in systematically altering the model's input or internal state during a single inference iteration. The main mutation strategies include:

[0078] Prompt Perturbation: This refers to the visual guidance feature... The implicit spatial attention focus is transformed into explicit cues (such as bounding boxes and points) acceptable to the segmentation model. Different inputs are generated by controllingly perturbing the initial cues. Let's assume... The derived initial region of interest is a bounding box. Then the j-th mutation box It can be obtained through the following formula:

[0079] ;

[0080] in, , This refers to the random position offset sampled within a preset range (e.g., ±5 pixels). This is a random scale scaling factor for sampling within a preset range (e.g., [0.9, 1.1]). Each input model is used as an independent box prompt to generate a candidate mask.

[0081] (2) Confidence Threshold Variation: With fixed input prompts, the predicted probability map is adjusted during model post-processing. Decision threshold for conversion to a binary mask By setting a set of different thresholds (For example, {0.3, 0.4, 0.5, 0.6, 0.7}), a series of masks with different boundary hardness and region size can be obtained:

[0082] ;

[0083] in This is an indicator function.

[0084] (3) Intrinsic Randomness Utilization: By utilizing the random sampling components (such as the randomness of point sampling) that may exist within the segmentation base model, different outputs can be obtained by setting different random seeds for each inference call.

[0085] By combining one or more of the above mutation strategies and repeatedly calling the segmentation model N times, a set of candidate masks with differences in boundary accuracy, region coverage, etc., can be obtained. This step, combined with the semantic guidance in step 4, ensures that the generated diverse candidates are within the scope of semantic constraints, thereby significantly improving the efficiency and reliability of selecting the optimal mask in subsequent steps.

[0086] Step 6: For each candidate segmentation mask, use a pre-trained cross-modal semantic matching model to calculate the cosine similarity between the image region features corresponding to the candidate segmentation mask and the text description features corresponding to the text report, and use it as the semantic matching score of the candidate segmentation mask.

[0087] In practical implementation, to address the lack of quantitative semantic evaluation standards in existing technologies, a pre-trained visual-language model is introduced as a "semantic arbitrator." Specifically: for each candidate segmentation mask, a visual-language matching model is used to calculate the semantic similarity between the image content corresponding to each candidate mask and the text description, obtaining a quantitative score. This step defines the core optimization objective of this invention—multimodal semantic affinity—and provides its quantitative calculation method, making the abstract "semantic consistency" measurable and comparable. The specific process is rigorously described as follows: First, a visual-language dual encoder model, such as the CLIP model or its biomedical-specific version BioMedCLIP, is pre-trained on a large-scale image-text pair dataset through contrastive learning. For the i-th candidate segmentation mask... Perform the following sub-steps:

[0088] (1) According to The pixel coordinates marked as foreground are used to crop the corresponding 3D region or the most representative 2D slice from the original lung CT image to obtain the image region. .

[0089] (2) Image region The image encoder input to the cross-modal semantic matching model We obtain a d-dimensional image feature embedding vector, denoted as... :

[0090] ;

[0091] (3) Input the original text report T obtained in step 1 into the text encoder of the same model. We obtain a d-dimensional text feature embedding vector, denoted as... ;

[0092] ;

[0093] The image encoder and text encoder map data from different modalities into the same shared d-dimensional semantic embedding space. Subsequently, image feature vectors are computed within this shared semantic space. With text feature vectors The cosine similarity between the two is used as the semantic matching score for the candidate mask. The complete definition of its calculation formula is as follows:

[0094] ;

[0095] Here, the operator represents the dot product operation of vectors. and Representing vectors respectively and The Euclidean norm (L2 norm) is calculated using the following formula:

[0096] ;

[0097] Where x represents a d-dimensional vector, xj represents the j-th component of vector x, and d is the dimension of the vector, corresponding to the embedding dimension of the vision-language matching model.

[0098] The rating It is a scalar with a range of [-1,1][-1,1]. The closer the value is to 1, the larger the image region corresponding to the candidate mask. The more semantically it matches and the more consistent it is with the text description T, the better.

[0099] Step 7: Compare the semantic matching scores of all candidate segmentation masks, and select the candidate segmentation mask with the highest score as the final lung nodule annotation result output.

[0100] In practice, the semantic matching scores corresponding to all candidate segmentation masks are compared, and the final annotation result is selected accordingly. This step completes the decision-making loop from "generating diverse candidates" to "selecting the best based on quantitative semantic standards," and is the output link for this method to achieve optimal selection. The specific mechanism is as follows: First, all semantic matching scores calculated in step 6 are collected to form a score set. Next, perform sorting and comparison operations to find the highest score. ,satisfy:

[0101] ;

[0102] Then, determine the candidate segmentation mask corresponding to the maximum score value, denoted as , i.e.:

[0103] ;

[0104] Ultimately, The optimal pixel-level segmentation result is output, thus completing the automatic annotation of lung nodules in the input lung CT images with high precision and high semantic consistency. This step makes decisions based on clear and quantifiable criteria, ensuring that the output of the entire annotation system has both high precision and high semantic consistency, and that its decision-making process is interpretable because the selection of the optimal result is directly based on its highest semantic affinity score with the clinical description.

[0105] Steps 5, 6, and 7 together form a complete iterative optimization closed loop of 'generation-quantitative evaluation-optimization', which is the core of this method to achieve adaptive semantic alignment.

[0106] By applying the method of this invention, adaptive optimization of a single inference task can be achieved without changing the internal parameters of the pre-trained model, effectively improving the semantic consistency between lung nodule annotation results and complex medical descriptions, and providing quantifiable explanatory basis for annotation decisions.

[0107] The automatic lung nodule annotation method based on multimodal semantic affinity iterative optimization provided in this embodiment constructs a closed-loop optimization process of "generation-ranking" and uses a cross-modal semantic matching model to quantitatively evaluate and dynamically select multiple candidate segmentation results, thereby achieving high-precision semantic alignment between the annotation results and the medical text description, and improving the adaptability and interpretability of the annotation process.

[0108] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof.

[0109] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An automatic lung nodule annotation method based on multimodal semantic affinity iterative optimization, characterized in that, include: Step 1: Obtain the lung CT images to be annotated and the corresponding text reports describing the features of the lung nodules; Step 2: Perform structured semantic parsing on the text report to extract key semantic information about nodule location and morphological features. Step 3: Extract visual features from lung CT images to obtain depth visual feature representation; Step 4: Based on key semantic information, perform semantic guidance processing on the deep visual feature representation to generate semantically enhanced visual guidance features; Step 5: Based on the semantically enhanced visual guidance features, the segmentation base model is invoked, and multiple candidate segmentation masks are generated by introducing an active mutation mechanism. This active mutation mechanism includes a cue-based mutation strategy, specifically: Initial bounding box cues are derived from visually guided features: in, and These represent the x and y coordinates of the top-left vertex of the bounding box in the image, respectively. and These represent the width and height of the bounding box, respectively. Through formula Generate the first A variation bounding box prompt ,in, and For random position offsets within a preset range, and The random scale scaling factor is within a preset range; Prompt each variant bounding box Each of these is used as input to call the basic segmentation model and generate the corresponding candidate segmentation mask. Step 6: For each candidate segmentation mask, use a pre-trained cross-modal semantic matching model to calculate the cosine similarity between the image region features corresponding to the candidate segmentation mask and the text description features corresponding to the text report, and use it as the semantic matching score of the candidate segmentation mask. Step 6 specifically includes: Step 6.1, for the i-th candidate segmentation mask ,according to The pixel coordinates marked as foreground are used to crop the corresponding image region from the lung CT image. ; Step 6.2, divide the image region The image encoder of the cross-modal semantic matching model is input to obtain the image feature embedding vector. ; Step 6.3: Input the text report into the text encoder of the cross-modal semantic matching model to obtain the text feature embedding vector. ; Step 6.4, calculate the image feature embedding vector. With text feature embedding vector The cosine similarity between them is used as a candidate segmentation mask. semantic matching score ; Step 7: Compare the semantic matching scores of all candidate segmentation masks, and select the candidate segmentation mask with the highest score as the final lung nodule annotation result output.

2. The method according to claim 1, characterized in that, Step 2 specifically includes: Based on the parsing function, the text report is parsed into a structured semantic information set S consisting of K triples as key semantic information using a preset parsing method. The triples are... ,in, Indicates the medical entity being described. This represents the attribute dimensions of a medical entity. This represents the specific descriptive value of the attribute dimension. This includes location, size, density, shape, and edge features.

3. The method according to claim 2, characterized in that, The preset parsing methods include implicit parsing methods based on large language models and explicit parsing methods based on rules and knowledge bases. The implicit parsing method based on a large language model includes: After embedding the text report into a predefined instruction prompt template, input it into a large language model to obtain a JSON output containing triples, and then convert the JSON output into a structured semantic information set S. The explicit parsing method based on rules and knowledge base includes: The text report is segmented and subjected to dependency parsing. Pattern matching is performed based on a predefined medical description dictionary to identify attribute values. As specific descriptive values ​​for attribute dimensions, and determined based on syntactic dependency relations. Modified medical entities , to form a triplet.

4. The method according to claim 3, characterized in that, The active mutation mechanism includes at least one of the following strategies: The cue variation strategy is as follows: the initial cue information is determined based on visual guidance features, and multiple different cue information are generated by randomly perturbing the position and / or scale parameters of the initial cue information and input into the segmentation base model respectively. Confidence threshold variation strategy: Fix the prompt information of the input segmentation base model, and binarize the probability map output by the model by setting a set of different thresholds to generate multiple different candidate segmentation masks; Strategy for utilizing internal randomness within the model: By setting different random seeds for multiple calls to the basic segmentation model, multiple different candidate segmentation masks can be obtained.

5. The method according to any one of claims 1 to 4, characterized in that, The cross-modal semantic matching model is a visual-language dual encoder model pre-trained through contrastive learning on a large-scale image-text pair dataset.

Images