Medical image segmentation system based on knowledge guidance and prompt generation

Abstract

The invention provides a medical image segmentation system based on knowledge guidance and prompt generation, which relates to the technical field of medical image segmentation and comprises a data acquisition module, an expert knowledge base, a knowledge scheduler, a parallel knowledge calculation unit, a prompt fusion device and a universal segmentation model with a prompt encoder. The expert knowledge base comprises a plurality of expert knowledge modules and is used for generating rich information prompts; the knowledge scheduler is used for selecting expert knowledge modules from the expert knowledge base; the parallel computing unit simultaneously executes an expert knowledge selection module; the prompt fusion device determines an optimal aggregation prompt set based on rich information prompts output by each expert knowledge module; and the universal segmentation model with the prompt encoder segments the medical image to be segmented into a corresponding segmented medical image based on the optimal aggregation prompt set. According to the method, the calculation of domain knowledge is separated from the segmentation capability of a general model, so that the problem of the general model in an automatic clinical process is solved.

Description

Technical Field

[0001] This invention belongs to the field of medical image segmentation technology, and in particular relates to a medical image segmentation system based on knowledge-guided and prompt-based generation. Background Technology

[0002] Medical image segmentation, as a core technology for precision medicine and intelligent diagnosis, carries the core tasks of clinical diagnosis, treatment planning, and prognostic assessment. The segmentation objects cover a full scale, from organ structures to minute lesions, and involve diverse imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, and pathological sections. Each modality, combined with a specific disease, possesses unique image representations, boundary features, and tissue textures, posing a significant challenge to achieving high-precision and robust segmentation algorithms. Relying entirely on manual delineation requires clinical experts to integrate in-depth knowledge of anatomy, pathology, and imaging physics for comprehensive judgment. This interdisciplinary, knowledge-intensive requirement places a huge cognitive load and time cost on segmentation work. Faced with the ever-increasing volume of image data, manual or semi-automatic modes that heavily rely on expert experience not only severely limit the efficiency of clinical workflows, but the core challenge lies in how to accurately and efficiently translate the prior knowledge of anatomy, pathology, and imaging physics in the minds of clinical experts into pixel-level spatial localization decisions.

[0003] Current mainstream deep learning-based image segmentation techniques have shown significant paradigmatic limitations. For example, the U-Net model relies on massive amounts of high-quality pixel-level labeled data, making it almost unusable in rare or emerging disease scenarios. Moreover, the U-Net model learns the statistical correlation between image pixels and segmentation masks, and its generalization ability and robustness are severely hampered when encountering blurred boundaries, atypical lesions, or artifacts that are inconsistent with the training data distribution. The MdeSAM model itself lacks any prior medical knowledge and cannot autonomously identify what constitutes a "lesion" or "normal tissue." Its performance relies entirely on high-quality prompts provided by human experts, failing to achieve true end-to-end automation. BRainMVP cannot generate an explicit, visual, and intuitive spatial guidance map; furthermore, while BRainMVP's ultimate goal is to improve the feature representation quality of downstream tasks and optimize feature extraction, it does not provide a clear, modular new path for transforming domain knowledge into automated segmentation decisions.

[0004] As disease presentations become increasingly complex and individualized, clinical practice requires handling segmentation tasks involving multimodal imaging, multiple targets, and dynamic changes. However, existing deep learning models are often trained and optimized in isolation for single scenarios. These specialized models have weak scenario transfer capabilities and struggle to maintain robust segmentation performance across modalities and diseases. Their performance deteriorates sharply once clinical conditions change slightly. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a medical image segmentation system based on knowledge-guided and prompt-based generation, comprising a data acquisition module, an expert knowledge base, a knowledge scheduler, a parallel knowledge computing unit, a prompt fusion unit, and a general segmentation model with a prompt encoder;

[0006] The data acquisition module is used to acquire the segmentation task and at least one medical image to be segmented;

[0007] The expert knowledge base includes multiple expert knowledge modules; the expert knowledge modules are used to judge the medical images to be segmented and generate rich information prompts.

[0008] The knowledge scheduler is used to determine at least one expert knowledge module from the expert knowledge base based on the segmentation task and the medical image to be segmented, and to generate an active list of expert knowledge modules.

[0009] The parallel computing unit is used to execute the expert knowledge modules in the expert knowledge module activation list simultaneously, and to determine the rich information prompts output by each expert knowledge module based on each expert knowledge module in the expert knowledge module activation list.

[0010] The prompt fusion unit is used to determine the optimal aggregated prompt set based on the rich information prompts output by each expert knowledge module;

[0011] The general segmentation model with cue encoder is used to segment the medical image to be segmented into corresponding segmented medical images based on the optimal aggregate cue set.

[0012] Furthermore, the method for constructing the expert knowledge module is as follows:

[0013] Establish several judgment rules for medical image diagnosis; decompose the judgment rules into a series of computer-understandable, structured judgment logics;

[0014] Based on a series of judgment logics based on decomposition, an image processing algorithm or machine learning algorithm is selected to complete the judgment logic. The selected image processing algorithm or machine learning algorithm is combined and encapsulated to generate a preliminary expert knowledge module.

[0015] The generated preliminary expert knowledge module is tested using an labeled validation dataset. Based on the test results, the parameters of the selected image processing or machine learning algorithm are adjusted and optimized until the performance of the selected image processing or machine learning algorithm meets the requirements for clinical application, thus obtaining the expert knowledge module.

[0016] Furthermore, the knowledge scheduler is used to determine at least one expert knowledge module from the expert knowledge base based on the segmentation task and the medical image to be segmented, and to generate an activation list of expert knowledge modules. The specific method is as follows:

[0017] Several clinical decision subtrees are constructed, including: clinical decision subtrees based on expert knowledge and clinical decision subtrees based on clinical data. The specific method for constructing clinical decision subtrees based on expert knowledge is as follows: acquire expert knowledge from multiple disease areas, visualize the decision logic flowchart in the expert knowledge, and generate clinical decision subtrees. The specific method for constructing clinical decision subtrees based on clinical data is as follows: use a clinical dataset containing imaging features and pathological diagnoses to train a decision tree algorithm in machine learning to generate a preliminary decision tree; evaluate, revise, and confirm the preliminary decision tree to obtain the clinical decision subtrees.

[0018] Semantic parsing is performed on the user-specified segmentation task, and a clinical decision subtree is selected from several pre-constructed clinical decision subtrees. Based on the selected clinical decision subtree, a diagnosis is performed on the medical image to be segmented, yielding a preliminary diagnostic result for the medical image to be segmented. Specifically:

[0019] The segmentation task is to determine the text information of the expert knowledge module. Based on one or more medical images to be segmented, hierarchical reasoning is performed starting from the root node of the clinical decision subtree. Each non-leaf node of the clinical decision subtree represents a key clinical judgment point and is bound to a node judgment module for judging the medical image to be segmented. Based on the judgment result, the module jumps to the corresponding child node. This process is carried out layer by layer until a leaf node is reached. Each leaf node is a core entity, which is the preliminary diagnosis result of the medical image to be segmented.

[0020] A clinical knowledge graph is constructed. The preliminary diagnostic results of the medical images to be segmented are input into the clinical knowledge graph for retrieval, and an active list of expert knowledge modules is generated.

[0021] Furthermore, the clinical knowledge graph includes several nodes and relational edges connecting two nodes. Each node is a medical entity, and the relational edges represent the clinical association logic and association weight between the two nodes. Each node is bound to a corresponding expert knowledge module.

[0022] The specific methods for constructing a clinical knowledge graph are as follows:

[0023] Obtain existing, publicly available medical knowledge bases and ontology libraries, extract the hierarchy and relationships of several primary medical entities associated with image diagnosis and several defined primary medical entities, and use them as the basic framework of the preliminary knowledge graph;

[0024] The process involves acquiring medical literature, using a large language model to mine several second medical entities and their relationships from the unstructured text of these documents, defining the relationships between two second medical entities as relation edges, and quantifying the strength of these relationships as the weights of the relation edges. These second medical entities and relation edges are then incorporated into the basic framework of a knowledge graph to obtain a preliminary knowledge graph. Medical literature includes textbooks, clinical guidelines, and research papers.

[0025] The preliminary knowledge graph is evaluated, revised, and optimized to obtain the clinical knowledge graph;

[0026] The preliminary diagnostic results of the medical image to be segmented are input into the clinical knowledge graph for retrieval, generating an activated list of expert knowledge modules. The specific method is as follows:

[0027] Centered on the node representing the preliminary diagnostic result of the medical image to be segmented, a graph traversal is performed along the high-weight relation edges to identify all directly related first-order neighbor nodes, obtain the expert knowledge module bound to each first-order neighbor node, summarize the expert knowledge modules bound to all first-order neighbor nodes, and generate an expert knowledge module activation list.

[0028] Furthermore, the rich information prompts include several prompt objects, a global confidence level, and a local confidence level for each prompt object; the types of prompt objects include point prompts, box prompts, and text prompts; the global confidence level is the confidence level of the expert knowledge module for all output prompt objects; and the local confidence level is the confidence level of the expert knowledge module for each output prompt object.

[0029] Furthermore, the specific method by which the prompt fusion processor determines the optimal aggregated prompt set based on the rich information prompts output by each expert knowledge module is as follows:

[0030] The discrete, information-rich cues output by each expert knowledge module are transformed into a continuous spatial evidence map.

[0031] Based on the Dempster-Shafer evidence theory, the spatial evidence maps of all expert knowledge modules in the expert knowledge module activation list are merged into the final confidence map.

[0032] Extract the aggregated cue set from the final confidence map, including several augmentation point cue and augmentation box cue;

[0033] The system obtains human-generated prompts from user input and reviews, corrects, and supplements the optimal aggregated prompt set output by the fusion prompter.

[0034] Furthermore, the specific method for transforming the discrete, information-rich cues output by each expert knowledge module into a continuous spatial evidence map is as follows:

[0035] Based on the rich information prompts output by each expert knowledge module, a spatial evidence map with the same size as the medical image to be segmented is generated, where the value range of each pixel is [0,1], representing the degree of confidence that the pixel belongs to the target segmentation region;

[0036] Different strategies are used to generate spatial evidence maps for different types of cue objects in rich information cues, including:

[0037] For point cues, a Gaussian kernel function is used to model the spatial confidence distribution, constructing a location coordinate system for the point cues on the spatial evidence map. A two-dimensional Gaussian confidence distribution centered at the base, with the peak value of the two-dimensional Gaussian confidence distribution as the base confidence level. , For local confidence level, Global confidence level; standard deviation of Gaussian kernel These are configurable hyperparameters used to control the range of influence of a single point prompt;

[0038] For box hints, a spatial evidence map is generated using a method of uniformly assigning confidence scores within the region. The confidence scores of all pixels within the rectangular frame of the box hint are uniformly set as the base confidence score. ;

[0039] Textual cues are not involved in the generation of the spatial evidence map. Instead, the textual cues are directly collected and transmitted to a general segmentation model with a cue encoder.

[0040] Furthermore, based on the Dempster-Shafer evidence theory, the spatial evidence maps of all expert knowledge modules in the expert knowledge module activation list are fused into a final confidence map. The specific method is as follows:

[0041] Obtain the spatial evidence map of all expert knowledge modules in the active list of expert knowledge modules, and define the identification framework for each pixel in the spatial evidence map. ,in, This indicates that the pixel belongs to the target segmentation region. This indicates that the pixel does not belong to the target segmentation region;

[0042] For the Spatial evidence map of expert knowledge modules Construct a BPA function based on the pixel value of any pixel. For all The BPA function of the spatial evidence map is applied pixel by pixel to iteratively fuse the Dempster-Shafer evidence theory. After pixel-by-pixel fusion calculation of all spatial evidence maps, a unique final confidence map that fuses all spatial evidence maps is generated.

[0043] The aggregated cue set is extracted from the final confidence map, including several augmentation point cue and augmentation box cue, specifically:

[0044] The non-maximum suppression algorithm is used to locate high local confidence regions in the final confidence map. All local confidence peak points are found in the final confidence map, and each peak point is regarded as the center of a potential target segmentation region. At the same time, other secondary peaks in the neighborhood of the local confidence peak point are suppressed.

[0045] Local topology analysis is performed on local confidence peak points to evaluate the rate of change of confidence of pixels in the neighborhood of the local confidence peak points. For local confidence peak points with steep gradients and concentrated confidence, the coordinates of the local confidence peak points are extracted and enhancement point prompts are generated.

[0046] For local confidence peaks with gentle gradients and diffuse confidence, the watershed algorithm is used as the starting point to determine the precise boundary of the high-confidence connected region and calculate its minimum bounding rectangle to obtain the enhanced box cue.

[0047] Furthermore, the general segmentation model with cue encoder includes a cue encoder, an image encoder, and a mask decoder;

[0048] The cue encoder is used to extract cue embedding vectors based on the optimal aggregated cue set;

[0049] An image encoder is used to encode the medical image to be segmented, thereby obtaining the medical image encoding features;

[0050] The mask decoder is used to generate pixel-level segmentation masks to obtain segmented medical images;

[0051] The prompt embedding vector and the medical image encoding features extracted by the image encoder are input into the mask decoder module. The prompt embedding vector is used as key guiding information to guide the mask decoder module to accurately locate and decode the target region that completely matches the prompt diagram in the global image features, generate a pixel-level segmentation mask, and obtain the segmented medical image.

[0052] Furthermore, the specific method for the prompt encoder to extract the prompt embedding vector based on the optimal aggregated prompt set is as follows:

[0053] For the enhanced point cues in the optimal aggregated cue set, a positional encoding technique is used to encode the two-dimensional spatial coordinates of the enhanced point cues. The mapping is transformed into a high-dimensional vector that carries spatial location information; a type embedding vector for representing semantic type, which is learned through pre-training, is superimposed on the high-dimensional vector to obtain the hint embedding vector for the enhanced point hint;

[0054] For the enhanced box cue in the optimal aggregated cue set, the coordinates of the top-left corner of the enhanced box cue are... and the coordinates of the lower right corner Normalize and perform position encoding to obtain the top-left corner position vector. and the bottom right corner position vector ;

[0055] Vector at the top left corner and the bottom right corner position vector The top left vector is obtained by superimposing a type embedding vector on each of the above. and the bottom right vector As shown in the formula below:

[0056]

[0057]

[0058] in, and These are the type embedding vectors at the top left and bottom right, respectively.

[0059] The top left vector and the bottom right vector The data is then spliced ​​together and input into a multilayer perceptron (MLP) for fusion to obtain... The augmented box hint embedding vector is used to enhance the dimensionality of the hint; self-attention or cross-attention can also be used to fuse the top-left corner vector. and the bottom right vector ,get The tooltip embedding vector of the augmented box tooltip; the tooltip embedding vector contains both range information and position information;

[0060] For text prompts in the optimal aggregate prompt set, a pre-trained text encoder is used to map the descriptive text into prompt embedding vectors for the text prompts, which indicate the category, attribute, or anatomical orientation of the segmented target.

[0061] The beneficial effects of adopting the above technical solution are as follows: The medical image segmentation system based on knowledge guidance and prompt generation provided by this invention transforms unstructured clinical diagnostic logic into a set of spatial localization prompts that can be understood by a general segmentation model through modular knowledge encapsulation, intelligent scheduling, and evidence-based fusion. By separating the computation of domain knowledge from the segmentation capabilities of the general model, this invention allows for independent iterative updates of the expert knowledge module and the segmentation model, achieving higher system flexibility and scalability, and solving the problem of general models in automated clinical processes.

[0062] The rich information prompts output by the expert knowledge module in this application include dual confidence levels: global confidence and local confidence. This dual quantification mechanism provides richer and more structured input information for the downstream prompt fusion unit, making it possible to finely fuse multi-source and multi-target evidence. It is a key design to improve the robustness and accuracy of the entire system's decision-making.

[0063] This invention proposes a unique hybrid reasoning method to overcome the limitations of single knowledge scheduling methods. First, it leverages the deterministic reasoning capabilities of clinical decision trees to perform convergent analysis on images, quickly pruning irrelevant branches and identifying core pathological entities. This addresses the problems of excessive search space and low efficiency that may result from using knowledge graphs alone. Subsequently, using the output of the decision tree as an anchor point, it utilizes the associative reasoning capabilities of the clinical knowledge graph to perform divergent expansion, recalling accompanying signs strongly correlated with the core entities. This avoids the narrow field of view and information omissions that may occur when using decision trees alone. Based on a mechanism of focusing first and then diverging, this invention achieves dual optimization in both depth and breadth of the expert knowledge retrieval process, ensuring the accuracy and completeness of the scheduling.

[0064] In the fusion of multi-source heterogeneous cues, this invention projects discrete cues (points, boxes) from different sources and with varying forms into a standardized, continuous "spatial evidence map" using functions such as Gaussian kernels, thus achieving homogenization of heterogeneous information. A pixel-by-pixel application of the DS combination rule performs nonlinear fusion of multi-source evidence, effectively handling uncertainty and conflict. Compared to simple weighted averaging, the result more closely approximates the optimal cognitive decision under multiple evidence. The final confidence map after fusion is then reverse-extracted using algorithms such as nonmaximum suppression and adaptive geometric analysis, transforming the continuous confidence distribution back into a set of refined and efficient discrete cues, thus forming a complete technical closed loop from multi-source, discrete, heterogeneous input to single-source, discrete, homogeneous output. Attached Figure Description

[0065] Figure 1 A schematic diagram of the medical image segmentation system based on knowledge guidance and prompt generation provided in Embodiment 1 of the present invention;

[0066] Figure 2 A schematic diagram of the expert knowledge base and expert knowledge module provided in Embodiment 1 of the present invention;

[0067] Figure 3 A schematic diagram of a clinical decision tree structure provided in Embodiment 1 of the present invention;

[0068] Figure 4 A schematic diagram of a clinical knowledge graph provided in Embodiment 1 of this invention;

[0069] Figure 5 The diagram below illustrates the process by which the prompt fusion device provided in Embodiment 1 of the present invention fuses rich information prompts into an optimal aggregated prompt set. Detailed Implementation

[0070] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0071] Example 1:

[0072] A medical image segmentation system based on knowledge-guided and prompt-based generation; please refer to [link / reference]. Figure 1 It includes a data acquisition module, an expert knowledge base, a knowledge scheduler, a parallel knowledge computing unit, a prompting fusion unit, and a general segmentation model with a prompting encoder; the expert knowledge base includes multiple expert knowledge modules;

[0073] The data acquisition module is used to acquire the segmentation task specified by the user and one or more medical images to be segmented;

[0074] The knowledge scheduler is used to analyze the segmentation task and medical images, select one or more expert knowledge modules associated with the segmentation task from the expert knowledge base, generate an expert knowledge module activation list and send it to the parallel computing unit;

[0075] The parallel computing unit is used to execute one or more expert knowledge modules from the expert knowledge module activation list simultaneously. Each expert knowledge module outputs a corresponding rich information prompt and inputs it into the prompt fusion unit.

[0076] The prompt fusion unit merges the rich information prompts output by one or more expert knowledge modules into an optimal aggregated prompt set, and inputs the optimal aggregated prompt set into a general segmentation model with a prompt encoder. The prompt fusion unit also includes an optional human-machine collaboration interface, which allows users to input human prompts and review, correct and supplement the optimal aggregated prompt set output by the fusion prompt unit.

[0077] A general segmentation model with cue encoders segments one or more medical images to be segmented based on the optimal aggregate cue set, resulting in one or more segmented medical images.

[0078] The expert knowledge base includes multiple expert knowledge modules, each containing an algorithm for a specific clinical segmentation rule. This implementation transforms the knowledge used to analyze images into a series of specific expert knowledge modules, each of which is an algorithmic encapsulation of a specific clinical segmentation rule, thereby constructing an scalable expert knowledge base that includes multiple independent expert knowledge modules.

[0079] The expert knowledge module outputs multi-source cue information that can be understood by a general segmentation model with a cue encoder. To facilitate subsequent fusion and optimization of the multi-source cue information, in this embodiment, the expert knowledge module is constructed to judge one or more medical images to be segmented and generate rich information cue. The expert knowledge module can output several rich information cue, which includes the cue object, global confidence, and local confidence of the cue object. The types of cue objects include point cue, box cue, and text cue. The global confidence is the confidence of the expert knowledge module in all the output rich information cue. The local confidence of the cue object represents the confidence of the expert knowledge module in one output rich information cue.

[0080] The specific method for constructing expert knowledge modules is as follows:

[0081] Obtain relevant guidance materials and clinical treatment guidelines for medical imaging, such as the "Guidelines for the Diagnosis and Treatment of Liver Cancer," and establish several judgment rules for medical image diagnosis; decompose the judgment rules into a series of computer-understandable, structured judgment logics;

[0082] Based on a series of judgment logics based on decomposition, an image processing algorithm or machine learning algorithm is selected to complete the judgment logic. The selected image processing algorithm or machine learning algorithm is combined and encapsulated to generate a standardized, independently callable preliminary expert knowledge module.

[0083] The generated preliminary expert knowledge module is tested using an labeled validation dataset. Based on the test results, the parameters of the selected image processing algorithm or machine learning algorithm are adjusted and optimized until the performance of the selected image processing algorithm or machine learning algorithm meets the requirements for clinical application, thus obtaining the expert knowledge module.

[0084] like Figure 2The example shown illustrates the construction of an expert knowledge module. The knowledge that "poorly vascularized liver metastases appear as low-density lesions in the portal venous phase" is established as a judgment logic of "finding connected regions within the liver area where the pixel Henlein unit (HU) value is significantly lower than the average value of the surrounding normal liver tissue." Based on this judgment logic, appropriate image processing or machine learning algorithms are selected. For example, "locating the liver region" can use a pre-trained organ segmentation model; "calculating the density benchmark" can use statistical methods; and "identifying connected regions" can use threshold segmentation and connected component analysis algorithms. The diagnostic knowledge regarding poorly vascularized liver metastases is algorithmically encapsulated. The diagnostic rule for "poorly vascularized liver metastases" states that poorly vascularized liver metastases often appear as low-density lesions on CT images in the portal venous phase, with a density significantly lower than the surrounding normal liver parenchyma. Based on this, this embodiment uses an automated processing flow to identify image regions that meet the characteristics of low-density lesions and generate accurate prompts, thereby providing crucial guidance for subsequent segmentation models. This includes the following steps:

[0085] S1: Establish a baseline for normal liver parenchyma density;

[0086] Using portal venous phase CT images of the liver as input, a pre-trained segmentation model is used to locate and segment the liver region in the portal venous phase CT images. In this embodiment, a pre-trained U-Net segmentation model is used to locate and segment the liver region. The distribution of Henlein units (HU) values ​​within the liver region is statistically analyzed, and a density benchmark value that can represent normal liver tissue is calculated and established.

[0087] S2: Based on the normal liver parenchyma density benchmark, segment low-density candidate regions;

[0088] Based on the density baseline, all low-density pixels with Henle unit values ​​lower than the density baseline are identified in the liver region to form a binary image.

[0089] Morphological opening operations are used to preprocess binary images to remove noise;

[0090] Connecting adjacent low-density pixels in the spatial domain yields independent candidate regions;

[0091] By combining area threshold filtering and roundness filtering, candidate regions with sizes smaller than a preset threshold or irregular shapes are removed, resulting in several low-density candidate regions.

[0092] S3: Extract key imaging features from low-density candidate regions, quantify the key imaging features, and calculate the confidence level;

[0093] For each low-density candidate region, key imaging features of the low-density candidate region are extracted, and the key imaging features are quantitatively evaluated to determine the degree of consistency between the low-density candidate region and the target lesion.

[0094] In the diagnosis of metastatic tumors in the hypoperfusion liver, key imaging features include: the density difference between the current low-density candidate region and the normal liver parenchyma baseline, and the clarity of the boundary of the current low-density candidate region.

[0095] Key imaging features are quantified and evaluated to obtain local confidence scores. The higher the local confidence score, the more typical the lesion features of the current low-density candidate area.

[0096] S4: Generate and output information prompts;

[0097] For low-density candidate regions where the local confidence level exceeds a preset threshold, determine the geometric center of the current low-density candidate region and generate a point hint.

[0098] The point hints and their corresponding local confidence scores are encapsulated into standardized rich-information hints;

[0099] The above steps encapsulate the knowledge about poorly vascularized liver metastases into an algorithm, resulting in an expert knowledge module used to diagnose poorly vascularized liver metastases.

[0100] The core task of the knowledge scheduler is to dynamically select and activate one or more expert knowledge modules associated with the segmentation task from the expert knowledge base, based on one or more medical images to be segmented and the segmentation task, generating an expert knowledge module activation list. To achieve accurate and comprehensive retrieval of expert knowledge modules, the segmentation task serves as the text information used to determine the expert knowledge modules. This embodiment establishes a two-level reasoning mechanism, such as... Figure 3 As shown, it includes:

[0101] Step A: Construct several clinical decision subtrees;

[0102] The clinical decision subtree is designed to simulate a structured diagnostic process, enabling intelligent guidance from broad tasks to specific expert knowledge modules;

[0103] Methods for constructing clinical decision subtrees include: constructing clinical decision subtrees based on expert knowledge and constructing clinical decision subtrees based on clinical data;

[0104] The specific method for constructing a clinical decision subtree based on expert knowledge is as follows:

[0105] Acquire expert knowledge from multiple disease areas, visualize the decision-making logic within that knowledge, and form a clinical decision subtree.

[0106] The specific method for constructing a clinical decision subtree based on clinical data is as follows:

[0107] Using a clinical dataset containing imaging features and pathological diagnoses, a decision tree algorithm in machine learning is trained to generate a preliminary decision tree;

[0108] The initial decision tree is evaluated, revised, and confirmed to ensure that each reasoning path conforms to clinical logic, forming a clinical decision subtree that combines statistical data patterns and expert medical knowledge.

[0109] Step B: Perform semantic parsing on the segmentation task specified by the user, and select a clinical decision subtree from the several clinical decision subtrees that have been constructed; diagnose one or more medical images to be segmented based on the selected clinical decision subtree to obtain the preliminary diagnostic results of the medical images to be segmented.

[0110] Based on one or more medical images to be segmented, hierarchical reasoning begins from the root node of the clinical decision subtree. Each non-leaf node of the clinical decision subtree represents a key clinical judgment point and is bound to a node judgment module for judging the medical image to be segmented. Based on the judgment result, the system jumps to the corresponding child node. This process proceeds layer by layer until a leaf node is reached. Each leaf node is a core entity, namely the preliminary diagnosis result of the medical image to be segmented, which serves as the central entity for retrieval and association expansion in the clinical knowledge graph in the second stage.

[0111] For example, if the user specifies the segmentation task as liver tumor segmentation, the corresponding liver cancer clinical decision subtree is selected; the liver cancer clinical decision subtree is then used to make judgments on the medical images to be segmented, such as... Figure 3 As shown, starting from the root node, the node judgment module "NDM-lesion initial detection" for initial lesion detection is called to determine whether there are suspicious space-occupying lesions in the medical image to be segmented.

[0112] like Figure 3In the example path of the liver cancer clinical decision subtree shown, if the judgment result of the expert knowledge module "NDM-Preliminary Lesion Detection" used for initial lesion detection called by the root node is "Yes", then it flows to the "Lesion shows significant low density" judgment node, and calls the node judgment module "NDM-Internal Density Characteristics of Lesion" used to determine the internal density of the lesion to perform preliminary density qualitative analysis; if in this path the lesion is not a significantly low-density cystic lesion, the judgment result is "No", and it continues to go deeper along the branch of solid lesions, flowing to the "Smooth and Regular Lesion Boundaries" judgment node; it calls the node judgment module "NDM-Tumor Boundary Morphology" used to analyze the growth morphology of the lesion; if If the lesion has smooth and regular boundaries, consistent with the characteristics of expansive growth, the judgment result is "yes". Based on this, the process proceeds to the deeper "internal texture is heterogeneous" judgment node. The node judgment module "NDM-local texture anomaly" used to judge the internal texture of the tumor further analyzes the homogeneity of the lesion. If the final analysis result is "yes", it indicates that the internal texture of the solid lesion with smooth boundaries is uneven, which is consistent with the common characteristics of malignant tumors. At this point, the reasoning path finally converges and arrives at the leaf node "suspected expansive growth tumor". The clinical conclusion represented by this leaf node is the core entity "hepatocellular carcinoma HCC" output by the clinical decision subtree, which is the preliminary judgment result of the medical image to be segmented.

[0113] Step C: Construct a clinical knowledge graph. Input the preliminary diagnostic results of the medical images to be segmented into the clinical knowledge graph for retrieval and generate an activated list of expert knowledge modules.

[0114] A clinical knowledge graph consists of several nodes and edges connecting two nodes. Each node is a medical entity, and the edges represent the clinical relationship and weight between the two nodes. Each node is bound to a corresponding expert knowledge module. The definitions of both nodes and edges are entirely dependent on knowledge in the medical field. Therefore, the construction process of the knowledge graph integrates multiple information sources to ensure the breadth, depth, and accuracy of its knowledge.

[0115] The specific methods for constructing a clinical knowledge graph are as follows:

[0116] Obtain existing, publicly available medical knowledge bases and ontology libraries, extract the hierarchy and relationships of several primary medical entities associated with image diagnosis and several defined primary medical entities, and use them as the basic framework of the preliminary knowledge graph;

[0117] The process involves acquiring medical literature and using a large language model to mine several second medical entities and their relationships from unstructured texts. For example, the relationship between "hepatocellular carcinoma" and "pseudocapsule" is often described as "commonly present." The relationships between two second medical entities are then used as edge relationships, and the strength of these relationships is quantified as the weight of each edge. These second medical entities and edge relationships are then incorporated into the basic framework of a knowledge graph to obtain a preliminary knowledge graph. Medical literature includes textbooks, clinical guidelines, and research papers.

[0118] The preliminary knowledge graph is evaluated, revised, and optimized to ensure that the knowledge in the preliminary knowledge graph is accurate and in line with clinical practice, thus obtaining a clinical knowledge graph. For example, the association weight between "HCC main lesion" and "tumor pseudocapsule" should be higher than its association weight with "cirrhosis background" to ensure that the knowledge in the graph is accurate and in line with clinical practice.

[0119] The core entity "hepatocellular carcinoma (HCC)" in this embodiment is entered into the clinical knowledge graph for retrieval, such as... Figure 4 As shown, it includes the following steps:

[0120] By semantic matching, the node representing the core entity "hepatocellular carcinoma (HCC)" is located in the knowledge graph, and the expert knowledge module bound to the node is obtained, namely the expert knowledge module "EKM-HCC main lesion segmentation" used to segment hepatocellular carcinoma lesions.

[0121] Centered on the node representing the core entity "hepatocellular carcinoma HCC", a graph traversal is performed along the high-weight relational edges to identify all directly related first-order neighbor nodes. In this embodiment, two strongly related nodes directly associated with "hepatocellular carcinoma HCC" are identified, namely the "tumor pseudocapsule" node and the "cirrhosis background" node.

[0122] The expert knowledge module bound to each first-order neighbor node is obtained. In this embodiment, "EKM-tumor boundary morphology analysis" is extracted from the "tumor pseudocapsule" node, and "liver parenchyma background analysis" is extracted from the "cirrhosis background" node.

[0123] The expert knowledge modules bound to all first-order neighbor nodes are aggregated to generate an expert knowledge module activation list. In this embodiment, the expert knowledge module activation set includes: EKM-HCC main lesion segmentation, EKM-tumor boundary morphology analysis, and liver cirrhosis background;

[0124] In the parallel knowledge computing unit, expert knowledge modules in the expert knowledge module activation list are executed simultaneously to obtain rich information prompts for each expert knowledge module and send them to the prompt fusion unit.

[0125] The prompt fusion unit receives and processes the rich information prompts corresponding to each expert knowledge module output by the parallel knowledge computing unit, and merges the rich information prompts from multiple expert knowledge modules, which may be redundant or conflicting, into an optimal aggregated prompt set. The prompt fusion unit also includes an optional human-machine collaboration interface, which allows users to input manual prompts and review, correct and supplement the optimal aggregated prompt set output by the fusion prompt unit.

[0126] The rich information hints from multiple expert knowledge modules are merged into an optimal aggregated hint set, such as... Figure 5 As shown, it includes the following steps:

[0127] Step 1: Transform the discrete, information-rich cues output by each expert knowledge module into a continuous spatial evidence map;

[0128] Based on the rich information prompts output by each expert knowledge module, a spatial evidence map with the same size as the medical image to be segmented is generated, where the value range of each pixel is [0,1], representing the degree of confidence that the pixel belongs to the target segmentation region;

[0129] Different strategies are used to generate spatial evidence maps for different types of cue objects in rich information cues, including:

[0130] For point cues, a Gaussian kernel function is used to model the spatial confidence distribution, constructing a location coordinate system for the point cues on the spatial evidence map. A two-dimensional Gaussian confidence distribution centered at the base, with the peak value of the two-dimensional Gaussian confidence distribution as the base confidence level. , For local confidence level, Global confidence level; standard deviation of Gaussian kernel As a configurable hyperparameter, it is used to control the influence range of a single point prompt. The significance of using the Gaussian kernel function to model the spatial confidence distribution is that it transforms an absolute discrete point coordinate into a smooth continuous probability field, which is more in line with the ambiguity of target localization in clinical practice, and allows the influence of the prompt to reasonably penetrate into its neighboring area, thereby enhancing the robustness of the algorithm.

[0131] For box hints, a spatial evidence map is generated using a method of uniformly assigning confidence scores within the region. The confidence scores of all pixels within the rectangular frame of the box hint are uniformly set as the base confidence score. This approach follows the semantics of box prompts, meaning that all regions within the box are considered equally as target segmentation regions.

[0132] Since text prompts carry semantic information rather than spatial location information, they are not involved in the generation of the spatial evidence map. Instead, the text prompts are directly collected and transmitted to a general segmentation model with a prompt encoder.

[0133] Step 2: Based on the Dempster-Shafer evidence theory, merge the spatial evidence maps of all expert knowledge modules in the expert knowledge module activation list into the final confidence map;

[0134] The system acquires spatial evidence maps of all expert knowledge modules in the activated list, and performs pixel-level fusion on inconsistencies and conflicts in these maps to form a comprehensive global information judgment. This embodiment introduces Dempster-Shafer evidence theory as a mathematical framework. Compared to traditional Bayesian probability, Dempster-Shafer evidence theory can explicitly distinguish between unsupported and uncertain evidence, making it particularly suitable for handling evidence of varying reliability from different EKMs.

[0135] Step 2.1: Obtain the spatial evidence map of all expert knowledge modules in the expert knowledge module activation list. For each pixel in the spatial evidence map, define the pixel identification framework. ,in, This indicates that the pixel belongs to the target segmentation region. This indicates that the pixel does not belong to the target segmentation region;

[0136] Step 2.2: Construct the basic probability assignment:

[0137] For the Spatial evidence map of expert knowledge modules Construct a BPA function based on the pixel value of any pixel. As shown in the formula below:

[0138]

[0139] in, Coordinates are pixels pixel values, To support pixels The amount of evidence belonging to the target segmentation region. To support pixels The amount of evidence that does not belong to the target segmentation region. For uncertain pixels The amount of evidence belonging to the target segmentation region;

[0140] In this embodiment, the following settings are provided: That is, the expert knowledge module only provides "support for pixels". Positive evidence that it "belongs to the target segmentation region" is provided, but no evidence of "supporting pixels" is provided. Negative evidence that it "does not belong to the target segmentation region" This indicates that the pixel could not be retrieved. The amount of evidence required to make a clear judgment on whether a region belongs to the target segmentation area.

[0141] For all The BPA function of the spatial evidence map is iteratively fused pixel-by-pixel using Dempster-Shafer evidence theory, as shown in the following formula:

[0142]

[0143] in, For the first Zhang Space Evidence Map The BPA function, For the first Zhang Space Evidence Map The BPA function, For the merged spatial evidence map The BPA function;

[0144] The first spatial evidence map BPA function and the second spatial evidence map The combination rules for fusing BPA functions are as follows: ;

[0145] When multiple spatial evidence maps consistently support the proposition that a pixel belongs to the target segmentation region at the same pixel location, the confidence in that pixel belonging to the target segmentation region will be significantly enhanced. When there is conflict among multiple spatial evidence maps, the combination rule based on Dempster-Shafer evidence theory can quantify the degree of conflict and reasonably redistribute the remaining confidence.

[0146] After pixel-by-pixel fusion calculation of all spatial evidence maps, a unique final confidence map is generated that integrates all spatial evidence maps. Each pixel value represents the final confidence level of the proposition that the pixel belongs to the target segmentation region after integrating all positive evidence and eliminating uncertainty.

[0147] Step 3: Extract the aggregated cue set from the final confidence map, including several augmentation point cue and augmentation box cue;

[0148] The non-maximum suppression algorithm is used to locate high local confidence regions in the final confidence map. All local confidence peak points are found in the final confidence map, and each peak point is regarded as the center of a potential target segmentation region. At the same time, other secondary peaks in the neighborhood of the local confidence peak point are suppressed, thereby effectively avoiding the generation of duplicate prompts for the same target entity.

[0149] Local topological analysis is performed on local confidence peak points to adaptively generate optimal cue geometry. This analysis aims to determine whether high local confidence regions exhibit a "pulse-like" or "plateau-like" morphology. Specifically, the rate of change of confidence among pixels in the neighborhood of a local confidence peak point is evaluated. For local confidence peak points with steep gradients and concentrated confidence, the region where the peak point is located is considered compact, so the coordinates of the peak point are directly extracted to generate a high-information-density enhancement point cue. For local confidence peak points with gentle gradients and diffuse confidence, the target range represented by the peak point is considered broader, requiring richer spatial description. Starting from a local confidence peak point with gentle gradients and diffuse confidence, the watershed algorithm is applied to determine the precise boundary of the high-confidence connected region and calculate its minimum bounding rectangle to generate a more generalized enhancement box cue.

[0150] Step 4: Obtain human prompts input by the user, and review, correct, and supplement the optimal aggregated prompt set output by the fusion prompter;

[0151] To meet the stringent requirements of high reliability and high safety in clinical applications and to build a reliable human-machine collaboration closed loop, the prompt fusion device in this embodiment also includes a human-machine collaboration interface, allowing users to input manual prompts and review, correct, and supplement the optimal aggregated prompt set output by the fusion prompt device, including:

[0152] All point or box hints in the aggregated hint set are overlaid on the medical image to be segmented, generating a visual aggregated hint that is presented to the user for review.

[0153] Obtain user feedback on the visual aggregation, supplement and correct the visual aggregation prompts based on the user feedback, and obtain the optimal aggregation prompt set.

[0154] A general segmentation model with a cue encoder includes a cue encoder, an image encoder, and a mask decoder. The optimal aggregated cue set is input into the cue encoder, which transforms the optimal aggregated cue set from multiple sources into a unified high-dimensional feature representation capable of mathematical operations, resulting in a cue embedding vector that guides the mask decoder in segmenting medical images. The image encoder encodes the medical image to be segmented, obtaining medical image encoded features. The mask decoder generates a pixel-level segmentation mask, resulting in the segmented medical image.

[0155] Extracting the hint embedding vector based on the optimal aggregated hint set, specifically, is as follows:

[0156] For the enhanced point cues in the optimal aggregated cue set, a positional encoding technique is used to encode the two-dimensional spatial coordinates of the enhanced point cues. The mapping is to a high-dimensional vector that carries spatial location information; at the same time, in order to distinguish whether the augmentation point cues belong to the target segmentation region or a non-target segmentation region, a type embedding vector for representing semantic type, which is learned through pre-training, is superimposed on the high-dimensional vector to obtain the cue embedding vector of the augmentation point cues.

[0157] For the enhanced box cue in the optimal aggregated cue set, the coordinates of the top-left corner of the enhanced box cue are... and the coordinates of the lower right corner Normalize and perform position encoding to obtain the top-left corner position vector. and the bottom right corner position vector ;

[0158] Vector at the top left corner and the bottom right corner position vector The top left vector is obtained by superimposing a type embedding vector on each of the above. and the bottom right vector As shown in the formula below:

[0159]

[0160]

[0161] in, and These are the type embedding vectors at the top left and bottom right, respectively.

[0162] The top left vector and the bottom right vector The data is then spliced ​​together and input into a multilayer perceptron (MLP) for fusion to obtain... The augmented box hint embedding vector is used to enhance the dimensionality of the hint; self-attention or cross-attention can also be used to fuse the top-left corner vector. and the bottom right vector ,get The tooltip embedding vector of the augmented box tooltip; the tooltip embedding vector contains both range information and position information;

[0163] For text prompts in the optimal aggregate prompt set, a pre-trained text encoder is used to map the descriptive text into prompt embedding vectors for the text prompts, which are used to indicate the category, attribute or anatomical orientation of the segmented target.

[0164] An image encoder is used to encode the medical image to be segmented, thereby obtaining the medical image encoding features;

[0165] The prompt embedding vector and the medical image encoding features extracted by the image encoder are input into the mask decoder module. The prompt embedding vector is used as key guiding information to guide the mask decoder module to accurately locate and decode the target region that completely matches the prompt diagram in the global image features, generate a pixel-level segmentation mask, and obtain the segmented medical image.

[0166] Compared to the U-Net paradigm, this embodiment encapsulates the abstract knowledge of professional doctors analyzing images into a series of explicit expert knowledge modules, and decouples the segmentation task into two stages: knowledge computation and general segmentation. This not only significantly improves the interpretability of the decision-making process through a visualized spatial guidance graph, but also significantly reduces the reliance on massive amounts of pixel-level labeled data because it learns more general rules rather than patterns. Compared to general models such as MedSAM, this embodiment, by introducing a knowledge scheduler and parallel knowledge computation units, achieves automated and intelligent generation from image analysis to segmentation suggestions. It transforms the basic model from a passively executed tool into an intelligent system capable of autonomous decision-making, especially demonstrating superior performance in data-sparse scenarios under the guidance of expert knowledge. Furthermore, compared to implicit information fusion methods such as BrainMVP, the explicit guidance diagrams and prompts generated in this embodiment provide clinicians with clear judgment criteria and build a reliable closed loop for efficient human-computer collaboration: the medical image segmentation system based on knowledge guidance and prompt generation in this embodiment can run fully automatically, and doctors can easily intervene at key nodes to supervise and guide the segmentation process by reviewing, correcting, or supplementing prompts, which is more in line with the application requirements of high-risk clinical decision-making scenarios.

[0167] Example 2:

[0168] This embodiment proposes an electronic device, including: one or more processors, and a memory for storing instructions, which, when executed by the one or more processors, cause the one or more processors to execute the medical image segmentation system based on knowledge guidance and prompt generation.

[0169] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements medical image segmentation based on knowledge guidance and prompting as described in the embodiments. It is understood that the electronic device may also include input / output (I / O) interfaces and communication components.

[0170] The processor is used to execute all or part of the steps in medical image segmentation based on knowledge guidance and prompts as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0171] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the medical image segmentation system based on knowledge guidance and prompting as described in the above embodiments.

[0172] Example 3:

[0173] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0174] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of medical image segmentation based on knowledge guidance and prompting as described in the various embodiments of this application.

[0175] The aforementioned storage media include: flash memory, hard disks, multimedia cards, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disks, optical discs, servers, APP (Application) app stores, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various modules of the knowledge-guided and prompt-based medical image segmentation system described above.

[0176] Example 4:

[0177] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned medical image segmentation system based on knowledge guidance and prompt generation.

[0178] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0179] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0180] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this disclosure and its equivalents, then the intent of this disclosure also includes these modifications and variations.

Claims

1. A medical image segmentation system based on knowledge guidance and prompt generation, characterized by, The system comprises a data acquisition module, an expert knowledge base, a knowledge dispatcher, a parallel knowledge calculation unit, a hint fusioner and a general segmentation model with a hint encoder. The data acquisition module is configured to acquire a segmentation task and at least one medical image to be segmented. The expert knowledge base comprises a plurality of expert knowledge modules. The expert knowledge modules are configured to judge the medical image to be segmented and generate rich information hints. The knowledge dispatcher is configured to determine at least one expert knowledge module from the expert knowledge base based on the segmentation task and the medical image to be segmented, and generate an expert knowledge module activation list. The parallel calculation unit is configured to simultaneously execute the expert knowledge modules in the expert knowledge module activation list, and determine the rich information hints output by each expert knowledge module based on each expert knowledge module in the expert knowledge module activation list. The hint fusioner is configured to determine an optimal aggregated hint set based on the rich information hints output by each expert knowledge module.

2. The medical image segmentation system based on knowledge guidance and prompt generation of claim 1, wherein, The general segmentation model with the hint encoder is configured to segment the medical image to be segmented into a corresponding segmented medical image based on the optimal aggregated hint set. The expert knowledge module is constructed by: establishing a plurality of judgment rules for medical image diagnosis, decomposing the judgment rules into a series of computer-understandable and structured judgment logics; selecting image processing algorithms or machine learning algorithms for completing the judgment logics based on the decomposed series of judgment logics, combining and packaging the selected image processing algorithms or machine learning algorithms to generate a preliminary expert knowledge module; 3. The medical image segmentation system based on knowledge guidance and prompt generation of claim 1, wherein, testing the generated preliminary expert knowledge module using a labeled verification data set, and adjusting and optimizing the parameters of the selected image processing algorithms or machine learning algorithms according to the test results until the performance of the selected image processing algorithms or machine learning algorithms meets the requirements of clinical application, thereby obtaining the expert knowledge module. The knowledge dispatcher is configured to determine at least one expert knowledge module from the expert knowledge base based on the segmentation task and the medical image to be segmented, and generate an expert knowledge module activation list, specifically by: constructing a plurality of clinical decision sub-trees, including constructing a clinical decision sub-tree based on expert knowledge and constructing a clinical decision sub-tree based on clinical data; the method of constructing a clinical decision sub-tree based on expert knowledge comprises: obtaining expert knowledge in multiple disease fields, mapping the decision logic in the expert knowledge into a graph, and generating a clinical decision sub-tree; the method of constructing a clinical decision sub-tree based on clinical data comprises: training a decision tree algorithm in machine learning using a clinical data set containing image features and pathological diagnosis, generating a preliminary decision tree, and evaluating, correcting and confirming the preliminary decision tree to obtain a clinical decision sub-tree; performing semantic analysis on a user-specified segmentation task, selecting a clinical decision sub-tree from the constructed plurality of clinical decision sub-trees, and diagnosing the medical image to be segmented based on the selected clinical decision sub-tree to obtain a preliminary diagnosis result of the medical image to be segmented, specifically by: The segmentation task is to determine the text information of the expert knowledge module, to start hierarchical reasoning from the root node of the clinical decision sub-tree based on one or more medical images to be segmented, each non-leaf node of the clinical decision sub-tree representing a key clinical judgment point, and binding a node judgment module for judging the medical image to be segmented, and jumping to the corresponding child node according to the judgment result, and after layer-by-layer progression, reaching a leaf node, each leaf node being a core entity, i.e. the preliminary diagnosis result of the medical image to be segmented; A clinical knowledge graph is constructed, and the preliminary diagnosis result of the medical image to be segmented is input into the clinical knowledge graph for retrieval to generate an expert knowledge module activation list.

4. The medical image segmentation system based on knowledge guidance and prompt generation of claim 3, wherein, The clinical knowledge graph includes a plurality of nodes and relationship edges connecting two nodes, each node being a medical entity, the relationship edge representing the association logic and association weight of the two nodes in the clinic, and each node binding a corresponding expert knowledge module; The specific method for constructing the clinical knowledge graph is: Obtain existing and public medical knowledge bases and ontology libraries, extract a plurality of first medical entities associated with image diagnosis, a plurality of hierarchies of the first medical entities, and a plurality of association relationships of the first medical entities, as a basic skeleton of the preliminary knowledge graph; Obtain medical literature, use a large language model to mine a plurality of second medical entities and a plurality of association relationships between the second medical entities from the unstructured text of the medical literature, take the association relationship between the two second medical entities as a relationship edge, and quantify the association strength between the two second medical entities as the weight of the relationship edge; Add the plurality of second medical entities and the relationship edge to the basic skeleton of the knowledge graph to obtain a preliminary knowledge graph; the medical literature includes textbooks, clinical guidelines, and research papers; The preliminary knowledge graph is evaluated, corrected, and optimized to obtain a clinical knowledge graph; The preliminary diagnosis result of the medical image to be segmented is input into the clinical knowledge graph for retrieval to generate an expert knowledge module activation list, and the specific method is: Taking the node representing the preliminary diagnosis result of the medical image to be segmented as the center, performing graph traversal along the high-weight relationship edge, identifying all first-order neighbor nodes directly associated with the center, obtaining the expert knowledge module bound to each first-order neighbor node, and aggregating all the expert knowledge modules bound to the first-order neighbor nodes to generate an expert knowledge module activation list.

5. The medical image segmentation system based on knowledge guidance and prompt generation of claim 1, wherein, The rich information prompt includes a plurality of prompt objects, a global confidence, and a local confidence of each prompt object; the types of prompt objects include point prompts, box prompts, and text prompts; the global confidence is the confidence of the expert knowledge module on all prompt objects output; and the local confidence is the confidence of the expert knowledge module on each prompt object output.

6. The medical image segmentation system based on knowledge guidance and prompt generation of claim 1, wherein, The specific method for the prompt fuser to determine the optimal aggregated prompt set based on the rich information prompts output by each expert knowledge module is: Convert the discrete rich information prompts output by each expert knowledge module into continuous spatial evidence maps; Fuse the spatial evidence maps of all expert knowledge modules in the expert knowledge module activation list into a final confidence map based on the Dempster-Shafer evidence theory. extract an aggregated cue set from the final confidence map, including a plurality of enhanced point cues and enhanced box cues; obtain a user inputted artificial cue to review, revise and supplement the optimal aggregated cue set output by the fusion cue generator.

7. The medical image segmentation system based on knowledge guidance and prompt generation of claim 6, wherein, The specific method for converting the discrete rich information cues output by each expert knowledge module into a continuous spatial evidence map is: Based on the rich information cues output by each expert knowledge module, a spatial evidence map is generated which is consistent in size with the medical image to be segmented, wherein the value range of each pixel is [0, 1], indicating the degree of certainty that the pixel belongs to the target segmentation region. Different strategies are adopted to generate spatial evidence maps for different types of cue objects in the rich information cues, including: For point hints, a Gaussian kernel function is adopted to model the spatial confidence distribution, which constructs a two-dimensional Gaussian-shaped confidence distribution centered at the location coordinates of the point hint with a peak value of the two-dimensional Gaussian-shaped confidence distribution being the base confidence , as the local confidence, as the global confidence; the standard deviation of the Gaussian kernel is a configurable hyper-parameter to control the influence range of a single point hint; For the frame prompt, a spatial evidence map is generated by using the method of uniform confidence assignment in the region, and the confidence of all pixel points inside the rectangular frame of the frame prompt is uniformly set as the basic confidence ; Text cues do not participate in the generation process of the spatial evidence map, and the text cues are directly collected and transmitted to the general segmentation model with a cue encoder.

8. The medical image segmentation system based on knowledge guidance and prompt generation of claim 7, wherein, Based on the Dempster-Shafer evidence theory, the spatial evidence maps of all expert knowledge modules in the expert knowledge module activation list are fused into a final confidence map, and the specific method is: acquire spatial evidence maps of all expert knowledge modules in the activated list of expert knowledge modules, for each pixel point of the spatial evidence map, define a recognition framework of the pixel point wherein, represents that the pixel point belongs to the target segmentation region, represents that the pixel point does not belong to the target segmentation region; For the first expert knowledge module spatial evidence map , based on the pixel value of any pixel point to build BPA function , for all BPA function of spatial evidence map, pixel by pixel application of Dempster-Shafer evidence theory for iterative fusion, after all pixel by pixel fusion calculation of spatial evidence map, generate a unique, fused all spatial evidence map final confidence map; An aggregated cue set is extracted from the final confidence map, including a plurality of enhanced point cues and enhanced box cues, and the specific method is: A non-maximum suppression algorithm is used to locate high local confidence regions in the final confidence map, find all local confidence peak points in the final confidence map, and regard each peak point as the center of a potential target segmentation region, while suppressing other secondary peaks in the neighborhood of the local confidence peak point. Local topological analysis is performed on the local confidence peak points to evaluate the rate of change of the confidence of the pixel points in the neighborhood of the local confidence peak points. For local confidence peak points with steep gradients and concentrated confidence, the coordinates of the local confidence peak points are extracted and an enhanced point cue is generated. For local confidence peak points with gentle gradients and dispersed confidence, the local confidence peak point is taken as the starting point, and a watershed algorithm is applied to determine the accurate boundary of the high confidence connected domain and calculate its minimum bounding rectangle to obtain an enhanced box cue.

9. The medical image segmentation system based on knowledge guidance and prompt generation of claim 1, wherein, The general segmentation model with a cue encoder includes a cue encoder, an image encoder and a mask decoder. The cue encoder is used to extract a cue embedding vector based on the optimal aggregated cue set. The image encoder is used to encode the medical image to be segmented to obtain medical image encoding features. The mask decoder is used to generate a pixel-level segmentation mask to obtain a segmented medical image. The cue embedding vector and the medical image encoding features extracted by the image encoder are input into the mask decoder module. The cue embedding vector is used as key guidance information to guide the mask decoder module to accurately locate and decode the target region consistent with the cue intention in the global image features, generate a pixel-level segmentation mask, and obtain a segmented medical image.

10. The medical image segmentation system based on knowledge guidance and prompt generation of claim 9, wherein, The specific method for the cue encoder to extract a cue embedding vector based on the optimal aggregated cue set is: For the enhanced point cues in the optimal aggregated cue set, a positional encoding technique is used to encode the two-dimensional spatial coordinates of the enhanced point cues. The mapping is transformed into a high-dimensional vector that carries spatial location information; a type embedding vector for representing semantic type, which is learned through pre-training, is superimposed on the high-dimensional vector to obtain the hint embedding vector for the enhanced point hint; For the enhanced box proposal in the optimal aggregation of proposal set, the left upper corner coordinate point of the enhanced box proposal and the right lower corner coordinate point are normalized and position encoded to obtain a left upper corner position vector and a right lower corner position vector ; a top-left corner position vector and a bottom-right corner position vector are respectively superimposed with a type embedding vector to obtain a top-left corner vector and a bottom-right corner vector as shown in the following formula: wherein, and are the top-left corner type embedding vector and the bottom-right corner type embedding vector, respectively; concatenate the upper-left corner vector and the lower-right corner vector and input a multi-layer perception (MLP) for fusion to obtain a prompt embedding vector of the enhanced bounding box prompt in dimension ; self-attention or cross-attention can also be used to fuse the upper-left corner vector and the lower-right corner vector to obtain a prompt embedding vector of the enhanced bounding box prompt in dimension ; the prompt embedding vector of the enhanced bounding box prompt contains both range information and position information; For text cues in the optimal aggregated cue set, a pre-trained text encoder is used to map the descriptive text into a cue embedding vector of the text cue, which is used to indicate the category, attribute or anatomical orientation of the segmentation target.

Images