Knowledge graph-based auxiliary diagnosis and treatment question-answering method, system, equipment, and medium

Abstract

The present invention provides a knowledge graph-based assisted diagnosis and treatment question-answering method, system, device, and medium. The method comprises: obtaining an original question and an image; processing and fusing them using a CLIP model to obtain fused features; processing the fused features to obtain a segmentation mask; processing the segmentation mask to obtain keywords; processing and concatenating the image and keywords to obtain a prompt vector; processing the prompt vector to obtain a descriptive text, and extracting structured information from the descriptive text; generating a medical question based on the original question and the structured information; and obtaining an answer based on the medical question. By fusing the original question and the image to obtain a medical question, and then obtaining an answer from the knowledge graph based on this medical question, the method avoids the limitation of users being limited to text input and allows users to simultaneously take a picture and enter some prompt text to ask assisted diagnosis and treatment questions.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a knowledge graph-based auxiliary diagnosis and treatment question-answering method, system, device, and medium. Background Art

[0002] With the development of artificial intelligence and big data technologies, the medical industry is accelerating its transformation to intelligent systems. Various intelligent medical question-answering systems have emerged, aiming to assist doctors in diagnosis and patient consultations through natural language processing. These systems analyze the natural medical language of user questions, convert them into machine-understandable information, retrieve relevant diagnostic and treatment knowledge from a knowledge base or knowledge graph, and return answers. However, because medical questions contain a large number of specialized terms and are expressed in a variety of ways, these systems place high demands on word segmentation, entity recognition, and semantic understanding. In practical applications, medical data comes from complex sources and in diverse formats, making the construction and maintenance of high-quality medical knowledge bases or knowledge graphs a key issue.

[0003] Existing technologies generally use knowledge graph technology to structure medical knowledge to support the efficient operation of medical intelligent question-answering systems. However, existing solutions still have several technical defects in practical applications: First, medical data sources are heterogeneous and of varying quality, and there are problems with insufficient accuracy in entity recognition and relationship extraction during the knowledge extraction process. Second, the entity alignment and fusion technology is single, which can easily lead to an increase in redundant nodes in the knowledge graph, affecting query efficiency. Third, it is difficult to parse user natural language questions, and existing systems find it difficult to accurately match complex questions with graph entities, resulting in reduced question-answering accuracy. Fourth, graph retrieval performance is limited by the complex graph structure, slow response speed, and the lack of an efficient knowledge update and expansion mechanism, making it difficult to adapt to the rapid evolution of medical knowledge.

[0004] Although existing technologies have improved the performance of question-answering systems to a certain extent, they still have shortcomings in practical applications. Specifically, in terms of Chinese natural language processing, the existing systems have low accuracy in word segmentation and semantic understanding due to the lack of standardized medical vocabulary and annotated data, which affects the entity matching and answer retrieval effects. In terms of entity recognition and intent recognition, the accuracy is insufficient and cannot fully understand the user's complex medical question intentions. In terms of multi-source knowledge fusion, there is a lack of effective entity alignment and de-redundancy mechanisms, which leads to an increase in redundant information in the knowledge base and a decrease in retrieval efficiency and answer quality. In addition, the system architecture is closed, with poor scalability and portability, making it difficult to adapt to the needs of rapid updates of medical knowledge and multi-scenario applications. Therefore, there is an urgent need to improve the existing technologies to enhance the overall accuracy, query efficiency and application flexibility of the medical question-answering system. Summary of the Invention

[0005] In view of the above-mentioned defects in the prior art, the present invention provides an auxiliary diagnosis and treatment question-answering method and system, equipment, and medium based on knowledge graph to solve the technical problems in the prior art of inaccurate intent recognition and mismatch between entities in user expressions and knowledge graph entities.

[0006] To achieve the above-mentioned and other related purposes, the present invention provides an auxiliary diagnosis and treatment question-answering method based on a knowledge graph, comprising: obtaining an original question and a picture input by a user; processing the original question and the picture using a CLIP model and fusing them to obtain a fusion feature; processing the fusion feature using a decoder to obtain a segmentation mask to find the area related to the original question; processing the segmentation mask using a multi-head attention mechanism based on a preset keyword library to filter out the most relevant keywords for the area related to the original question; processing the picture and the keywords using a CLIP model and then splicing them to obtain a prompt vector; processing the prompt vector using an LLM model to obtain a description text, and extracting structured information from the description text; generating a medical question that integrates image information based on the original question and the structured information; and obtaining answer content from a knowledge graph based on the medical question.

[0007] In one embodiment of the present invention, the original question and the image are processed using a CLIP model and then fused to obtain fused features, including: using the CLIP model to process the image and the original question respectively to obtain image features and text features; adding the image features and the text features element by element, and then performing layer normalization processing; concatenating the image features and the text features and then inputting them into a multilayer perceptron for nonlinear transformation; and adding the features after the layer normalization processing and the features output by the multilayer perceptron to obtain the fused features.

[0008] In one embodiment of the present invention, the fused features are processed using a decoder to obtain a segmentation mask, including: processing the fused features using multiple convolutional layers to obtain multiple features of different scales; upsampling the features of different scales to restore the features of different scales to the size of the fused features; adding the multiple upsampled features element by element, and then processing them through a convolutional layer to obtain a single-channel mask feature; and processing the single-channel mask feature using a sigmoid function to obtain the segmentation mask.

[0009] In one embodiment of the present invention, the segmentation mask is processed using a multi-head attention mechanism based on a preset keyword library to filter out the most relevant keywords for the area related to the original question, including: using multiple attention heads to respectively calculate the semantic similarity between the segmentation mask and each keyword in the keyword library; splicing the outputs of the multiple attention heads and integrating them through linear transformation to obtain a matching matrix; and filtering out the keywords based on the maximum value in the matching matrix.

[0010] In one embodiment of the present invention, the prompt vector is processed using an LLM model to obtain a description text, and structured information is extracted from the description text, including: converting the prompt vector into a continuous vector representation through the embedding layer of the LLM model; using the multi-layer Transformer structure and autoregressive mechanism inside the decoder of the LLM model to process the continuous vector representation to obtain the description text; and decomposing the description text into the structured information through natural language processing technology, wherein the structured information includes lesion location, morphological characteristics, and clinical prompts.

[0011] In one embodiment of the present invention, based on the medical question, answer content is obtained from the knowledge graph, including: using a pre-trained first language model and a first neural network to process the medical question to obtain user intent; processing the medical question to obtain a first entity, and linking the first entity to a node in the knowledge graph to obtain a standard entity name; based on the user intent, selecting a corresponding query template from a predefined sentence template, and filling the standard entity name into the query template to obtain a query statement; based on the query statement, querying and returning entities and their attributes from the knowledge graph, and generating answer content in combination with the user intent.

[0012] In one embodiment of the present invention, the first language model is a first BERT language model, and the first neural network is a TextCNN neural network; the medical question is processed using the pre-trained first language model and the first neural network to obtain the user intent, including: using the pre-trained first BERT language model to semantically encode the medical question to obtain a high-dimensional semantic vector of the medical question; inputting the high-dimensional semantic vector of the medical question into the pre-trained TextCNN neural network, extracting local features in the vector through a convolution operation and performing nonlinear activation and pooling processing to obtain a probability distribution of the user intent; and obtaining the user intent based on the maximum probability of the probability distribution of the user intent.

[0013] In one embodiment of the present invention, the first entity is linked to a node in the knowledge graph to obtain a standard entity name, including: according to the entity name or description corresponding to the node in the knowledge graph, querying whether there is a node with the same name or alias as the first entity: if so, using the name of the corresponding node in the knowledge graph as the standard entity name; if not, using a second BERT language model to calculate the semantic similarity between the name or description of the first entity and the entity corresponding to the node in the knowledge graph, and selecting the name of the node with the highest semantic similarity as the standard entity name.

[0014] In one embodiment of the present invention, the knowledge graph is constructed through the following steps: obtaining medical text; using a named entity recognition module to process the medical text to obtain an optimal entity label sequence corresponding to the medical text; splitting the optimal entity label sequence according to attribute mapping and syntactic templates based on domain rules to obtain entities and their corresponding attributes; using a relationship extraction module to extract semantic relationships between all entities to construct relationships between entities; and constructing the knowledge graph based on the entities and their corresponding attributes and the relationships between the entities.

[0015] In one embodiment of the present invention, the medical text data is processed using an entity recognition model to obtain an optimal entity label sequence corresponding to the medical text, including: using the RoBERTa pre-trained language model to perform word segmentation and token encoding on the medical text to obtain a context-sensitive word vector sequence; using a bidirectional long short-term memory network to capture the front and back semantic dependencies of the word vector sequence to obtain a bidirectional sequence representation that integrates the local context; using a multi-head self-attention mechanism to perform cross-token semantic association modeling on the bidirectional sequence representation, and aggregating multi-head outputs to obtain a semantically enhanced vector representation; using a conditional random field to perform label prediction on each position in the semantically enhanced vector representation to obtain the optimal entity label sequence corresponding to the medical text.

[0016] In one embodiment of the present invention, a conditional random field is used to perform label prediction on each position in the semantically enhanced vector representation to obtain the optimal entity label sequence corresponding to the medical text, including: mapping the semantically enhanced vector representation to the label space through a linear transformation to obtain a label emission score for each position in the sequence; calculating the final scores of all possible label sequence paths through dynamic programming using the Viterbi algorithm based on the emission scores and the transfer matrix in the conditional random field; selecting the label sequence path with the highest final score as the optimal path, and using the label sequence corresponding to the optimal path as the optimal entity label sequence.

[0017] In one embodiment of the present invention, after splitting the optimal entity label sequence according to the domain rule-based attribute mapping and syntactic template to obtain the entity and its corresponding attribute, the step also includes: using a pre-trained third BERT language model to calculate the semantic similarity between different entity names and descriptions, and performing entity fusion based on the similarity calculation results; the knowledge graph is constructed based on the fused entities.

[0018] In order to achieve the above-mentioned purpose and other related purposes, the present invention also provides an auxiliary diagnosis and treatment question answering system based on knowledge graph, comprising: a request acquisition module for acquiring the original question and picture input by the user; an image and text embedding fusion module for processing the original question and the picture using the CLIP model and then fusing them to obtain fusion features; a fine-grained decoding segmentation module for processing the fusion features using a decoder to obtain a segmentation mask to find the area related to the original question; a region and keyword matching module for using a multi-head attention mechanism to match the segmentation features to the preset keyword library. The system comprises a plurality of modules, a plurality of modules, a plurality of modules, and a plurality of modules. The plurality of modules are configured to perform mask processing to filter out the most relevant keywords in the area related to the original question; a prompt vector calculation module is used to process the original question and the keywords using the CLIP model and then splice them to obtain a prompt vector; a fine-grained description generation module is used to process the prompt vector using the LLM model to obtain a description text and extract structured information from the description text; a medical question generation module is used to generate a medical question that integrates image information based on the original question and the structured information; and a question and answer generation module is used to obtain the answer content from the knowledge graph based on the medical question.

[0019] In one embodiment of the present invention, the question and answer generation module includes: a user intent recognition module, which is used to use a pre-trained first language model and a first neural network to process the medical question and obtain the user intent; an entity linking module, which is used to process the medical question and obtain a first entity, and link the first entity to a node in the knowledge graph to obtain a standard entity name; a query parsing module, which is used to select a corresponding query template from a predefined sentence template according to the user intent, and fill the standard entity name into the query template to obtain a query statement; an answer generation module, which is used to query and return entities and their attributes from the knowledge graph according to the query statement, and generate answer content in combination with the user intent.

[0020] In one embodiment of the present invention, it also includes: a data acquisition module for acquiring medical text; a named entity recognition module for processing the medical text to obtain the optimal entity label sequence corresponding to the medical text; a label sequence splitting module for splitting the optimal entity label sequence according to the attribute mapping and syntactic template based on domain rules to obtain entities and their corresponding attributes; an entity relationship extraction module for extracting the semantic relationships between all entities to construct the relationships between entities; a knowledge graph construction module for constructing the knowledge graph based on the entities and their corresponding attributes and the relationships between the entities.

[0021] In one embodiment of the present invention, it also includes: an entity alignment and fusion module, which is used to use the pre-trained third BERT language model to calculate the semantic similarity between different entity names and descriptions, and perform entity fusion based on the similarity calculation results.

[0022] To achieve the above-mentioned purpose and other related purposes, the present invention also provides an electronic device, including a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute the computer program stored in the memory to implement the method provided in any one of the above embodiments.

[0023] To achieve the above-mentioned object and other related objects, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to enable a computer to execute the method provided in any one of the above-mentioned embodiments.

[0024] Beneficial effects of the present invention: The present invention proposes an auxiliary diagnosis and treatment question-answering method, system, device, and medium based on a knowledge graph. The method obtains a medical question that integrates image information by fusing the original question and the image, and then obtains the answer content from the knowledge graph based on the medical question. This avoids the limitation that users can only input text. For some scenarios that cannot be well described or the user cannot accurately describe the actual situation, the user is allowed to take pictures and enter some prompt text at the same time to ask auxiliary diagnosis and treatment questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the embodiments or the description of the prior art. The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A flowchart of a question-and-answer method for auxiliary diagnosis and treatment provided by one embodiment of the present invention;

[0027] Figure 2 A detailed flow chart of step S20 provided in one embodiment of the present invention;

[0028] Figure 3 A detailed flowchart of step S30 provided in one embodiment of the present invention;

[0029] Figure 4 A detailed flow chart of step S40 provided in one embodiment of the present invention;

[0030] Figure 5 A detailed flowchart of step S60 provided in one embodiment of the present invention;

[0031] Figure 6 A detailed flow chart of step S80 provided in one embodiment of the present invention;

[0032] Figure 7 A flowchart of user intent recognition provided by one embodiment of the present invention;

[0033] Figure 8 A flowchart for constructing a knowledge graph provided in one embodiment of the present invention;

[0034] Figure 9 A processing flow chart of a named entity recognition module provided in one embodiment of the present invention;

[0035] Figure 10 A schematic diagram of an auxiliary diagnosis and treatment question-answering system provided by an embodiment of the present invention;

[0036] Figure 11 A schematic structural diagram of an electronic device provided by an embodiment of the present invention.

[0037] Description of the accompanying drawings: 101, processor; 102, memory. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present invention through specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. It should be noted that the following embodiments and the features in the embodiments can be combined with each other unless they conflict. In addition to the specific methods, equipment, and materials used in the embodiments, based on the understanding of the prior art by those skilled in the art and the description of the present invention, any methods, equipment, and materials of the prior art that are similar or equivalent to the methods, equipment, and materials in the embodiments of the present invention can also be used to implement the present invention.

[0039] It should be understood that the terms used in the examples of the present invention are for describing specific embodiments rather than for limiting the scope of protection of the present invention. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those generally understood by those skilled in the art.

[0040] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In some of the embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0041] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations that may be implemented by the methods and computer program products of various embodiments disclosed in the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0042] See Figure 1 , Figure 1 An embodiment of the present invention provides an auxiliary diagnosis and treatment question-answering method based on a knowledge graph, comprising steps S10 to S80.

[0043] Step S10: Obtain the original question and picture input by the user. In this step, the user can be either a patient or a doctor, regardless of identity, and anyone can consult. Medical questions do not need to adopt a special format or use professional expression terms, because in the subsequent processing process, user intent understanding and entity linking will be performed, which is more convenient for user input. The original question input by the user can be, for example, "What should I do if my stomach hurts after taking aspirin?", "Can I take nitroglycerin when I have angina pectoris?", "How should the wound in the picture be treated?", etc. The picture can be, for example, an image of the injured area of ​​the hand taken and uploaded by the user.

[0044] Step S20: Use the CLIP model to process the original question and the image and then fuse them to obtain a fusion feature.

[0045] See Figure 2 In a specific embodiment of the present invention, step S20 includes steps S21 to S24.

[0046] Step S21: Use the CLIP model to process the image and the original question respectively to obtain image features and text features. This can be expressed as:

[0047] ;

[0048] ;

[0049] Where I represents the image, T represents the original question, ϕ(I) and ψ(T) are image features and text features, respectively.

[0050] Step S22: After adding the image features and text features element by element, layer normalization (for example, using the LayerNorm layer) is performed to stabilize the training process.

[0051] Step S23: After the image features and text features are spliced ​​together, they are input into a multi-layer perceptron for nonlinear transformation to capture more complex feature interactions.

[0052] Step S24: Add the layer-normalized features to the features output by the multilayer perceptron to obtain fused features. This design preserves the original feature information while enhancing the expressive power of the features through nonlinear transformation. This process enables the full fusion of images and text within the same semantic space.

[0053] These three steps can be expressed as follows:

[0054] ;

[0055] Where F is the fusion feature.

[0056] Step S30: Use the decoder to process the fused features to obtain a segmentation mask to find the area related to the original question.

[0057] See Figure 3In a specific embodiment of the present invention, step S30 includes: S31, processing the fused features using multiple convolutional layers to obtain multiple features at different scales; S32, upsampling the features at different scales to restore them to the size of the fused features; S33, element-wise adding the multiple upsampled features and then processing them through a convolutional layer to obtain a single-channel mask feature; S34, processing the single-channel mask feature using a sigmoid function and squeezing the output value to the range [0, 1] to obtain a segmentation mask. The segmentation mask is used to represent the probability that each pixel belongs to a region related to the original question.

[0058] Step S40: Based on the preset keyword library, the segmentation mask is processed using a multi-head attention mechanism to filter out the most relevant keywords in the area related to the original question.

[0059] See Figure 4 In a specific embodiment of the present invention, step S40 includes steps S41 to S43.

[0060] Step S41: Use multiple attention heads to calculate the semantic similarity between the segmentation mask and each keyword in the keyword library. This step can be expressed as:

[0061] ;

[0062] Where, Head i represents the output feature of the i-th attention head, M is the segmentation mask obtained in step S30, and K is the keyword library. The multi-head attention mechanism calculates the semantic similarity between the segmentation mask and each keyword in the keyword library through multiple attention heads.

[0063] Step S42: Concatenate the outputs of multiple attention heads and integrate them through linear transformation to obtain a matching matrix. This step can be expressed as:

[0064] ;

[0065] Where β is the final matching matrix, W O is the linear transformation matrix.

[0066] Step S43: Filter out keywords based on the maximum value in the matching matrix, providing key semantic information for subsequent description generation. This step can be expressed as:

[0067] ;

[0068] In the formula, k is the filtered keyword.

[0069] Step S50: Use the CLIP model to process the image and keywords and then splice them to obtain a prompt vector. This step can be expressed as:

[0070] ;

[0071] Where ψ(k) is the text feature corresponding to the keyword obtained by processing the keyword using the CLIP model.

[0072] Step S60: Use the LLM model to process the prompt vector to obtain a description text, and extract structured information from the description text.

[0073] See Figure 5 In a specific embodiment of the present invention, step S60 includes: S61, converting the prompt vector into a continuous vector representation through the embedding layer of the LLM model; S62, using the multi-layer Transformer structure and autoregressive mechanism inside the decoder of the LLM model to process the continuous vector representation to obtain a description text; S63, using natural language processing technology, decomposing the description text into structured information, the structured information including lesion location, morphological characteristics and clinical prompts.

[0074] The above steps S61 to S63 can be expressed by the formula:

[0075] ;

[0076] ;

[0077] In the formula, Prompt is the prompt vector, Embed is the embedding layer, and Decoder is the LLM decoder. Among the decomposed features, Location is the lesion location information, Morphology is the morphological feature information, and ClinicalTip is the clinical tip information. For example, if a user uploads a tongue image and the original question "Judgment of my health status based on the image," then by processing the original question and image, the structured information obtained may be, for example, "There is erythema in the middle of the tongue, the tongue coating is thick and greasy, the color is pale, the edges are tooth-marked, and the shape is oval."

[0078] Step S70: Generate a medical question incorporating the image information based on the original question and structured information. Taking the original question and structured information in the previous paragraph as an example, the resulting medical question might be, "I have red spots in the middle of my tongue, a thick, greasy coating, a pale color, tooth marks on the edges, and an oval shape. Please judge my health status."

[0079] Step S80 obtains the answer content from the knowledge graph based on the medical question.

[0080] See Figure 6 In a specific embodiment of the present invention, step S80 includes steps S200 to S500.

[0081] Step S200: Process the medical question using the pre-trained first language model and the first neural network to obtain the user's intention. In order to accurately answer the medical question, it is necessary to first understand the user's intention.

[0082] See Figure 7 In a specific embodiment of the present invention, the first language model is a first BERT language model, and the first neural network is a TextCNN neural network. Step S200 specifically includes: S201, using the pre-trained first BERT language model to semantically encode the medical question to obtain a high-dimensional semantic vector of the medical question; S202, inputting the high-dimensional semantic vector of the medical question into the pre-trained TextCNN neural network, extracting local features from the vector through convolution operations, and performing nonlinear activation and pooling processing to obtain a probability distribution of user intent; S203, obtaining the user intent based on the maximum probability of the probability distribution of user intent.

[0083] Through steps S201-S203, the present invention accurately identifies the user's intent regarding disease etiology, symptom analysis, medication recommendations, and other related questions. This facilitates mapping natural language questions to pre-set Q&A templates or query types, significantly improving the accuracy of user intent recognition. The intent recognition technology of this embodiment effectively addresses the recognition challenges presented by the diverse and complex nature of natural language questions, enhancing the Q&A system's ability to understand user needs.

[0084] In this embodiment, the pre-trained first BERT language model is first used to obtain the contextual semantic representation of the question, and then its output is introduced into the TextCNN neural network for convolution operation. The TextCNN neural network extracts local key information, such as keywords and phrase features, through multiple convolution kernels of different widths to supplement the long-range semantics learned by the first BERT language model. The convolution layer is followed by pooling and fully connected layers, and finally outputs the probability distribution of intent categories. Combining BERT's global understanding ability and TextCNN's sensitivity to local features, it is possible to more accurately judge the type of question in medical question-answering scenarios.

[0085] It's important to note that the types of user intent are predefined. The TextCNN neural network is trained based on these predefined user intents, enabling it to output the probability distribution corresponding to each user intent. This step also supports the expansion of intent categories. Incremental training can be completed by simply adding new category labels to the training data and refining the samples. Common user intents include medication consultation, first aid guidance, and side effect management.

[0086] Step S300: Process the medical question to obtain the first entity, and link the first entity to the node in the knowledge graph to obtain the standard entity name. The entity resolution process for medical questions is the same as that for knowledge graph construction, which will be described in detail in subsequent steps. The first entity in the medical question is linked before the knowledge graph query, so that the name of the first entity can be standardized. This solves the difficulty of entity matching caused by the diversity of user expressions, ensures the accurate positioning of entities during the query process, and thus improves the hit rate and accuracy of questions and answers.

[0087] In a specific embodiment of the present invention, the first entity is linked to a node in the knowledge graph to obtain a standard entity name, including: based on the entity name or description corresponding to the node in the knowledge graph, querying whether there is a node with the same name or alias as the first entity: if so, using the name of the corresponding node in the knowledge graph as the standard entity name; if not, using a second BERT language model to calculate the semantic similarity between the name or description of the first entity and the entity corresponding to the node in the knowledge graph, and selecting the name of the node with the highest semantic similarity as the standard entity name. In this step, a correct mapping is established between the name of the first entity and the entity node in the knowledge graph through literal matching, alias dictionary or semantic-based similarity calculation. For example, for abbreviations or synonyms entered by the user, this step can find the corresponding standard entity and use its index to perform a query.

[0088] In this step, we first directly determine whether the entity name or alias is exactly the same. If they are exactly the same, it means that the user's expression of the entity is relatively accurate, and the name of the corresponding node can be directly used as the standard entity name; otherwise, the semantic similarity calculation is required through the second BERT language model. If the semantic similarity is low or there is ambiguity, the candidate screening can also be carried out based on the contextual intent and query results, and feedback to the user for confirmation when necessary. Entity linking accurately maps the concepts in the question to the entity nodes of the knowledge graph to ensure that subsequent queries can accurately locate the target information. In addition, alias dictionaries and entity alias rules can be configured to support the online addition and rapid update of new entities, thereby having better maintainability.

[0089] Step S400: Based on the user's intent, a corresponding query template is selected from the predefined sentence templates, and the standard entity name is filled into the query template to obtain a query statement. This step mainly parses the medical question into a knowledge graph query statement based on the above-mentioned intent recognition and entity linking results to obtain accurate diagnosis and treatment answers. Among them, through the predefined sentence templates, each user intent corresponds to a query template, and then the linked entity is substituted into the template to generate a complete knowledge graph query statement (such as Neo4j's Cypher query). When constructing this query statement, it fully utilizes the index and relational structure of the graph database, allowing the system to efficiently locate the medical entities and their attributes related to the user's needs and quickly return the answer. For example, during the query process, only the relationship edges associated with the matching entities need to be accessed, thus avoiding the additional overhead caused by the full graph search. In this way, efficient knowledge graph retrieval is achieved, further improving the query efficiency and response speed of the system, and effectively alleviating the query burden problem after the fusion of multi-source data.

[0090] Step S500: Based on the query statement, query and return entities and their attributes from the knowledge graph, and generate answer content in combination with user intent. In this step, for the entities and their attribute information returned by the query, the structured data can be organized into natural language answers in combination with user intent and according to predefined answer templates or rules. For example, information such as the name of the disease, symptom description, and recommended treatment plan can be spliced ​​together to form a complete diagnosis and treatment recommendation statement. In addition, this step also supports the generation of answers in the context of multiple rounds of dialogue, such as supplementing or modifying the previous answer in continuous questions and answers. By converting the information retrieved from the knowledge graph into diagnosis and treatment recommendations that are easy for users to understand, the user experience and the practicality of the question-and-answer system are improved.

[0091] In each of the above steps, the knowledge graph is pre-constructed. You can use the existing knowledge graph in the existing technology or construct it yourself.

[0092] See Figure 8 In a specific embodiment of the present invention, the knowledge graph is constructed through steps S510 to S550.

[0093] Step S510: Obtain medical text. Specifically, diagnosis and treatment-related data can be collected from multiple medical resources (such as electronic medical records, medical literature, and public medical databases). These relevant data can be pre-processed, cleaned, and deduplicated to obtain high-quality medical text.

[0094] Step S520: Use the named entity recognition module to process the medical text to obtain the optimal entity label sequence corresponding to the medical text.

[0095] See Figure 9In a specific embodiment of the present invention, step S520 includes steps S521 to S524.

[0096] Step S521: Use the RoBERTa pre-trained language model to perform word segmentation and token encoding on the medical text to obtain a context-sensitive word vector sequence. The RoBERTa pre-trained language model is responsible for converting the input medical text into a vector representation that the model can process. It is equivalent to the embedding layer. The BERT model is composed of word vectors (Token Embedding), position encoding (Position Embedding), and sentence encoding (Segment Embedding), of which word vectors and position encoding are key components of the embedding layer. In this step, a strongly optimized version of the BERT model, namely the RoBERTa pre-trained language model, is used as the embedding layer. By training on longer sentences, optimization of the training method is achieved.

[0097] Based on the BERT model (hereinafter referred to as BERT), the RoBERTa pre-trained language model (hereinafter referred to as RoBERTa) has made the following specific improvements: (1) Changing to a dynamic masking mechanism: RoBERTa changes from BERT's static masking mechanism to a dynamic masking mechanism. During the training of the original BERT, the mask tags for different batches of text sequences may be the same, which means that although the training dataset is copied multiple times, each sequence is actually masked in only a limited number of different ways. Furthermore, since BERT runs multiple training epochs, this means that each sequence with the same mask will be passed to BERT training multiple times. It was later discovered that using dynamic masks can solve this problem, because dynamic masks will generate a new "[Mask]" for each sequence passed into the model, improving the model's ability to adapt to different masking patterns during training, helping the model learn richer sequence representations, and improving the model's generalization ability. (2) Removing the NSP task: RoBERTa removes the NSP task in BERT. The purpose of NSP is to determine whether two sentences are coherent. It is not necessary for downstream tasks and may even introduce noise. Removing the NSP task does not affect the performance of the model on downstream tasks, and even slightly improves it. Therefore, canceling the NSP task allows the model to focus more on text encoding and representation learning. (3) Larger Batch Size: At the data level, RoBERTa also adds a large-scale corpus for pre-training. Compared with the 16GB training data used by BERT, RoBERTa has increased to 160GB, which helps to more accurately capture the bidirectional relationship in the text and improve the overall performance of the model. (4) Byte text encoding: In addition, the original BERT uses a character-level BPE (Byte-Pair Encoding) word segmentation method with a size of 30K, while RoBERTa chooses a larger byte-level BPE vocabulary BBPE (Bytes-level BPE, containing 50K subword units) as the minimum unit for constructing subwords to train BERT, enhancing the ability to understand rare and complex texts.

[0098] Step S522: Use a bidirectional long short-term memory network to capture the semantic dependencies between the previous and next word vector sequences, and obtain a bidirectional sequence representation that integrates the local context. The bidirectional long short-term memory network learns context-related vectors through the representor. Traditional methods such as CNN, RNN or unidirectional LSTM have some limitations in entity extraction tasks. For example, these methods do not take into account that the importance of different word units in the named entity recognition task may be different, and insufficient attention may be allocated to some elements. Therefore, considering these problems, we combine the bidirectional long short-term memory network (BiLSTM) and the multi-head self-attention mechanism (MHA) to form the encoding layer of the model, and further process the vector output by the RoBERTa layer. In this step, the LSTM is improved by adding a reverse process, and the contextual information of the medical data is automatically learned through the bidirectional LSTM.

[0099] Step S523: Use a multi-head self-attention mechanism to model cross-token semantic associations in the bidirectional sequence representation, aggregating the multi-head outputs to obtain a semantically enhanced vector representation. As mentioned above, the encoding layer also includes a multi-head self-attention mechanism (MHA). This mechanism is introduced to improve the model's ability to capture interactive features between elements of any distance, further enhancing entity extraction performance.

[0100] The output matrix H of the bidirectional long short-term memory network (BiLSTM) is used as the input of the multi-head self-attention mechanism (MHA), and then processed according to the following steps: First, each feature of the input sequence is divided into several groups as heads, and each head is independently transformed by Q, K, and V, where Q, K, and V are query, key, and value respectively, X is the input matrix, and W represents the corresponding weight matrix; Second, the attention score is calculated using Q and K by dot product, and the attention score is normalized by applying the maxsoft function to generate attention weights, and the attention weights are weighted summed to obtain the specific representation of the input sequence by the "head"; Third, the outputs of the h "heads" are spliced ​​together and linearly transformed through the output weight matrix Wo to obtain the final output matrix.

[0101] Step S524: Use a conditional random field to predict labels for each position in the semantically enhanced vector representation to obtain the optimal entity label sequence corresponding to the medical text. The conditional random field (CRF) is equivalent to the decoding layer, which assigns an entity type to each word or phrase, such as symptom, disease, or drug.

[0102] In a specific embodiment of the present invention, step S524 includes the following steps: (1) mapping the semantically enhanced vector representation to the label space through linear transformation to obtain the label emission score of each position in the sequence; (2) calculating the final scores of all possible label sequence paths through dynamic programming of the Viterbi algorithm based on the emission scores and the transfer matrix in the conditional random field; (3) selecting the label sequence path with the highest final score as the optimal path, and using the label sequence corresponding to the optimal path as the optimal entity label sequence.

[0103] Step S524 combines the output of the previous BiLSTM-MHA layer with the label prediction for each position in the sequence. These labels typically have dependencies. Without a conditional random field, the encoding layer ignores these dependencies, and the model predicts labels for each independent position, leading to inconsistencies in the label sequence. Therefore, in the NER task (i.e., named entity recognition) of this invention, a linear chain conditional random field is used to receive the output from the previous layer. To reflect the dependencies between labels, the conditional random field (CRF) adds a transition matrix M to describe the transition probabilities between different labels. It considers all possible sequence paths and calculates path scores.

[0104] The calculation formula for the final score S in step (2) above is as follows:

[0105] ;

[0106] Where M i,j represents the transfer matrix score from label i to label j, E i Represents the emission score, and the path with the highest score is the correct labeling path. Specifically, the Viterbi algorithm can be used to find the label with the highest score to predict the most likely entity label. The optimal label calculation formula is as follows:

[0107] ;

[0108] In the formula is the predicted optimal label sequence, y is the label sequence (i.e., the entity label corresponding to each word), and X is the input sequence (i.e., the words or characters of the original text).

[0109] For the conditional random field (CRF), the transfer matrix M needs to be obtained through training. The parameters are optimized during the training process by introducing the negative log-likelihood of the true label sequence to calculate the loss function and minimize it. The calculation formula of the loss function is as follows:

[0110] ;

[0111] ;

[0112] Where y true Indicates the correct path, y n represents the nth path, P(y true |x) represents the probability distribution of the correct path, S i represents the final score of the i-th path.

[0113] The named entity recognition module built based on RoBERTa-BiLSTM-MHA-CRF can more accurately identify various entity boundaries and entity types in Chinese medical questions, thereby significantly improving the accuracy of named entity recognition and effectively solving the technical problem of low entity recognition accuracy in the medical field by traditional methods.

[0114] Taking the input medical text "The patient has a history of hypertension and diabetes" as an example, after processing the above steps S521 to S524, the optimal entity label sequence obtained is, for example: ["O", "O", "O", "B-Disease", "I-Disease", "I-Disease", "O", "B-Disease", "I-Disease", "I-Disease", "O"].

[0115] Step S530: Split the optimal entity tag sequence based on the domain rule-based attribute mapping and syntax template to obtain the entity and its corresponding attributes. The optimal entity tag sequence obtained in step S520 is just a set of tag sequences. Through this step, the entity and its corresponding attributes can be obtained. The entity is standardized, and its corresponding attributes are also formatted, which is more conducive to the construction of the knowledge graph.

[0116] Step S540: Use the relationship extraction module to extract the semantic relationships between all entities to construct the relationships between entities. By analyzing the semantic and syntactic information, the entity pairs output by the named entity recognition module are predicted to construct the relationship edges between entities. Specifically, for example, rule-based methods or deep learning methods can be used to extract common entity relationship types in the medical field, such as "disease-symptoms", "disease-examination", "drug-therapy", etc. The extracted relationships are used to form triples in the knowledge graph and are stored in the graph database together with the entity information. Through the relationship extraction module, rich connections between entities are established in the knowledge graph, enabling the question-answering system to query based on the relationship network between entities, thereby indirectly improving the accuracy and efficiency of question-answering.

[0117] Step S550: Construct a knowledge graph based on the entities, their corresponding attributes, and the relationships between them. When constructing the knowledge graph, a graph database (such as Neo4j) can be used for storage and management. Each entity is defined as a node in the knowledge graph. The semantic relationships between entities are connected by edges, and attribute information is added to the nodes and edges. Graph databases offer efficient indexing and query capabilities, allowing for rapid location of target nodes based on entity labels and attributes. Using a graph database for storage ensures efficient management of large-scale medical knowledge and provides technical support for addressing the decline in query efficiency caused by multi-source data fusion.

[0118] In a specific embodiment of the present invention, after step S530, the following further includes: S560, using a pre-trained third BERT language model to calculate semantic similarities between different entity names and descriptions, and performing entity fusion based on the similarity calculation results. Based on this, when constructing the knowledge graph, the knowledge graph is constructed based on the fused entities.

[0119] In response to the redundancy problem after the fusion of multi-source heterogeneous medical data, the present invention uses the pre-trained third BERT language model to calculate the semantic similarity between different entity names or descriptions after extracting candidate entities and their attributes, and matches and merges entities with high similarity. For entities that are judged to be the same, the present invention unifies them into one entity node and integrates their attribute information, thereby eliminating redundant entities caused by multi-source data. The fused entities and their relationships are stored in the graph database in the form of triples to construct a complete and consistent medical knowledge graph. Through step S560, the redundancy of multi-source data is effectively eliminated, the efficiency and reliability of knowledge graph queries are improved, and the serious redundancy problem of multi-source data fusion is solved.

[0120] It should be noted that the step division of the various methods above is only for the purpose of clear description. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0121] An embodiment of the present invention also provides an auxiliary diagnosis and treatment question-answering system based on a knowledge graph, which includes a request acquisition module, an image and text embedding fusion module, a fine-grained decoding and segmentation module, a region and keyword matching module, a prompt vector calculation module, a fine-grained description generation module, a medical question generation module, and a question-answer generation module. Among them, the request acquisition module is used to obtain the original question and picture input by the user; the image and text embedding fusion module is used to use the CLIP model to process the original question and picture and then fuse them to obtain fusion features; the fine-grained decoding and segmentation module is used to use the decoder to process the fusion features to obtain a segmentation mask to find the area related to the original question; the area and keyword matching module is used to process the segmentation mask based on the preset keyword library using the multi-head attention mechanism to filter out the most relevant keywords for the area related to the original question; the prompt vector calculation module is used to use the CLIP model to process the original question and keywords and then splice them to obtain a prompt vector; the fine-grained description generation module is used to use the LLM model to process the prompt vector to obtain a description text and extract structured information from the description text; the medical question generation module is used to generate a medical question that integrates image information based on the original question and structured information; the question and answer generation module is used to obtain the answer content from the knowledge graph based on the medical question.

[0122] It should be noted that the auxiliary diagnosis and treatment question-and-answer system of this embodiment corresponds to the auxiliary diagnosis and treatment question-and-answer method described above, and the functional modules in the auxiliary diagnosis and treatment question-and-answer system correspond to the corresponding steps in the auxiliary diagnosis and treatment question-and-answer method. The auxiliary diagnosis and treatment question-and-answer system of this embodiment can be implemented in conjunction with the auxiliary diagnosis and treatment question-and-answer method. That is, unless there is a conflict, the relevant technical details mentioned in the auxiliary diagnosis and treatment question-and-answer method of the above embodiment can also be applied to the auxiliary diagnosis and treatment question-and-answer system of this embodiment.

[0123] In addition to the above modules, the knowledge graph-based auxiliary diagnosis and treatment question-answering system can also include the following modules.

[0124] The data collection module is used to obtain medical texts. This module is responsible for collecting diagnosis and treatment-related data from multiple medical resources (such as electronic medical records, medical literature, and public medical databases).

[0125] The data preprocessing module is used to preprocess the collected data. Preprocessing operations include data format conversion, word segmentation, entity candidate extraction, and attribute standardization. This removes noise and unifies the data format to provide high-quality input for subsequent processing. This module ensures that medical data from different sources remains consistent in semantics and format, laying the foundation for subsequent entity recognition and knowledge fusion, and reducing redundancy and ambiguity caused by data heterogeneity.

[0126] The named entity recognition module processes the medical text to obtain the optimal entity label sequence corresponding to the medical text. This module uses a combined model based on the RoBERTa pre-trained language model, a bidirectional long short-term memory network (BiLSTM), a multi-head self-attention mechanism, and a conditional random field (CRF) to perform entity recognition on the pre-processed medical text. This module corresponds to steps S521-S524 described above. This module can accurately identify entities such as disease names, examination items, and medication regimens in medical questions and determine their boundaries and types. This significantly improves the accuracy of entity recognition in Chinese medical text and effectively alleviates the low accuracy of traditional methods for entity recognition in the medical field.

[0127] The tag sequence splitting module is used to split the optimal entity tag sequence according to the domain rule-based attribute mapping and syntax template to obtain entities and their corresponding attributes. This step corresponds to step S530 above. This module is used to convert the optimal entity tag sequence into entities and their corresponding attributes.

[0128] The entity relationship extraction module extracts the semantic relationships between all entities to build relationships between them. This module first uses the named entity recognition module to extract medical entities such as diseases, symptoms, drugs, and treatment plans from the text. It then extracts semantic relationships between entities (such as disease-symptom, disease-treatment, and drug-side effect relationships) through rule matching, template patterns, or grammatical dependency analysis.

[0129] The knowledge graph construction module is used to construct the knowledge graph based on the entities, their corresponding attributes, and the relationships between them. Taking Neo4j graph database storage as an example, the constructed entities and their relationships are stored in the Neo4j graph database in the form of a knowledge graph. As a native graph database, Neo4j can intuitively store medical knowledge in the form of nodes (entities) and edges (relationships). Labels and attribute indexes are created in Neo4j for different entity categories (e.g., disease, symptom, treatment, examination, etc.), and fixed identifiers are also used for relationship types.

[0130] The entity alignment and fusion module is used to calculate the semantic similarity between different entity names and descriptions using the pre-trained third-party BERT language model, and to perform entity fusion based on the similarity calculation results. This module aligns and fuses synonymous and duplicate entities in multi-source heterogeneous data. The pre-trained third-party BERT language model is used to calculate the semantic representation of candidate entities, and entities with similar expressions in different data sources are matched; for entities determined to be the same, they are unified into a single node in the knowledge graph, and their attribute information is merged to eliminate data redundancy. The fused entities and their relationships are updated to the graph database in the form of triples to ensure the consistency and integrity of the knowledge graph. The entity alignment and fusion module effectively solves the problem of redundant entities in the multi-source data fusion process, improving the quality of the knowledge graph and query efficiency.

[0131] The system login and user management module is responsible for user identity authentication and permission management. Users must pass login verification before accessing the Q&A function, and different user roles have varying access rights. System administrators can use this module to create and manage user accounts and set access control policies to ensure the secure use of medical data and knowledge. Furthermore, this module records user operation logs, providing a basis for subsequent audits and analysis. Through user management capabilities, the system ensures security and stability in a multi-user environment.

[0132] The knowledge update and maintenance module is used to dynamically update the medical knowledge graph to cope with the continuous emergence of new knowledge. The system can regularly crawl information from authoritative medical data sources (such as the latest published guidelines, clinical trial data, etc.), and automatically perform processes such as entity recognition, relationship extraction, and entity alignment to integrate new knowledge into the existing knowledge base in the form of triples. In addition, the system also provides a knowledge review tool that allows experts to verify or modify the automatically extracted knowledge. Through a continuous knowledge update mechanism, this module ensures that the medical knowledge integrated in the system remains timely and complete, thereby effectively improving the scalability of the system and solving the problem of difficulty in adapting new medical knowledge.

[0133] The system management module is used for operational monitoring and configuration management of the entire question-and-answer system. System administrators can use this module to view the operational status, performance indicators, and log information of each functional module, and to set or adjust system parameters. This module supports dynamic load balancing and cluster expansion, automatically allocating computing resources based on access volume to ensure stable system operation even under high concurrency conditions. The system management module is also responsible for operational maintenance tasks such as version updates and backup management, ensuring smooth system iteration and data security. Through centralized management and monitoring, this module effectively improves the system's reliability and maintainability, enhancing its applicability in complex environments.

[0134] Knowledge Graph Visualization Module: This module provides a visual interface to help users intuitively browse and analyze entities and relationships within the medical knowledge graph. Users can use the interactive graphical interface to view knowledge networks related to specific diseases or medical concepts, zooming in, out, and filtering nodes and edges. The visualization module also supports customized local focus on the graph, highlighting key entities and their relationships, and assisting users in understanding the information structure within the knowledge graph. Furthermore, the module can export visualization results as reports or images, facilitating expert review and sharing.

[0135] The adoption of the above-mentioned modular system architecture enhances the scalability of the system. The various functional modules of the system communicate with each other through clear interfaces, are independent of each other and have low coupling; when new functions need to be added or the latest medical knowledge needs to be introduced, it is only necessary to configure and expand the relevant modules without changing the entire system. For example, if a new disease classification or treatment plan is introduced, it can be achieved by updating the knowledge graph construction module or the knowledge update module, and other modules do not need to be modified. The system also provides standardized data interfaces and service interfaces to facilitate integration with third-party data sources, medical equipment or algorithm models, thereby quickly expanding new medical resources and functional components. The above-mentioned modular and interface design ensures that the system can be flexibly expanded as medical knowledge evolves and user needs change, effectively solving the problem of insufficient system scalability.

[0136] See Figure 10 In a specific embodiment of the present invention, in addition to adopting the above modular system design architecture, a hierarchical model can also be used to design the system. Figure 10 In the system, there are data layer, model layer, logic layer and business layer. It is important to note that Figure 10 The sub-modules in the hierarchical design may have different names from the aforementioned modules because they are not in a one-to-one correspondence.

[0137] The data layer is responsible for acquiring, cleaning, structuring, and storing medical knowledge data, providing high-quality foundational information resources for the model and logic layers. Specifically, the data layer includes the following modules: data acquisition and preprocessing, entity relationship building, and Neo4j graph database storage. The data layer design utilizes a modular and configurable pipeline approach, enabling rapid adaptation and integration when changes to data sources or preprocessing processes occur, resulting in excellent maintainability and scalability.

[0138] The model layer stores all models involved in the system, such as the RoBERTa-BiLSTM-MHA-CRF-based named entity recognition model in the named entity recognition module, the BERT-TextCNN-based intent recognition model in the user intent recognition module (BERT can also be replaced by RoBERTa), and the BERT-based sentence similarity calculation entity alignment model in the entity alignment and fusion module.

[0139] The logic layer primarily responds to user queries, analyzing and understanding natural language input questions and ultimately generating answers by leveraging resources provided by the model and data layers. Core functions include a request acquisition module, a user intent recognition module (excluding the model), an entity linking module, a query parsing module, and an answer generation module.

[0140] The business layer is responsible for the system's user interface, backend service deployment and maintenance, and high-concurrency processing, ensuring the entire system can operate stably and effectively provide services to end users. Examples include front-end interaction design, back-end service deployment, assisted intelligent diagnosis, knowledge graph visualization, and problem management (such as database management).

[0141] Front-end Interaction Design: The front-end module provides users with an intuitive and easy-to-use human-computer interaction interface. Typically, a dual-end design is employed, encompassing both a web page and a mobile app. The interface features a simple and clear design, including elements such as a user input field, sample questions, a search button, and a results display area. The front-end utilizes modern technologies (such as HTML5, CSS3, JavaScript, and frameworks like Vue and React) to achieve a responsive layout, compatible with various devices. After the user enters a question, the front-end sends a request to the back-end via asynchronous communication methods such as AJAX or WebSocket. Preliminary validation of the user input (such as filtering sensitive words) can also be performed on the interface side. When the back-end returns an answer, the front-end renders the answer and displays it to the user, offering rich text or graphic presentation (such as highlighting entities in the knowledge graph or displaying fragments of the relationship graph) to enhance readability and user trust. The front-end also includes a logging module to track user behavior and feedback, providing data support for subsequent model optimization. The front-end design utilizes a component-based architecture, facilitating the expansion of new features (such as multi-round question-and-answer support and voice input) while ensuring system maintainability.

[0142] Backend Service Deployment: The backend hosts the logic layer functional modules and provides computing services for the frontend and model layers. The backend typically utilizes a distributed architecture, deploying various functional modules through microservices or modular services, such as intent recognition, question-and-answer parsing, and entity linking. Each service communicates with other modules via RESTful APIs and can be independently upgraded. The backend is implemented using mature web frameworks such as Django, Flask, or Spring Boot, enabling rapid development and reliable operation. The system is deployed on cloud servers or private data centers and managed using containerization technologies such as Docker and container orchestration tools such as Kubernetes, supporting autoscaling and load balancing. The load balancer distributes user requests across multiple backend nodes, ensuring smooth responsiveness even when traffic surges. During backend deployment, monitoring and logging systems (such as Prometheus and ELK) are also configured to monitor the health and performance of each service. Any anomalies can trigger automatic alerts or initiate emergency response plans. The backend deployment design fully considers security, with access control and encryption protocols (HTTPS) protecting user data. Modular services make the backend highly scalable and maintainable, and services or resources can be dynamically added or subtracted when business needs change.

[0143] Database management: The business layer maintains the underlying storage system, including the knowledge graph database and auxiliary databases. For the Neo4j graph database, regular backup, compression, and migration tests should be performed to ensure data security and consistency. In addition, Neo4j should be configured with cluster deployment or read-write separation mode to improve availability and concurrent query performance. Relational databases (such as MySQL, PostgreSQL) or NoSQL databases (such as Redis) can be used to store user information, question and answer logs, session data, etc. System administrators monitor these data tables or key-value stores through the database management module, including connection pool configuration, index optimization, and scheduled maintenance to ensure stable and efficient data access. For hot data and repeated query results, the system can also retain the latest results in the memory cache to reduce database load and speed up response. The database management layer is designed with an emphasis on scalability: when the amount of data or access volume continues to grow, storage capacity can be seamlessly expanded through horizontal table / partitioning, database cluster expansion, and other methods.

[0144] See Figure 11 , Figure 11 An electronic device provided in one embodiment of the present invention includes a processor 101, a memory 102 and a communication bus; the communication bus is used to connect the processor 101 and the memory 102; the processor 101 is used to execute the computer program stored in the memory 102 to implement the above-mentioned auxiliary diagnosis and treatment question and answer method.

[0145] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and the computer program is used to enable a computer to execute the above-mentioned auxiliary diagnosis and treatment question-and-answer method.

[0146] In general, the present invention achieves remarkable results in solving key problems in the field of Chinese medical text, such as low entity recognition accuracy, difficulty in identifying user natural language intent, multi-source data redundancy affecting query efficiency, and insufficient system scalability, by adopting advanced entity recognition and intent classification technologies, combined with multi-source data fusion and de-redundancy processing. By adopting a joint named entity recognition module based on the RoBERTa pre-trained language model, a bidirectional long short-term memory network, a multi-head self-attention mechanism, and a conditional random field, the accuracy of entity recognition in Chinese medical text is significantly improved. By integrating the user intent recognition module of BERT and TextCNN, the semantics of medical questions input by users are accurately parsed. Entity alignment and redundancy are performed through the entity alignment and fusion module, and semantically similar entities from different sources are merged into the same node, significantly eliminating data redundancy in the knowledge graph and improving query efficiency. At the same time, the modular and interface-based design supports dynamic updating of the medical knowledge base and functional modules, enhancing the scalability of the system when new knowledge and needs emerge.

[0147] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A knowledge graph-based auxiliary diagnosis and treatment question answering method, characterized in that: include: Get the original question and picture entered by the user; The original question and the image are processed using the CLIP model and then fused to obtain a fusion feature; Processing the fused features using a decoder to obtain a segmentation mask to find areas related to the original question; According to a preset keyword library, the segmentation mask is processed using a multi-head attention mechanism to filter out the most relevant keywords in the area related to the original question; The image and the keyword are processed and spliced ​​using the CLIP model to obtain a prompt vector; Processing the prompt vector using an LLM model to obtain a description text, and extracting structured information from the description text; generating a medical question sentence integrated with image information according to the original question sentence and the structured information; According to the medical question, the answer content is obtained from the knowledge graph.

2. The auxiliary diagnosis and treatment question-answering method based on knowledge graph according to claim 1 is characterized in that: The CLIP model is used to process the original question and the image and then fuse them to obtain fusion features, including: The CLIP model is used to process the image and the original question respectively to obtain image features and text features; After adding the image features and the text features element by element, a layer normalization process is performed; The image features and the text features are spliced ​​and then input into a multi-layer perceptron for nonlinear transformation; The features after layer normalization processing and the features output by the multi-layer perceptron are added to obtain the fused features.

3. The auxiliary diagnosis and treatment question-answering method based on knowledge graph according to claim 1 is characterized in that: The decoder is used to process the fused features to obtain a segmentation mask, including: Processing the fused features using multiple convolutional layers to obtain multiple features of different scales; Performing upsampling on the features of different scales to restore the features of different scales to the size of the fused feature; After adding multiple upsampled features element by element, they are processed through a convolutional layer to obtain a single-channel mask feature; The single-channel mask feature is processed using a Sigmoid function to obtain the segmentation mask.

4. The auxiliary diagnosis and treatment question-answering method based on knowledge graph according to claim 1 is characterized in that: Based on the preset keyword library, the segmentation mask is processed using a multi-head attention mechanism to filter out the most relevant keywords in the area related to the original question, including: Utilizing multiple attention heads to respectively calculate the semantic similarity between the segmentation mask and each keyword in the keyword library; The outputs of the multiple attention heads are concatenated and integrated through linear transformation to obtain a matching matrix; The keywords are screened out according to the maximum value in the matching matrix.

5. The knowledge graph-based auxiliary diagnosis and treatment question-answering method according to claim 1, characterized in that: The prompt vector is processed using the LLM model to obtain a description text, and structured information is extracted from the description text, including: Convert the hint vector into a continuous vector representation through the embedding layer of the LLM model; Utilizing the multi-layer Transformer structure and autoregressive mechanism within the decoder of the LLM model, the continuous vector representation is processed to obtain the description text; The description text is decomposed into the structured information by natural language processing technology, and the structured information includes lesion location, morphological characteristics and clinical prompts.

6. The knowledge graph-based auxiliary diagnosis and treatment question-answering method according to claim 1, characterized in that: Based on the medical question, the answer content is obtained from the knowledge graph, including: Processing the medical question using a pre-trained first language model and a first neural network to obtain user intent; Processing the medical question to obtain a first entity, and linking the first entity to a node in a knowledge graph to obtain a standard entity name; According to the user's intention, a corresponding query template is selected from predefined sentence templates, and the standard entity name is filled into the query template to obtain a query statement; According to the query statement, the entities and their attributes are queried and returned from the knowledge graph, and the answer content is generated in combination with the user intention.

7. The knowledge graph-based auxiliary diagnosis and treatment question-answering method according to claim 6, characterized in that: The knowledge graph is constructed through the following steps: Access to medical texts; Using a named entity recognition module, the medical text is processed to obtain an optimal entity label sequence corresponding to the medical text; Splitting the optimal entity tag sequence according to domain rule-based attribute mapping and syntax template to obtain entities and their corresponding attributes; Using the relationship extraction module, the semantic relationships between all entities are extracted to build relationships between entities; Constructing the knowledge graph according to the entities and their corresponding attributes and the relationships between the entities; After splitting the optimal entity tag sequence according to the attribute mapping and syntax template based on the domain rules to obtain the entity and its corresponding attribute, the method further includes: Use the pre-trained third BERT language model to calculate the semantic similarity between different entity names and descriptions, and perform entity fusion based on the similarity calculation results; The knowledge graph is constructed based on the fused entities.

8. A knowledge graph-based auxiliary diagnosis and treatment question-answering system, characterized in that: include: The request acquisition module is used to obtain the original question and image input by the user; An image and text embedding fusion module is used to process the original question and the image using the CLIP model and then fuse them to obtain a fusion feature; A fine-grained decoding and segmentation module is used to process the fused features using a decoder to obtain a segmentation mask to find the area related to the original question; A region and keyword matching module is used to process the segmentation mask using a multi-head attention mechanism based on a preset keyword library to filter out the most relevant keywords for the region related to the original question; a prompt vector calculation module, configured to process the original question and the keyword using a CLIP model and then concatenate them to obtain a prompt vector; A fine-grained description generation module is used to process the prompt vector using the LLM model to obtain a description text and extract structured information from the description text; A medical question generation module, configured to generate a medical question integrated with image information based on the original question and the structured information; The question and answer generation module is used to obtain the answer content from the knowledge graph based on the medical question.

9. An electronic device, characterized in that: The system comprises a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is used to enable a computer to execute the method according to any one of claims 1 to 7.

Images