Medical large model reinforcement learning alignment method based on knowledge graph fact consistency

Abstract

The application relates to the field of artificial intelligence and medical health technology, and in particular to a medical large model reinforcement learning alignment method based on knowledge graph fact consistency. The method obtains a medical pre-training model and a structured knowledge graph, and obtains an initial alignment model through supervised fine-tuning. Then, human preference scores and fact verification results are generated by combining model output and the knowledge graph, a composite reward model containing the two is constructed, and a target reward function is formed by weighted combination to balance preference and fact accuracy. Finally, the initial model is trained by reinforcement learning using the function to obtain a final alignment model. The application improves the professional accuracy and fact consistency of the output content of the medical large model.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and healthcare technology, and in particular to a reinforcement learning alignment method for large medical models based on knowledge graph fact consistency. Background Technology

[0002] In the fields of artificial intelligence and healthcare, large-scale language models are increasingly being used, particularly demonstrating their potential in scenarios such as assisted diagnosis, patient consultation, and medical knowledge question answering. Existing technologies typically employ reinforcement learning methods based on human feedback to align pre-trained models, optimizing their output to align with human preferences. The common practice involves collecting quality scores from human experts on the model's generated answers, assessing their safety, usefulness, and harmlessness, and training a reward model based on these scores. Subsequently, this reward model is used to further optimize the initially fine-tuned language model through reinforcement learning algorithms, such as proximal policy optimization, aiming to make the model generate responses that better align with human values ​​and expectations.

[0003] However, these conventional alignment methods primarily focus on human preferences for the overall quality of responses, which has significant limitations. The medical field demands extremely high accuracy of information; model outputs must align with authoritative medical knowledge. Existing methods lack direct, structured constraints on the factual correctness of generated content. Models may generate seemingly fluent, harmless responses that conform to general human preferences, but the medical facts they contain, such as drug interactions, descriptions of disease symptoms, or treatment plans, may contradict actual medical consensus. Such factual errors can have serious consequences in a medical context. Although human raters consider accuracy during evaluation, this indirect feedback mechanism, relying on subjective judgment, is neither precise nor reliable enough to systematically ensure that model outputs align with the vast and dynamically changing medical knowledge system. Therefore, effectively integrating and strengthening hard constraints on factual consistency in the large-scale model alignment process has become a pressing technical problem. Summary of the Invention

[0004] This invention provides a reinforcement learning alignment method for large medical models based on knowledge graph fact consistency, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides a reinforcement learning alignment method for large medical models based on knowledge graph fact consistency, comprising:

[0006] Acquire pre-trained language models in the medical field, as well as structured knowledge graphs containing medical entities and relationships between them;

[0007] A training dataset is determined, and the pre-trained language model is supervised and fine-tuned using the training dataset to obtain an initial alignment model;

[0008] Based on the initial alignment model and the structured knowledge graph, human expert preference scores and fact consistency verification results are generated. Based on the human expert preference scores and fact consistency verification results, a composite reward model is generated, which consists of a first reward component and a second reward component.

[0009] The target reward function is obtained by weighting the first reward component and the second reward component, wherein the weight coefficient of the second reward component indicates the proportion of factual accuracy in the overall reward;

[0010] The initial alignment model is trained using the target reward function to obtain the final aligned large medical model.

[0011] Determine the training dataset, and use the training dataset to perform supervised fine-tuning of the pre-trained language model to obtain an initial alignment model, including:

[0012] Acquire question-answer pair data in the medical field, and extract the question text and answer text from each question-answer pair;

[0013] Medical entity recognition is performed on the question text to extract medical entities from the question text as query entities, and the entity nodes corresponding to the query entities are located in the structured knowledge graph.

[0014] Starting from the entity node, perform a multi-hop path search in the structured knowledge graph to obtain a path segment from the query entity to the target entity;

[0015] The path segment is linearized to obtain linearized path segment text. The question text, the linearized path segment text, and the answer text are then concatenated according to a preset template to generate a single training sample.

[0016] For each question-answer pair, corresponding training samples are generated to obtain the training dataset.

[0017] The training dataset is input into the pre-trained language model for parameter updates, enabling the pre-trained language model to learn the mapping relationship for generating the answer text given the question text and the path fragment text, thus obtaining the initial alignment model.

[0018] Based on the initial alignment model and the structured knowledge graph, human expert preference scores and fact consistency verification results are generated. Based on the human expert preference scores and the fact consistency verification results, a composite reward model is generated, including:

[0019] Multiple input questions are sampled from the prompt dataset, and the input questions are input into the initial alignment model, which then generates multiple candidate answers for each input question.

[0020] The candidate answers are evaluated for quality and scored to obtain the human expert preference score;

[0021] A medical entity extraction operation is performed on the candidate answers to identify the medical entities appearing in the candidate answers. These medical entities are then used as query objects for retrieval in the structured knowledge graph.

[0022] In the structured knowledge graph, locate the entity node corresponding to the medical entity, and extract the triplet of relationships between the medical entities as facts of the knowledge graph;

[0023] Text fragments representing relationships between entities are extracted from the candidate answers. Semantic comparison is performed between the entity relationships represented in the text fragments and the facts in the knowledge graph to obtain the fact consistency verification result.

[0024] The human expert preference score is converted into the first reward component, and the factual consistency verification result is converted into the second reward component. The composite reward model is then determined based on the first reward component and the second reward component.

[0025] Text fragments representing relationships between entities are extracted from the candidate answers. A semantic comparison is performed between the entity relationships represented in the text fragments and the facts in the knowledge graph to obtain factual consistency verification results, including:

[0026] Syntactic analysis is performed on the candidate responses to identify sentence segments containing medical entities.

[0027] Determine the predicate structure representing the association between medical entities from the sentence fragment, and obtain the text fragment representing the relationship between entities based on the predicate structure;

[0028] The relation descriptors in the text fragment are converted into standardized relational semantic representations, and the relation types in the relation triples indicated by the facts in the knowledge graph are converted into corresponding relational semantic representations. The degree of consistency between the two relational semantic representations is obtained by calculating the similarity in the semantic vector space.

[0029] Based on the consistency degree value, the consistency relationship between the entity relationship expressed by the text fragment and the facts of the knowledge graph is determined to be contradictory, supportive, or irrelevant, thus obtaining the consistency relationship determination result;

[0030] Based on the consistency relationship determination result, the verification value of the text segment is obtained, and based on the verification values ​​of all text segments in the candidate answer, the factual consistency verification result of the candidate answer is obtained.

[0031] The target reward function is obtained by weighting and combining the first reward component and the second reward component, including:

[0032] During reinforcement learning training, response samples generated by the initial alignment model on the validation set are collected. These response samples are then compared with standard answers annotated by human experts, and the semantic similarity between the response samples and the standard answers is calculated to obtain the human preference alignment value.

[0033] The consistency of the medical entity relationship descriptions in the answer sample with the relationship triples in the structured knowledge graph is compared, and the accuracy of the knowledge fact is obtained by counting the proportion of the number of relationship descriptions in the answer sample that are consistent with the structured knowledge graph to the total number of relationship descriptions.

[0034] A first weight coefficient is generated based on the human preference alignment value, and a second weight coefficient is generated based on the knowledge fact accuracy.

[0035] The combined reward value is obtained based on the product of the first reward component and the first weight coefficient, and the product of the second reward component and the second weight coefficient;

[0036] Generate the target reward function, which receives candidate answers as input and outputs the combined reward value.

[0037] The initial alignment model is trained using reinforcement learning with the target reward function to obtain the final aligned large medical model, including:

[0038] Input questions are sampled from the training dataset and fed into the initial alignment model, which then generates corresponding answer text based on the current model parameters.

[0039] The answer text is input into the target reward function, which calculates the weighted value of the first reward component and the first weight coefficient, and the weighted value of the second reward component and the second weight coefficient based on the answer text, to obtain the reward signal corresponding to the answer text.

[0040] The policy gradient loss function is determined based on the reward signal, and the knowledge graph consistency constraint term is determined based on the consistency comparison result between the entity relation representation extracted from the answer text and the relation triples in the structured knowledge graph;

[0041] A comprehensive loss function is obtained based on the policy gradient loss function and the knowledge graph consistency constraint term;

[0042] The model parameters of the initial alignment model are updated based on the comprehensive loss function until the reward signal of the initial alignment model converges on the validation set, thereby obtaining the final aligned large medical model.

[0043] The policy gradient loss function is determined based on the reward signal, and the knowledge graph consistency constraint term is determined based on the consistency comparison result between the entity relation representation extracted from the answer text and the relation triples in the structured knowledge graph, including:

[0044] Calculate the generation probability of the initial alignment model generating the answer text, and numerically combine the generation probability with the reward signal to obtain the policy gradient term;

[0045] The policy gradient loss function is obtained based on the policy gradient terms of all response texts within the training batch;

[0046] Perform medical entity recognition on the response text to obtain pairs of medical entities appearing in the response text, and determine entity relationship representations from the response text based on the pairs of medical entities;

[0047] The entity relationship description is compared with the relationship triples in the structured knowledge graph to identify contradictory entity relationship descriptions and their corresponding semantic similarity deviation values.

[0048] For entity relationship statements that are determined to be contradictory, a contradiction penalty value is calculated, and the contradiction penalty value is proportional to the semantic similarity deviation value.

[0049] Based on the contradiction penalty value of all contradictory entity relation representations in the response text, the consistency constraint term of the knowledge graph is determined.

[0050] A second aspect of this invention provides a reinforcement learning alignment system for large medical models based on knowledge graph fact consistency, comprising:

[0051] The model acquisition unit is used to acquire pre-trained language models in the medical field, as well as structured knowledge graphs containing medical entities and relationships between entities.

[0052] A supervised fine-tuning unit is used to determine the training dataset and perform supervised fine-tuning on the pre-trained language model using the training dataset to obtain an initial alignment model.

[0053] A composite reward unit is used to generate human expert preference scores and fact consistency verification results based on the initial alignment model and the structured knowledge graph, and to generate a composite reward model based on the human expert preference scores and fact consistency verification results. The composite reward model consists of a first reward component and a second reward component.

[0054] The weighting combination unit is used to perform a weighted combination based on the first reward component and the second reward component to obtain the target reward function, wherein the weight coefficient of the second reward component indicates the proportion of factual accuracy in the overall reward;

[0055] The reinforcement learning unit is used to train the initial alignment model using the target reward function to obtain the final aligned large medical model.

[0056] A third aspect of the present invention provides an electronic device, comprising:

[0057] processor;

[0058] Memory used to store processor-executable instructions;

[0059] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0060] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0061] This invention significantly improves the professional accuracy and factual reliability of content generated by large-scale medical models. By introducing a structured knowledge graph as an external authoritative knowledge source, it provides the model with a solid foundation of medical facts, effectively compensating for the knowledge blind spots or memory biases inherent in single-language models within specific professional domains. Utilizing the knowledge graph to perform factual consistency checks on the model's output can accurately identify and quantify factual fallacies such as entity relationship errors and attribute contradictions in the responses, providing clear and calculable feedback signals on factual accuracy for reinforcement learning.

[0062] This invention combines human expert preference scoring with automated fact-consistency checks to construct a composite reward model, enabling multi-dimensional and refined evaluation of the model's output quality. Human preference scoring ensures the model's overall quality in terms of fluency, usefulness, and ethical compliance, while fact-consistency checks focus on the accuracy of underlying medical knowledge. By weighting these two reward components and allowing adjustment of the proportion of factual accuracy in the overall reward, this method can flexibly seek the optimal balance between the model's "usefulness" and "accuracy" according to the needs of practical application scenarios, with particular emphasis on the high factual accuracy required in the medical field.

[0063] This invention employs a reinforcement learning strategy based on a composite reward model to further align the initial model, driving it to actively learn and generate responses that align with both human expert preferences and strict adherence to medical facts. This process optimizes the model's internal parameters, gradually internalizing structured relationships within the knowledge graph. This allows the model to spontaneously generate high-quality, fact-consistent content without frequent retrieval of external knowledge bases. The resulting large-scale medical model not only uses natural and fluent language but also possesses high credibility and reliability in its representation of medical facts, reducing the potential for medical risks due to inaccurate information. It provides safe and reliable technical support for serious applications such as medical consultation, assisted diagnosis, and patient education. Attached Figure Description

[0064] Figure 1 This is a flowchart illustrating the reinforcement learning alignment method for a large medical model based on knowledge graph fact consistency, according to an embodiment of the present invention.

[0065] Figure 2 This is a flowchart illustrating the process of determining the target reward function according to an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0068] Figure 1This is a flowchart illustrating the reinforcement learning alignment method for a large medical model based on knowledge graph fact consistency, according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0069] Acquire pre-trained language models in the medical field, as well as structured knowledge graphs containing medical entities and relationships between them;

[0070] A training dataset is determined, and the pre-trained language model is supervised and fine-tuned using the training dataset to obtain an initial alignment model;

[0071] Based on the initial alignment model and the structured knowledge graph, human expert preference scores and fact consistency verification results are generated. Based on the human expert preference scores and fact consistency verification results, a composite reward model is generated, which consists of a first reward component and a second reward component.

[0072] The target reward function is obtained by weighting the first reward component and the second reward component, wherein the weight coefficient of the second reward component indicates the proportion of factual accuracy in the overall reward;

[0073] The initial alignment model is trained using the target reward function to obtain the final aligned large medical model.

[0074] Determine the training dataset, and use the training dataset to perform supervised fine-tuning of the pre-trained language model to obtain an initial alignment model, including:

[0075] Acquire question-answer pair data in the medical field, and extract the question text and answer text from each question-answer pair;

[0076] Medical entity recognition is performed on the question text to extract medical entities from the question text as query entities, and the entity nodes corresponding to the query entities are located in the structured knowledge graph.

[0077] Starting from the entity node, perform a multi-hop path search in the structured knowledge graph to obtain a path segment from the query entity to the target entity;

[0078] The path segment is linearized to obtain linearized path segment text. The question text, the linearized path segment text, and the answer text are then concatenated according to a preset template to generate a single training sample.

[0079] For each question-answer pair, corresponding training samples are generated to obtain the training dataset.

[0080] The training dataset is input into the pre-trained language model for parameter updates, enabling the pre-trained language model to learn the mapping relationship for generating the answer text given the question text and the path fragment text, thus obtaining the initial alignment model.

[0081] After acquiring a pre-trained language model in the medical field, a training dataset containing knowledge graph augmentation information needs to be constructed. This involves collecting publicly available medical question-and-answer corpora and extracting question-and-answer pairs. Each pair contains a patient's description of symptoms or a disease consultation question, along with a doctor's diagnostic suggestions or treatment plan. The question text is preprocessed to remove irrelevant punctuation and stop words. Then, a BiLSTM-CRF-based named entity recognition model is used to identify medical entities, including disease names, symptom descriptions, drug names, or examination items. The identified entities are used as query entry points to retrieve corresponding nodes in the constructed medical knowledge graph. The knowledge graph is stored in triplet form, where nodes represent medical concepts and edges represent semantic relationships between concepts.

[0082] Starting from the query entity node, a breadth-first search strategy is used to expand the knowledge graph, with a search depth of 2 to 3 hops. For each hop, 3 to 5 relation edges with the highest confidence are selected, forming a path sequence from the query entity to the relevant target entity. For example, when the query entity is "diabetes," the path "diabetes-symptoms-polydipsia" or "diabetes-treatment drugs-metformin" can be obtained. The obtained path fragments are linearized, converting the graph structure into a text sequence. Each edge is expressed using the template "Entity A is connected to entity B through relation R," and multiple paths are connected by separators.

[0083] The question text, linearized knowledge path, and answer text are concatenated according to the template "Question:[Question Content] Knowledge:[Path Text] Answer:[Answer Content]" to form a single training sample. This process is repeated for all question-answer pairs in the dataset, resulting in a training set containing 50,000 to 100,000 samples. The AdamW optimizer is used to fine-tune the pre-trained model, with a learning rate set to... The batch size was 16, and the training rounds were 3 to 5. During training, the model learned to generate answers that conformed to medical facts based on the question and knowledge graph paths, minimizing the difference between the predicted and actual answers using a cross-entropy loss function. The fine-tuned model parameters were initially aligned with medical knowledge, enabling the generation of answers containing knowledge graph support information, serving as the initial alignment model for subsequent reinforcement learning stages. The entire supervised fine-tuning process enabled the model to establish an end-to-end mapping capability between question understanding, knowledge retrieval, and answer generation, laying the foundation for introducing a composite reward mechanism.

[0084] Based on the initial alignment model and the structured knowledge graph, human expert preference scores and fact consistency verification results are generated. Based on the human expert preference scores and the fact consistency verification results, a composite reward model is generated, including:

[0085] Multiple input questions are sampled from the prompt dataset, and the input questions are input into the initial alignment model, which then generates multiple candidate answers for each input question.

[0086] The candidate answers are evaluated for quality and scored to obtain the human expert preference score;

[0087] A medical entity extraction operation is performed on the candidate answers to identify the medical entities appearing in the candidate answers. These medical entities are then used as query objects for retrieval in the structured knowledge graph.

[0088] In the structured knowledge graph, locate the entity node corresponding to the medical entity, and extract the triplet of relationships between the medical entities as facts of the knowledge graph;

[0089] Text fragments representing relationships between entities are extracted from the candidate answers. Semantic comparison is performed between the entity relationships represented in the text fragments and the facts in the knowledge graph to obtain the fact consistency verification result.

[0090] The human expert preference score is converted into the first reward component, and the factual consistency verification result is converted into the second reward component. The composite reward model is then determined based on the first reward component and the second reward component.

[0091] Several input questions are randomly sampled from a pre-prepared dataset of prompts, covering medical scenarios such as disease diagnosis, symptom analysis, and drug consultation. These sampled input questions are then fed into an initial alignment model. For each input question, generation parameters such as the temperature coefficient and sampling strategy are adjusted to generate 3 to 5 semantically similar but differently worded candidate answers for the same question. In the quality assessment phase, licensed medical experts are invited to score the candidate answers. The scoring dimensions include medical accuracy, fluency, and practicality, with each dimension using a scoring range of 0 to 10. The scores from the three dimensions are weighted and summed according to preset weighting coefficients to obtain the human expert preference score for each candidate answer.

[0092] The medical entity extraction operation employs named entity recognition (NID) technology. Using a sequence labeling model trained on a medical corpus, it scans the text of candidate responses character by character to identify medical entities such as disease names, drug names, symptom descriptions, and examination items, and marks their start and end positions in the text. The extracted medical entities are used as query keywords to retrieve the corresponding entity node identifiers from the index table of a structured knowledge graph. The knowledge graph is stored in triples, with each triple containing a head entity, a relation type, and a tail entity. After locating the entity node corresponding to a medical entity, all outgoing and incoming edges of that node are traversed to extract relation triples related to that entity, such as "hypertension - treatment drugs - nifedipine" or "diabetes - common symptoms - polydipsia and polyuria".

[0093] During the extraction of relational text fragments, dependency parsing techniques are employed to analyze the syntactic structure of candidate answers, identifying verb phrases, prepositional phrases, and noun phrases expressing relationships between entities. Through subject-verb-object and attributive-head relationships in the syntactic tree, the two medical entities involved in the text fragment and their relational descriptions are determined. When semantically comparing the extracted relational text fragments with facts in the knowledge graph, the text fragments are first semantically encoded to obtain vector representations. Simultaneously, the relational triples in the knowledge graph are converted into natural language descriptions and encoded as vectors. The cosine similarity between the text fragment vector and the knowledge graph relation vector is calculated. When the similarity exceeds a preset threshold of 0.85, the relational statement is considered consistent with the knowledge graph fact and assigned a value of 1; when the similarity is below the threshold, it is considered inconsistent and assigned a value of 0. The average of the consistency judgment results for all relational text fragments in the candidate answer is calculated to obtain the factual consistency verification result of the candidate answer.

[0094] The human expert preference ratings are normalized and mapped to the 0-1 range, serving as the first reward component. The results of the factual consistency verification will be directly used as the second reward component. The compound reward model adopts a weighted linear combination form, expressed as follows: ,in and For the weighting coefficients, satisfying By adjusting The value controls the proportion of factual accuracy in the overall reward, with a typical range of 0.3 to 0.5.

[0095] Text fragments representing relationships between entities are extracted from the candidate answers. A semantic comparison is performed between the entity relationships represented in the text fragments and the facts in the knowledge graph to obtain factual consistency verification results, including:

[0096] Syntactic analysis is performed on the candidate responses to identify sentence segments containing medical entities.

[0097] Determine the predicate structure representing the association between medical entities from the sentence fragment, and obtain the text fragment representing the relationship between entities based on the predicate structure;

[0098] The relation descriptors in the text fragment are converted into standardized relational semantic representations, and the relation types in the relation triples indicated by the facts in the knowledge graph are converted into corresponding relational semantic representations. The degree of consistency between the two relational semantic representations is obtained by calculating the similarity in the semantic vector space.

[0099] Based on the consistency degree value, the consistency relationship between the entity relationship expressed by the text fragment and the facts of the knowledge graph is determined to be contradictory, supportive, or irrelevant, thus obtaining the consistency relationship determination result;

[0100] Based on the consistency relationship determination result, the verification value of the text segment is obtained, and based on the verification values ​​of all text segments in the candidate answer, the factual consistency verification result of the candidate answer is obtained.

[0101] For the text content generated from candidate answers, syntactic dependency analysis is performed using natural language processing techniques. This analysis process employs a dependency parser to identify the subject-verb-object structure, modification relations, and clause structure of sentences, with a focus on marking sentence components containing medical entities. Specifically, a named entity recognition model scans each sentence in the candidate answers. When medical entities such as disease names, drug names, symptom descriptions, and treatment methods are detected, the complete sentence or clause containing these entities is extracted as the sentence fragment to be analyzed. For example, when the candidate answer contains "diabetic patients should take metformin to control blood sugar," this complete sentence is extracted as a sentence fragment.

[0102] Within a given sentence fragment, a dependency parsing tree is used to identify predicate components connecting two or more medical entities. These predicate structures typically center around verbs or nouns indicating relationships, such as keywords like "treat," "cause," "alleviate," and "apply to." The parsing process traces the semantic chain from the subject entity to the object entity along the dependency path, extracting triple structures in the form of "entity A - relational term - entity B." When the sentence fragment is "Aspirin can prevent cardiovascular disease," the extracted predicate structure is "aspirin" (drug entity), "prevention" (relational predicate), and "cardiovascular disease" (disease entity), thus constructing a text fragment representing the relationship between entities.

[0103] The extracted relation descriptors from the text fragments are mapped to a standardized semantic space. These descriptors are then converted into fixed-dimensional vector representations using a pre-trained semantic encoder trained on a medical corpus through contrastive learning. Simultaneously, predefined relation types in the knowledge graph, such as treats, causes, and contraindications, are also converted into corresponding relation semantic vectors using the same encoder. In the semantic vector space, the cosine similarity formula is used to measure the angle between two relation vectors, yielding a similarity score ranging from 0 to 1 as a consistency value. A similarity score higher than 0.8 indicates a high degree of consistency between the relation described in the text fragment and the relation type in the knowledge graph.

[0104] A threshold is set based on the calculated consistency value. When the consistency value exceeds a preset high threshold (e.g., 0.75), it is determined to be a supporting relationship, indicating that the text fragment matches the facts in the knowledge graph. When the consistency value is lower than a preset low threshold (e.g., 0.3) and there is a contradictory statement between the text fragment and the knowledge graph, it is determined to be a contradictory relationship. When the consistency value is between the two thresholds or the relationship is not covered in the knowledge graph, it is determined to be an irrelevant relationship. This determination process also examines the matching of entity pairs. Relationship-level consistency determination is only performed when two entities in the text fragment correspond to the first and last entities in the knowledge graph triple.

[0105] For each text segment, a verification value is assigned based on the consistency relationship determination result. Text segments supporting the relationship receive a positive verification value of 1, contradictory relationships receive a negative verification value of -1, and irrelevant relationships receive a neutral verification value of 0. The overall factual consistency verification result of the candidate answer is calculated by aggregating the verification values ​​of all text segments, using a weighted average or normalized summation method. The final output is a continuous score ranging from -1 to 1, which quantitatively represents the overall consistency between the candidate answer and the facts in the knowledge graph.

[0106] Figure 2 This is a schematic diagram illustrating the process of determining the target reward function according to an embodiment of the present invention. Figure 2 As shown, the target reward function is obtained by weighting and combining the first reward component and the second reward component, including:

[0107] During reinforcement learning training, response samples generated by the initial alignment model on the validation set are collected. These response samples are then compared with standard answers annotated by human experts, and the semantic similarity between the response samples and the standard answers is calculated to obtain the human preference alignment value.

[0108] The consistency of the medical entity relationship descriptions in the answer sample with the relationship triples in the structured knowledge graph is compared, and the accuracy of the knowledge fact is obtained by counting the proportion of the number of relationship descriptions in the answer sample that are consistent with the structured knowledge graph to the total number of relationship descriptions.

[0109] A first weight coefficient is generated based on the human preference alignment value, and a second weight coefficient is generated based on the knowledge fact accuracy.

[0110] The combined reward value is obtained based on the product of the first reward component and the first weight coefficient, and the product of the second reward component and the second weight coefficient;

[0111] Generate the target reward function, which receives candidate answers as input and outputs the combined reward value.

[0112] In constructing the target reward function, evaluation data is first extracted from the actual performance of the initial alignment model on the validation set. Medical questions from the validation set are input into the initial alignment model one by one, and sample responses generated by the model are collected. These sample responses need to be paired and compared with standard answers pre-annotated by clinical medical experts. Semantic similarity is calculated using sentence vector embedding technology, encoding both the sample responses and the standard answers into high-dimensional vector representations. The degree of semantic closeness is quantified by calculating the cosine similarity between the two vectors. The cosine similarity value ranges from -1 to 1, with values ​​closer to 1 indicating greater semantic similarity. This similarity value is used as the alignment value for human preferences, reflecting the degree of conformity between the model output and the expectations of human experts.

[0113] To assess the accuracy of factual information, medical entity relations need to be extracted from the answer samples. Named entity recognition technology is used to identify medical entities such as disease names, drug names, and symptom descriptions in the answers. Then, relation extraction algorithms are used to identify semantic relationships between entities, such as "disease-treatment-drug" or "symptom-indication-disease" patterns. The extracted relation expressions are converted into triples and matched item by item with standard triples stored in the structured knowledge graph. The matching process considers the synonym mapping of entities and the equivalence of relation types. The ratio of the total number of relation expressions in the answer samples to the number of relation expressions that are completely consistent with the knowledge graph is the accuracy of the factual information. The value of this indicator ranges from 0 to 1; a higher value indicates that the medical knowledge output by the model is more consistent with the authoritative knowledge base.

[0114] The weighting coefficients are generated using an adaptive adjustment mechanism. The first weighting coefficient is set dynamically based on the alignment value of human preferences. When the alignment value is low, this weight is increased to strengthen the learning of human preferences; when the alignment value reaches a high level, this weight is gradually decreased. The second weighting coefficient is inversely related to the accuracy of the knowledge facts. When the accuracy is insufficient, this weight is significantly increased to strengthen the constraint of factual consistency; when the accuracy tends to stabilize, this weight is moderately decreased to avoid overfitting the knowledge graph. The sum of the two weighting coefficients is always kept at 1 to ensure the stability of the overall reward value range.

[0115] The combined reward value is calculated using linear weighting. The first reward component is multiplied by the first weight coefficient to obtain the human preference contribution, and the second reward component is multiplied by the second weight coefficient to obtain the factual accuracy contribution. The two contributions are then added together to obtain the final combined reward value. This value serves as an immediate feedback signal during reinforcement learning training, guiding the gradient update direction of the model parameters.

[0116] The target reward function is defined as a mapping relationship that takes any candidate answer text as input. Internally, it executes the complete evaluation process described above, including semantic similarity calculation, relation extraction and matching, weight coefficient determination, and weighted summation, ultimately outputting the combined reward value corresponding to the candidate answer. This function is repeatedly called in each training step of reinforcement learning, providing a continuous reward signal for policy optimization.

[0117] The initial alignment model is trained using reinforcement learning with the target reward function to obtain the final aligned large medical model, including:

[0118] Input questions are sampled from the training dataset and fed into the initial alignment model, which then generates corresponding answer text based on the current model parameters.

[0119] The answer text is input into the target reward function, which calculates the weighted value of the first reward component and the first weight coefficient, and the weighted value of the second reward component and the second weight coefficient based on the answer text, to obtain the reward signal corresponding to the answer text.

[0120] The policy gradient loss function is determined based on the reward signal, and the knowledge graph consistency constraint term is determined based on the consistency comparison result between the entity relation representation extracted from the answer text and the relation triples in the structured knowledge graph;

[0121] A comprehensive loss function is obtained based on the policy gradient loss function and the knowledge graph consistency constraint term;

[0122] The model parameters of the initial alignment model are updated based on the comprehensive loss function until the reward signal of the initial alignment model converges on the validation set, thereby obtaining the final aligned large medical model.

[0123] In each iteration of reinforcement learning, a number of input questions in the medical field are randomly selected from a pre-prepared training dataset as sample batches. These input questions cover various medical scenarios such as disease diagnosis consultation, medication guidance query, and symptom analysis request. Each input question is fed into the initial alignment model, which performs forward inference based on the currently stored parameter configuration and outputs a complete answer text sequence through word-by-word generation autoregressive method.

[0124] The generated response text is then fed into the target reward function module for scoring. This module first calls the first reward component calculator, which scores the fluency, professionalism, and human preference relevance of the response text based on a pre-trained preference scoring network, thus obtaining a preference score. Simultaneously, a second reward component calculator is activated. This calculator compares the answer text with the structured knowledge graph: it uses a named entity recognition tool to extract medical entities from the answer text, and then uses a relation extraction model to identify the relational expressions between entities, forming a set of relation triples to be verified; these triples are then matched one by one with the standard triples stored in the knowledge graph, and the proportion of successfully matched triples is counted to obtain the fact consistency score. The target reward function calculates a weighted sum according to preset weight coefficients, specifically as follows: ,in The value of is usually set to a high value to emphasize the importance of factual accuracy, and the final output is a scalar reward signal R.

[0125] After obtaining the reward signal, a policy gradient loss function is constructed to guide model parameter optimization. A near-end policy optimization algorithm framework is employed, calculating the ratio of the log probability of the current policy generating the response text to the log probability of the old policy. This ratio is then multiplied by the reward signal and pruned to limit the update magnitude, forming the policy gradient loss term. To further ensure the model output aligns with medical knowledge, a knowledge graph consistency constraint is introduced: for each relation triple extracted from the response text, a query is performed in the knowledge graph to determine if a corresponding standard triple exists; if not, a penalty is applied; the penalty values ​​for all inconsistent triples are summed and then multiplied by an adjustment coefficient. , obtain constraint terms The policy gradient loss term is added to the knowledge graph consistency constraint term to form the comprehensive loss function. .

[0126] The gradient of the comprehensive loss function with respect to the model parameters is calculated using the backpropagation algorithm. An adaptive learning rate optimizer is used to update all trainable parameters of the initially aligned model. After each fixed number of training iterations, the model performance is evaluated on an independent validation set: questions from the validation set are sampled and input into the model to generate answers. The average reward signal value of these answers under the target reward function is calculated. The trend of this average reward value is continuously monitored. When the fluctuation of this value is less than a set threshold for several consecutive evaluation periods, the training process is considered to have converged, and the model parameters have reached their optimal state. The current model is then saved as the final aligned medical model.

[0127] The policy gradient loss function is determined based on the reward signal, and the knowledge graph consistency constraint term is determined based on the consistency comparison result between the entity relation representation extracted from the answer text and the relation triples in the structured knowledge graph, including:

[0128] Calculate the generation probability of the initial alignment model generating the answer text, and numerically combine the generation probability with the reward signal to obtain the policy gradient term;

[0129] The policy gradient loss function is obtained based on the policy gradient terms of all response texts within the training batch;

[0130] Perform medical entity recognition on the response text to obtain pairs of medical entities appearing in the response text, and determine entity relationship representations from the response text based on the pairs of medical entities;

[0131] The entity relationship description is compared with the relationship triples in the structured knowledge graph to identify contradictory entity relationship descriptions and their corresponding semantic similarity deviation values.

[0132] For entity relationship statements that are determined to be contradictory, a contradiction penalty value is calculated, and the contradiction penalty value is proportional to the semantic similarity deviation value.

[0133] Based on the contradiction penalty value of all contradictory entity relation representations in the response text, the consistency constraint term of the knowledge graph is determined.

[0134] In reinforcement learning training, constructing the policy gradient loss function first requires obtaining the probability distribution of the generated response text from the initial alignment model. For the input query text and the sequence of response texts generated by the model, the conditional probability of each generated token is calculated, and the log probability of the entire sequence is accumulated to obtain the generation probability value of the response text. This generation probability value is then multiplied numerically with the reward signal obtained from the composite reward model to form the policy gradient term for a single sample. Within a training batch, the average of the policy gradient terms for all samples is calculated and then negative to obtain the policy gradient loss function for that batch, which guides the gradient update direction of the model parameters.

[0135] The construction of consistency constraints in the knowledge graph requires deep semantic parsing of the generated response text. First, a medical entity recognition tool is applied to analyze the response text. This tool, based on a pre-trained medical named entity recognition model, can identify medical entities such as disease names, symptom descriptions, drug names, and treatment plans in the text. For the identified medical entities, a sliding window strategy is used to extract entity pairs that co-occur in the text. The window size is set to 50 characters to ensure that semantically related entity combinations are captured.

[0136] For each extracted medical entity pair, dependency parsing is used to determine the relationship between the two entities. Specifically, a dependency parser is used to identify the syntactic path between entities, extracting predicate phrases, verb phrases, or relational indicator words that connect the two entities, forming a triplet form of "entity 1 - relational description - entity 2". For example, from "diabetic patients should take metformin", the relational expression "diabetes - should take - metformin" is extracted.

[0137] The extracted relation representations are compared for consistency with the standard relation triples stored in the structured knowledge graph. First, all triples containing the same entity pair are retrieved from the knowledge graph; if no corresponding triple exists, the comparison is skipped. For cases where a triple exists, a pre-trained semantic similarity model is used to calculate the cosine similarity between the extracted relation representation and the semantic vectors of the relation descriptions in the knowledge graph. A similarity threshold of 0.7 is set; a similarity below this threshold is considered a contradiction, and a semantic similarity deviation value is calculated, defined as 1 minus the actual similarity value.

[0138] For entity relationship statements identified as contradictory, an exponential penalty mechanism is used to calculate the contradiction penalty value. The contradiction penalty value is equal to the base penalty coefficient multiplied by the square of the semantic similarity deviation value, with the base penalty coefficient set to 5.0. This design ensures that contradictions with greater semantic deviations are penalized more severely. For all contradictory relationship statements identified in the response text, their contradiction penalty values ​​are accumulated and normalized by dividing by the total number of words in the response text to obtain the knowledge graph consistency constraint term for that sample. During batch training, the average of the constraint term for all samples is calculated and linearly combined with the policy gradient loss function. The weight coefficient of the constraint term is set to 0.3 to ensure that the model strictly adheres to the factual relationships in the medical knowledge graph while optimizing the generation quality.

[0139] The medical large-scale model reinforcement learning alignment system based on knowledge graph fact consistency according to embodiments of the present invention includes:

[0140] The model acquisition unit is used to acquire pre-trained language models in the medical field, as well as structured knowledge graphs containing medical entities and relationships between entities.

[0141] A supervised fine-tuning unit is used to determine the training dataset and perform supervised fine-tuning on the pre-trained language model using the training dataset to obtain an initial alignment model.

[0142] A composite reward unit is used to generate human expert preference scores and fact consistency verification results based on the initial alignment model and the structured knowledge graph, and to generate a composite reward model based on the human expert preference scores and fact consistency verification results. The composite reward model consists of a first reward component and a second reward component.

[0143] The weighting combination unit is used to weight and combine the first reward component and the second reward component to obtain the target reward function, wherein the weight coefficient of the second reward component indicates the proportion of factual accuracy in the overall reward.

[0144] The reinforcement learning unit is used to train the initial alignment model using the target reward function to obtain the final aligned large medical model.

[0145] A third aspect of the present invention provides an electronic device, comprising:

[0146] processor;

[0147] Memory used to store processor-executable instructions;

[0148] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0149] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0150] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reinforcement learning alignment method for large-scale medical models based on knowledge graph fact consistency, characterized in that, include: Acquire pre-trained language models in the medical field, as well as structured knowledge graphs containing medical entities and relationships between them; A training dataset is determined, and the pre-trained language model is supervised and fine-tuned using the training dataset to obtain an initial alignment model; Based on the initial alignment model and the structured knowledge graph, human expert preference scores and fact consistency verification results are generated. Based on the human expert preference scores and fact consistency verification results, a composite reward model is generated, which consists of a first reward component and a second reward component. The target reward function is obtained by weighting the first reward component and the second reward component, wherein the weight coefficient of the second reward component indicates the proportion of factual accuracy in the overall reward; The initial alignment model is trained using the target reward function to obtain the final aligned large medical model.

2. The method according to claim 1, characterized in that, Determine the training dataset, and use the training dataset to perform supervised fine-tuning of the pre-trained language model to obtain an initial alignment model, including: Acquire question-answer pair data in the medical field, and extract the question text and answer text from each question-answer pair; Medical entity recognition is performed on the question text to extract medical entities from the question text as query entities, and the entity nodes corresponding to the query entities are located in the structured knowledge graph. Starting from the entity node, perform a multi-hop path search in the structured knowledge graph to obtain a path segment from the query entity to the target entity; The path segment is linearized to obtain linearized path segment text. The question text, the linearized path segment text, and the answer text are then concatenated according to a preset template to generate a single training sample. For each question-answer pair, corresponding training samples are generated to obtain the training dataset. The training dataset is input into the pre-trained language model for parameter updates, enabling the pre-trained language model to learn the mapping relationship for generating the answer text given the question text and the path fragment text, thus obtaining the initial alignment model.

3. The method according to claim 1, characterized in that, Based on the initial alignment model and the structured knowledge graph, human expert preference scores and fact consistency verification results are generated. Based on the human expert preference scores and the fact consistency verification results, a composite reward model is generated, including: Multiple input questions are sampled from the prompt dataset, and the input questions are input into the initial alignment model, which then generates multiple candidate answers for each input question. The candidate answers are evaluated for quality and scored to obtain the human expert preference score; A medical entity extraction operation is performed on the candidate answers to identify the medical entities appearing in the candidate answers. These medical entities are then used as query objects for retrieval in the structured knowledge graph. In the structured knowledge graph, locate the entity node corresponding to the medical entity, and extract the triplet of relationships between the medical entities as facts of the knowledge graph; Text fragments representing relationships between entities are extracted from the candidate answers. Semantic comparison is performed between the entity relationships represented in the text fragments and the facts in the knowledge graph to obtain the fact consistency verification result. The human expert preference score is converted into the first reward component, and the factual consistency verification result is converted into the second reward component. The composite reward model is then determined based on the first reward component and the second reward component.

4. The method according to claim 3, characterized in that, Text fragments representing relationships between entities are extracted from the candidate answers. A semantic comparison is performed between the entity relationships represented in the text fragments and the facts in the knowledge graph to obtain factual consistency verification results, including: Syntactic analysis is performed on the candidate responses to identify sentence segments containing medical entities. Determine the predicate structure representing the association between medical entities from the sentence fragment, and obtain the text fragment representing the relationship between entities based on the predicate structure; The relation descriptors in the text fragment are converted into standardized relational semantic representations, and the relation types in the relation triples indicated by the facts in the knowledge graph are converted into corresponding relational semantic representations. The degree of consistency between the two relational semantic representations is obtained by calculating the similarity in the semantic vector space. Based on the consistency degree value, the consistency relationship between the entity relationship expressed by the text fragment and the facts of the knowledge graph is determined to be contradictory, supportive, or irrelevant, thus obtaining the consistency relationship determination result; Based on the consistency relationship determination result, the verification value of the text segment is obtained, and based on the verification values ​​of all text segments in the candidate answer, the factual consistency verification result of the candidate answer is obtained.

5. The method according to claim 1, characterized in that, The target reward function is obtained by weighting and combining the first reward component and the second reward component, including: During reinforcement learning training, response samples generated by the initial alignment model on the validation set are collected. These response samples are then compared with standard answers annotated by human experts, and the semantic similarity between the response samples and the standard answers is calculated to obtain the human preference alignment value. The consistency of the medical entity relationship descriptions in the answer sample with the relationship triples in the structured knowledge graph is compared, and the accuracy of the knowledge fact is obtained by counting the proportion of the number of relationship descriptions in the answer sample that are consistent with the structured knowledge graph to the total number of relationship descriptions. A first weight coefficient is generated based on the human preference alignment value, and a second weight coefficient is generated based on the knowledge fact accuracy. The combined reward value is obtained based on the product of the first reward component and the first weight coefficient, and the product of the second reward component and the second weight coefficient; Generate the target reward function, which receives candidate answers as input and outputs the combined reward value.

6. The method according to claim 1, characterized in that, The initial alignment model is trained using reinforcement learning with the target reward function to obtain the final aligned large medical model, including: Input questions are sampled from the training dataset and fed into the initial alignment model, which then generates corresponding answer text based on the current model parameters. The answer text is input into the target reward function, which calculates the weighted value of the first reward component and the first weight coefficient, and the weighted value of the second reward component and the second weight coefficient based on the answer text, to obtain the reward signal corresponding to the answer text. The policy gradient loss function is determined based on the reward signal, and the knowledge graph consistency constraint term is determined based on the consistency comparison result between the entity relation representation extracted from the answer text and the relation triples in the structured knowledge graph; A comprehensive loss function is obtained based on the policy gradient loss function and the knowledge graph consistency constraint term; The model parameters of the initial alignment model are updated based on the comprehensive loss function until the reward signal of the initial alignment model converges on the validation set, thereby obtaining the final aligned large medical model.

7. The method according to claim 6, characterized in that, The policy gradient loss function is determined based on the reward signal, and the knowledge graph consistency constraint term is determined based on the consistency comparison result between the entity relation representation extracted from the answer text and the relation triples in the structured knowledge graph, including: Calculate the generation probability of the initial alignment model generating the answer text, and numerically combine the generation probability with the reward signal to obtain the policy gradient term; The policy gradient loss function is obtained based on the policy gradient terms of all response texts within the training batch; Perform medical entity recognition on the response text to obtain pairs of medical entities appearing in the response text, and determine entity relationship representations from the response text based on the pairs of medical entities; The entity relationship description is compared with the relationship triples in the structured knowledge graph to identify contradictory entity relationship descriptions and their corresponding semantic similarity deviation values. For entity relationship statements that are determined to be contradictory, a contradiction penalty value is calculated, and the contradiction penalty value is proportional to the semantic similarity deviation value. Based on the contradiction penalty value of all contradictory entity relation representations in the response text, the consistency constraint term of the knowledge graph is determined.

8. A medical large-scale model reinforcement learning alignment system based on knowledge graph fact consistency, used to implement the method as described in any one of claims 1-7, characterized in that, include: The model acquisition unit is used to acquire pre-trained language models in the medical field, as well as structured knowledge graphs containing medical entities and relationships between entities. A supervised fine-tuning unit is used to determine the training dataset and perform supervised fine-tuning on the pre-trained language model using the training dataset to obtain an initial alignment model. A composite reward unit is used to generate human expert preference scores and fact consistency verification results based on the initial alignment model and the structured knowledge graph, and to generate a composite reward model based on the human expert preference scores and fact consistency verification results. The composite reward model consists of a first reward component and a second reward component. The weighting combination unit is used to perform a weighted combination based on the first reward component and the second reward component to obtain the target reward function, wherein the weight coefficient of the second reward component indicates the proportion of factual accuracy in the overall reward; The reinforcement learning unit is used to train the initial alignment model using the target reward function to obtain the final aligned large medical model.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images