Retrieval enhancement generation method and device based on medical knowledge fine tuning language model

Abstract

The invention discloses a retrieval enhancement generation method and device based on a medical knowledge fine-tuning language model. The method comprises the steps of constructing a medical knowledge base, performing fine-tuning on the medical language model, inputting medical questions by a user, retrieving the knowledge base, screening knowledge fragments, generating answers and the like. And the medical knowledge base is constructed through semantic partitioning, so that the integrity and continuity of related knowledge are ensured. And a medical knowledge fine tuning model is used as a screener to further screen high-quality knowledge fragments most related to user questions, so that the professionality and reliability of generating answers are improved. The method is suitable for the medical field, and can provide more accurate and more professional medical consultation services for patients and doctors. The fine-adjusted medical model is innovatively used as a filter, knowledge fragments are evaluated from multiple dimensions, it is ensured that the knowledge fragments are highly related to questions, powerful support is provided for generating professional and accurate medical answers, and the precise medical consultation requirement of a user is better met.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and more specifically, to a retrieval-augmented generation method and device for fine-tuning a language model based on medical knowledge. Background Art

[0002] With the rapid development of the medical field, the types of diseases and treatment methods are constantly increasing, and medical information has become multi-dimensional and complex. Patients' demand for medical knowledge consultation continues to rise, and they expect to receive more comprehensive and professional health guidance. However, doctors have limited time and energy and it is difficult to meet the consultation needs of numerous patients. Although the emergence of online health communities provides a platform for patients and doctors to obtain medical information and communicate, in the face of complex medical information, there may be cases of information missing or misleading patients in these communities. Ensuring the accuracy and accessibility of medical information has become a key concern in the medical field.

[0003] Large language models (LLMs) in the general domain have made remarkable progress in natural language task processing. Models such as ChatGPT, after being pre-trained on large-scale corpora, have the ability to generate text and understand semantics, and can answer common health questions of patients and basic medical knowledge to a certain extent. However, due to the training corpora being mostly general texts, general models have limitations in complex medical scenarios, and their professionalism and reliability are not as good as those of human experts, and they may generate misleading information.

[0004] To address the problem of the poor performance of general LLMs in specific domains, researchers have actively explored solutions. One effective method is to develop LLMs pre-trained with specific domain knowledge. In the medical field, this method has shown remarkable results. Specifically, by pre-training on medical-specific corpora, such as medical literature, clinical guidelines, and electronic health records, the model can deeply learn and understand professional terms, concepts, and complex knowledge systems in the medical field. This pre-training process enables the model to have a more profound understanding of the expression methods, semantic logics, and professional knowledge of medical language, thus significantly improving the model's performance in medical natural language processing tasks. After such pre-training, when processing medical tasks, the model can more accurately identify and understand medical information and generate results that better meet the requirements of medical professionals. This improvement has, to a certain extent, made up for the deficiencies of general models in the medical field, provided strong support for the development of medical natural language processing technology, and laid a solid foundation for further optimizing the application of the model in the medical field.

[0005] However, there are also some problems with pre-training techniques. First of all, pre-training on large-scale datasets requires a large amount of computing resources, which not only leads to high costs but also low efficiency. The pre-training process usually requires the use of high-performance computing devices and consumes a large amount of time and energy, which to a certain extent limits the training scale and speed of the model. Secondly, the knowledge of the pre-trained model is relatively static and has a weak perception ability for dynamically changing medical knowledge. Medical knowledge is constantly updated and developed, with new research results, clinical guidelines, and treatment methods emerging continuously. After the pre-trained model is trained, it is difficult to obtain and integrate this new knowledge in a timely manner, resulting in less than ideal performance when facing the latest medical information.

[0006] To address these issues, researchers have proposed a technique called Retrieval-Augmented Generation (RAG). The core idea of RAG technology is to combine model generation with external knowledge base retrieval. Specifically, when processing a user's input question, the RAG model first retrieves knowledge fragments related to the question from the external knowledge base, and then combines these knowledge fragments with the model's own generation ability to generate more accurate and comprehensive answers. In this way, RAG technology can enhance the model's perception ability of dynamic knowledge. When medical knowledge is updated, only the external knowledge base needs to be updated, and the model can refer to the latest knowledge when generating answers, thus ensuring the timeliness and accuracy of the answers. In addition, RAG technology reduces the demand for large-scale pre-training computing resources to a certain extent because the model does not need to learn all the knowledge during the training phase but obtains the required information through retrieval, making the training and application of the model more efficient and flexible.

[0007] However, most of the existing RAG frameworks are designed for general domains and mainly consider the characteristics and requirements of general knowledge in the design and implementation process, without fully considering the uniqueness and professionalism of the medical field, resulting in poor application effects of the framework in the medical field and low accuracy and efficiency in generating answers to medical questions. Summary of the Invention

[0008] Aiming at the problem of low accuracy and efficiency in generating answers to medical questions by the RAG framework in the prior art, the present invention provides a retrieval-augmented generation method based on fine-tuning a language model with medical knowledge, which integrates a fine-tuned medical knowledge language model as a filter to perform retrieval-augmented generation. It innovatively uses the fine-tuned medical model as a filter to evaluate knowledge fragments from multiple dimensions to ensure its high relevance to the question, providing strong support for generating professional and accurate medical answers and better meeting the needs of users for precision medical consultations.

[0009] To solve the above technical problems, a retrieval-augmented generation method for fine-tuning a language model based on medical knowledge is provided in the first aspect of the present invention, including:

[0010] Based on an open-source knowledge base and common medical problems, a knowledge base is constructed by using the method of semantic chunking. Among them, constructing a knowledge base by using the method of semantic chunking includes: classifying and organizing medical knowledge, and dividing the knowledge base into multiple knowledge fragments. Among them, each knowledge fragment semantic block contains relevant medical concepts, terms, and knowledge points;

[0011] Fine-tune the base model with professional corpus in the medical field to obtain the fine-tuned model as a filter;

[0012] Obtain the medical problem input by the user;

[0013] According to the matching degree between the medical problem input by the user and the knowledge fragments in the knowledge base, return the top K knowledge fragments related to the medical problem input by the user from the knowledge base;

[0014] Use the filter to screen the top K knowledge fragments to obtain the target knowledge fragments;

[0015] Input the target knowledge fragments into the generation model to generate an answer to the user's medical problem.

[0016] In one implementation, classifying and organizing medical knowledge and dividing the knowledge base into multiple knowledge fragments includes:

[0017] Use natural language processing technology to perform topic clustering on medical texts;

[0018] Classify medical knowledge or concepts with similar semantic depths into the same knowledge fragment, where similar semantic depths include similarity in text semantics and medical entity relationships.

[0019] In one implementation, the method further includes:

[0020] Within each knowledge fragment, construct a knowledge graph to visually display the relationships between medical concepts, terms, and knowledge points.

[0021] In one implementation, according to the relationship between the medical problem input by the user and the knowledge fragments in the knowledge base, returning the top K knowledge fragments related to the medical problem input by the user from the knowledge base includes:

[0022] Calculate the cosine similarity between the medical problem input by the user and the knowledge fragments in the knowledge base;

[0023] Sort according to the size of the cosine similarity from high to low, and select the top K knowledge fragments as the returned top K knowledge fragments.

[0024] In one implementation, a filter is used to screen the top K knowledge fragments to obtain target knowledge fragments, including:

[0025] The filter screens medical knowledge fragments from the dimensions of the deep semantics of medical texts and the relationships between medical entities, and retains the knowledge fragments with the screening result of True. Among them, in the dimension of the deep semantics of medical texts, the relevance between medical knowledge fragments and medical problems is captured through deep semantic matching; in the dimension of the relationships between medical entities, the consistency or relevance between medical problems and medical entities in knowledge fragments is judged by constructing a relationship network between entities.

[0026] In one implementation, in the dimension of the deep semantics of medical texts, the relevance between medical knowledge fragments and medical problems is captured through deep semantic matching, including:

[0027] The filter pre-learns the explicit labels of "whether relevant" between medical knowledge fragments and medical problems, judges the surface matching between medical problems and knowledge fragments based on a binary decision method, and mines potential semantic commonalities.

[0028] In one implementation, the method further includes constructing a multi-dimensional screening Prompt template for the filter. The multi-dimensional screening Prompt template includes a system prompt word and a standardized input template. The system prompt word is used as a guiding instruction for task execution to specify the behavior pattern and answer format of the filter, and the standardized input template defines a unified input format for the input of the model, including medical problem text, knowledge fragment text, and format requirements for answers.

[0029] Based on the same inventive concept, a retrieval-enhanced generation device for fine-tuning a language model based on medical knowledge is provided in the second aspect of the present invention, including:

[0030] A knowledge base construction module, configured to construct a knowledge base based on an open-source knowledge base and common medical problems by using a semantic chunking method. Among them, constructing a knowledge base by using a semantic chunking method includes: classifying and organizing medical knowledge, and dividing the knowledge base into multiple knowledge fragments, where each knowledge fragment semantic chunk contains relevant medical concepts, terms, and knowledge points;

[0031] A medical language model fine-tuning module, configured to fine-tune a base model by using professional corpora in the medical field to obtain a fine-tuned model as a filter;

[0032] A medical problem acquisition module, configured to acquire a medical problem input by a user;

[0033] A knowledge base retrieval module, configured to return the top K knowledge fragments related to the medical question input by the user from the knowledge base according to the matching degree between the medical question input by the user and the knowledge fragments in the knowledge base;

[0034] A knowledge fragment screening module, configured to screen the top K knowledge fragments by using a filter to obtain target knowledge fragments; An answer generation module, configured to input the target knowledge fragments into a generation model to generate an answer to the user's medical question

[0035] Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the retrieval enhanced generation method based on medical knowledge fine-tuning a language model described in the first aspect.

[0036] Based on the same inventive concept, a fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the retrieval enhanced generation method based on medical knowledge fine-tuning a language model described in the first aspect.

[0037] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0038] The present invention provides a retrieval enhanced generation method based on medical knowledge fine-tuning a language model. The main steps include constructing a medical knowledge base, fine-tuning a medical language model, a user inputting a medical question, retrieving the knowledge base, screening knowledge fragments, and generating an answer. Using the method of semantic chunking to construct the knowledge base can ensure the integrity and coherence of relevant knowledge. Using the model fine-tuned with medical knowledge as a filter can further screen the most relevant high-quality knowledge fragments from the knowledge fragments retrieved from the knowledge base, improving the professionalism and reliability of the generated answer. The present invention is applicable to the medical field, can provide more accurate and professional medical consultation services for patients and doctors, and has the advantage of seamlessly connecting to any base model. Compared with traditional methods, the present invention innovatively uses the fine-tuned medical model as a filter to evaluate knowledge fragments from multiple dimensions, ensuring its high relevance to the question, providing strong support for generating professional and accurate medical answers, and better meeting the user's precise medical consultation needs. Description of the Drawings

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0040] Figure 1 This is the overall flowchart of the retrieval augmented generation method for fine-tuning a language model based on medical knowledge in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of a traditional retrieval augmented generation framework in the prior art;

[0042] Figure 3 This is a schematic diagram of a retrieval augmented generation framework that incorporates a fine-tuned model based on medical knowledge as a filter in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the knowledge base being partitioned into fixed-sized chunks in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of full-parameter fine-tuning and LoRA fine-tuning of the model in an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of the LoRA principle in an embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of a multi-dimensional screening Prompt template for medical knowledge in an embodiment of the present invention;

[0047] Figure 8 This is a module diagram of a retrieval augmented generation device for fine-tuning a language model based on medical knowledge in an embodiment of the present invention;

[0048] Figure 9 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention;

[0049] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed implementation manners

[0050] Through a large amount of research and practice, the inventors of this application have found that most of the existing RAG frameworks are designed for general domains. During the design and implementation process, they mainly consider the characteristics and requirements of general knowledge and fail to fully consider the uniqueness and professionalism of the medical field. The medical field is highly professional and dynamic, involving complex disciplinary knowledge systems and strict verification standards. Medical knowledge not only includes a large number of professional terms, complex concepts and principles, but also specific knowledge in aspects such as the diagnosis, treatment, and prevention of various diseases. These knowledge need to be strictly clinically verified and scientifically evaluated before being applied to actual medical practice. The questions patients consult often have significant characteristics of the medical field, and may involve specific disease symptoms, treatment methods, drug use, etc. These questions require accurate and professional medical knowledge to answer. In addition, the knowledge fragment structure and data format of medical knowledge bases also differ greatly from those of general knowledge bases. Medical knowledge usually has complex structures and relationships, and specific coding, classification, and annotation methods are required to represent and organize it. For example, medical literature often contains a large number of charts, formulas, and professional terms, which are relatively rare in general knowledge bases. When general RAG technologies process this medical knowledge with special structures and formats, it is often difficult to achieve ideal results. Due to the above reasons, the application of general RAG technologies in the medical field is restricted and difficult to meet the requirements of high accuracy and high reliability in the medical field. In the medical field, any error or inaccurate information may lead to serious consequences, so the requirements for the accuracy and reliability of knowledge are extremely high. The application of general RAG technologies in the medical field needs to be further improved and optimized to adapt to the special needs and challenges of the medical field.

[0051] Based on the above considerations, the RAG framework designed in this invention that integrates a medical knowledge fine-tuned language model is a RAG framework specifically for the medical field. It is improved by introducing a fine-tuned medical language model as a knowledge filter and medical knowledge semantic chunking and other technologies. By integrating high-quality medical professional knowledge and optimizing the retrieval and generation processes in combination with the characteristics of the medical field, it can make LLMs perform better in the medical field and provide more professional and reliable answers. This helps to improve the efficiency and accuracy of patients' access to professional health knowledge, and can also provide auxiliary support for medical workers and relieve their work pressure. The RAG framework of this invention is expected to provide more powerful support for the application of artificial intelligence in the medical field and promote the improvement of medical service quality.

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0053] Embodiment 1

[0054] This embodiment discloses a retrieval-augmented generation method for fine-tuning a language model based on medical knowledge. Please refer to Figure 1 , including:

[0055] Step 1: Based on an open-source knowledge base and common medical problems, construct a knowledge base using the semantic chunking method. Among them, constructing the knowledge base using the semantic chunking method includes: classifying and organizing medical knowledge, and dividing the knowledge base into multiple knowledge fragments. Among them, each knowledge fragment semantic chunk contains relevant medical concepts, terms, and knowledge points;

[0056] Step 2: Fine-tune the base model using professional corpora in the medical field to obtain a fine-tuned model as a filter;

[0057] Step 3: Obtain the medical problem input by the user;

[0058] Step 4: According to the matching degree between the medical problem input by the user and the knowledge fragments in the knowledge base, return the top K knowledge fragments related to the medical problem input by the user from the knowledge base;

[0059] Step 5: Use the filter to screen the top K knowledge fragments to obtain target knowledge fragments;

[0060] Step 6: Input the target knowledge fragments into the generation model to generate an answer to the user's medical problem.

[0061] Specifically, the schematic diagram of the traditional RAG framework is as shown in Figure 2 . Its basic process is to retrieve relevant knowledge fragments from the knowledge base according to the question raised by the user, and then input these knowledge fragments and the user's question into the generator together to finally generate an answer. This traditional RAG framework performs well in dealing with problems in general fields. However, in the medical field, due to the professionalism and complexity of medical knowledge, the traditional RAG framework may generate inaccurate answers due to problems with the quality and relevance of knowledge fragments.

[0062] The RAG framework integrating medical knowledge fine-tuning language model proposed by the present invention is as shown in Figure 3As shown, its main improvement lies in integrating a model fine-tuned with medical knowledge as a filter. Specifically, this fine-tuned model further filters the knowledge fragments retrieved from the knowledge base to ensure that only high-quality and highly relevant knowledge fragments to the user's question can be input into the generator. In this way, the generator can generate answers based on more accurate and reliable knowledge fragments, thus significantly improving the accuracy and professionalism of the answers. This improvement is particularly applicable to the medical field because medical knowledge not only requires a high degree of professionalism but also strict quality control to ensure that patients can obtain accurate and reliable health information.

[0063] In the specific implementation process, in step 1, the open-source knowledge base includes PubMed, medical textbooks, the medical section of Wikipedia, etc., and common medical questions include common patient questions, doctor's diagnosis guidelines, etc. Through the method of semantic chunking, the knowledge base can be divided into multiple knowledge fragments, and each knowledge fragment semantic block contains relevant medical concepts, terms, and knowledge points for subsequent retrieval and matching (step 4).

[0064] Suppose the knowledge base D contains multiple knowledge fragments S 1 , S 2 , …, S n , then the knowledge base can be represented by the following formula:

[0065] D = {S 1 , S 2 , …, S n}

[0066] Among them, S 1、 S n represent the first and the nth knowledge fragments respectively.

[0067] In step 2, the base model is fine-tuned with medical knowledge to obtain the filter Filter, and the filter Filter is used to assist in the screening of knowledge fragments. Specifically, the base model is fine-tuned using the professional corpus in the medical field to make it better understand medical knowledge, improve the performance of the model in the medical field, and then assist in the subsequent screening of knowledge fragments. Suppose M represents the base model and the filter obtained after fine-tuning is Filter, then this process can be represented as:

[0068] Filter = FineTune(M)

[0069] FineTune represents fine-tuning.

[0070] In step 3, the user can input a medical question Q through the system interface or relevant channels. The question is usually a consultation on disease symptoms, treatment methods, drug use, etc. Suppose the system triggers the subsequent knowledge retrieval and generation process based on the medical question Q input by the user, and this process can be expressed as:

[0071] Q = UserInput

[0072] UserInput represents the user input.

[0073] The matching degree between the medical question input by the user in step 4 and the knowledge fragments in the knowledge base can include relevance, consistency, etc. During the retrieval process, natural language processing technology is used to analyze and understand the medical question Q input by the user, and the medical question Q is retrieved in the constructed medical knowledge base to obtain the top K knowledge fragments Snippets related to Q.

[0074] In step 5, a filter Filter is used to screen the top K knowledge fragments Snippets.

[0075] The medical question Q and each knowledge fragment are input into the filter Filter in turn. Using the semantic understanding and judgment ability of the filter, these top K knowledge fragments Snippets are further screened and evaluated from multiple dimensions to determine whether the knowledge fragment can support the answer to the user's question Q, and the high-quality knowledge fragment Snippets-pro (target knowledge fragment) most relevant to the user's question is obtained. This process can be expressed as:

[0076] Snippets-pro = {S i ∣ Filter(Q, S i ) = True for i = 1, 2, …, K}

[0077] K represents the Kth knowledge fragment.

[0078] In step 6, the high-quality knowledge fragment is input to the generation model Generator.

[0079] The high-quality knowledge fragment obtained by screening is used as the input and provided to the generation model Generator. The generation model Generator generates an answer Answer to the user's medical question based on these knowledge fragments, combined with its own language generation ability. The generated answer will try to be accurate, clear, and easy to understand, and at the same time conform to medical professional knowledge and norms, providing valuable health guidance and information to the user. The generation process can be expressed by the following formula:

[0080] Answer = Generator(Q, Snippets-pro)

[0081] The present invention mainly proposes a retrieval enhanced generation method that fuses medical knowledge to fine-tune a language model as a retriever, based on the rapid development in the medical field and the increasing demand of patients for professional health knowledge, considering the limitations existing in the application of existing general RAG in the medical field. Through the present invention, it is possible to provide more accurate professional medical consultation services for patients, while assisting medical workers in improving work efficiency and reducing work burden, and ultimately promoting the improvement of the quality and efficiency of medical services.

[0082] In one implementation, medical knowledge is classified and sorted, and the knowledge base is divided into multiple knowledge fragments, including:

[0083] Using natural language processing technology to perform topic clustering on medical texts;

[0084] Medical knowledge or concepts with similar semantic depths are grouped into the same knowledge fragment, where similar semantic depths include similarities in text semantics and medical entity relationships.

[0085] Specifically, when constructing a medical knowledge base, the traditional method of dividing into fixed-size chunks, as Figure 4 shown, this method may cause knowledge in the same chapter to be split into different knowledge fragments, thus disrupting relevant knowledge. This chunking method ignores the internal logic and semantic associations between knowledge, which may lead to fragmentation between knowledge fragments and affect the integrity and coherence of knowledge. For example, a complete description of a treatment method for a certain disease may be split into multiple parts and scattered in different knowledge fragments, which not only increases the complexity of knowledge retrieval but also may lead to incomplete and inaccurate information.

[0086] In contrast, the method of chunking according to semantics can effectively overcome this drawback. Semantic chunking is based on the semantic and logical relationships of knowledge to perform chunking, integrating relevant knowledge into the same knowledge fragment. This method can ensure the integrity and coherence of knowledge, making the knowledge fragments more in line with human cognitive habits and knowledge organization methods.

[0087] In addition, compared with the text chunking in the prior art, the present invention is no longer limited to performing medical text chunking only from the perspectives of literal text similarity and paragraph extraction, but realizes semantic chunking through various means. In the specific process, the ways to achieve semantic chunking include:

[0088] Medical topic clustering. Using natural language processing technology (such as BERT) to perform topic clustering on medical texts, and grouping medical knowledge or concepts with similar semantic depths into the same knowledge fragment. For example, all texts related to "cardiovascular diseases" will be clustered into one knowledge fragment.

[0089] Among them, semantic depth similarity means that the knowledge fragments are not literally similar, but are similar in terms of text semantics, medical entity relationships, etc. The process of judging whether the semantics are deeply similar is completed by a large language model. If the large language model believes that two knowledge fragments are similar in terms of text semantics, medical entity relationships, etc., then they are considered to have semantic depth similarity.

[0090] In one implementation, the method further includes:

[0091] Inside each knowledge fragment, a knowledge graph is constructed to visually display the relationships between medical concepts, terms, and knowledge points.

[0092] Specifically, the semantic chunking method further includes the construction of a medical knowledge graph. In this implementation, inside each knowledge fragment, a knowledge graph is further constructed to visualize the relationships between medical concepts, terms, and knowledge points. For example, in the "diabetes" knowledge fragment, the knowledge graph can show the relationship between "insulin" and "blood glucose control".

[0093] In addition, it also includes the dynamic update of medical knowledge. Medical research results and clinical guidelines that support the dynamic update of the knowledge base are added to the corresponding knowledge fragments to ensure the timeliness of the medical knowledge base.

[0094] Compared with traditional text chunking methods, the present invention is no longer limited to medical text chunking only from the perspectives of literal text similarity and paragraph extraction. Specifically, the semantic chunking strategy used in the present invention has the following advantages:

[0095] (1) Efficient indexing retrieval and precise matching. Through semantic chunking, users can use specific medical topics as indexes to quickly and precisely locate relevant knowledge fragments, improving retrieval efficiency. For example, when a doctor diagnoses "hypertension", they can directly access the "hypertension" knowledge fragment to obtain all relevant diagnostic and treatment information.

[0096] (2) Integration of multi-source medical knowledge. Semantic chunking not only contains text information, but also integrates various forms of knowledge such as knowledge graphs, clinical guidelines, and research literature, providing users with comprehensive medical knowledge support. For example, in the "cancer" knowledge fragment, users can simultaneously obtain information such as the pathological mechanism of cancer, treatment plans, and the latest research progress.

[0097] In step 2, a base model is fine-tuned using medical knowledge to obtain a filter Filter, and the filter Filter is used to assist in the screening of knowledge fragments.

[0098] Such as Figure 5As shown in the figure, the fine-tuning of the base model can be divided into two categories: full-parameter fine-tuning and LoRA (Low-Rank Adaptation) fine-tuning. Both methods fine-tune the base model with medical task data to obtain the fine-tuned model. However, the LoRA fine-tuning method has significant advantages compared to full-parameter fine-tuning. Full-parameter fine-tuning adjusts all the parameters of the base model, which not only has a high computational cost but also easily leads to overfitting, especially when the amount of data is limited. In contrast, LoRA adjusts the key parts of the model by introducing low-rank matrices, greatly reducing the number of parameters to be trained, thus reducing the computational cost and memory occupancy. This method improves the training efficiency while maintaining the model performance, and is particularly suitable for fine-tuning models in resource-constrained environments. In addition, LoRA can better retain the general knowledge of the base model and avoid losing important information due to over-adjustment, thus achieving more stable and reliable performance on specific tasks.

[0099] In this embodiment, the base model is fine-tuned using professional medical corpora to enable it to better understand medical knowledge and improve the model's performance in the medical field, thereby assisting in the subsequent screening of knowledge fragments. Assume that M represents the base model, and the filter obtained after fine-tuning is Filter. Then this process can be expressed as:

[0100] Filter = FineTune(M)

[0101] The principle of LoRA is as Figure 6 shown. LoRA does not directly modify the pre-trained weight matrix W, but instead adjusts W by adding a low-rank matrix A·B. The adjusted weight matrix is W + A·B, and the input vector x passes through the adjusted weight matrix to obtain the output vector h. Assume that ΔM = A·B, where ΔM represents the adjustment amount of the weight during the LoRA training process, which is the changing part of W. Then the fine-tuning process can be further expressed as:

[0102] Filter = M + ΔM

[0103] Figure 6 where d is the dimension of the input and output, and r is the rank of the low-rank matrix (i.e., the reduced number of parameters). During the training process, only the low-rank matrices A and B are trained, while the pre-trained weight matrix W remains unchanged. This method can significantly reduce the number of training parameters, improve the training efficiency, and at the same time maintain the performance of the pre-trained model.

[0104] In step 3, the user inputs a medical question Q through the system interface or relevant channels. The question is usually a consultation regarding disease symptoms, treatment methods, drug use, etc. Assume that the system triggers the subsequent knowledge retrieval and generation process based on the medical question Q input by the user, and the medical question Q input by the user will be used in the subsequent process of retrieving knowledge fragments and generating answers.

[0105] In one embodiment, according to the relationship between the medical problem input by the user and the knowledge fragments in the knowledge base, the top K knowledge fragments related to the medical problem input by the user are returned from the knowledge base, including:

[0106] Calculate the cosine similarity between the medical problem input by the user and the knowledge fragments in the knowledge base;

[0107] Sort according to the size of the cosine similarity from high to low, and select the top K knowledge fragments as the returned top K knowledge fragments.

[0108] In the specific implementation process, the medical knowledge base D is retrieved according to the medical problem input by the user. The medical problem Q input by the user is analyzed and understood by using natural language processing technology, and retrieved in the constructed medical knowledge base according to the medical problem Q to obtain the top K knowledge fragments Snippets related to Q. Assuming retrieval is performed according to cosine similarity, then the cosine similarity between the medical problem Q and each knowledge fragment S i in the knowledge base D can be expressed as:

[0109]

[0110] In this embodiment, the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm is used for retrieval to measure the relevance between the problem Q and the knowledge fragment Si in the knowledge base. TF-IDF is a commonly used text feature weighting method, which can effectively represent the importance of words in a document and is used to measure the similarity between texts.

[0111] First, calculate the term frequency TF. Assuming that t is a word in the knowledge fragment S, the term frequency of t can be expressed as:

[0112]

[0113] Among them, the numerator count(t, S) represents the total number of times t appears in S, and the denominator represents the total number of all words in S.

[0114] Then calculate the inverse document frequency IDF. IDF can measure the generality of a certain word in all knowledge fragments and can be expressed as:

[0115]

[0116] Among them, |D| is the total number of knowledge fragments in the knowledge base D, and df(t, D) is the number of knowledge fragments containing the word t.

[0117] Combining term frequency and inverse document frequency, TF-IDF can be expressed as:

[0118] TF-IDF(t, d, D) = TF(t, d) × IDF(t, D)

[0119] Assume that TF-IDF(Q) is the TF-IDF vector representation of question Q, and TF-IDF(S i ) is the TF-IDF vector representation of knowledge snippet S i . Then the cosine similarity calculation formula between the two is:

[0120]

[0121] The numerator TF-IDF(Q) · TF-IDF(S i ) is the dot product of the vectors, and the denominator ‖TF-IDF(Q)‖‖TF-IDF(S i )‖ is the product of the vector norms.

[0122] For all knowledge snippets S i Sort them in descending order according to the cosine similarity CosineSimilarity(Q, S i ), and select the top K knowledge snippets. These snippets are the most relevant to question Q. The top K knowledge snippets Snippets relevant to question Q can be expressed as:

[0123] Snippets = {S 1 , S 2 , …, S K}.

[0124] In one implementation, use a filter to filter the top K knowledge snippets to obtain the target knowledge snippets, including:

[0125] Filter the medical knowledge snippets from the dimensions of the deep semantics of medical texts and the relationships of medical entities through the filter, and retain the knowledge snippets with the screening result of True. Among them, in the dimension of the deep semantics of medical texts, capture the relevance between medical knowledge snippets and medical questions through deep semantic matching; in the dimension of the relationships of medical entities, judge the consistency or relevance between medical questions and medical entities in knowledge snippets by constructing an entity relationship network.

[0126] Specifically, use the filter Filter to filter the top K knowledge snippets Snippets. The filter Filter will perform multi-level semantic understanding and judgment on each knowledge snippet S i to analyze whether this snippet matches question Q in the medical context.

[0127] The medical problem Q and each knowledge snippet are sequentially input into the filter Filter. Using the semantic understanding and judgment ability of the filter, these Top K knowledge snippets Snippets are further filtered and evaluated from multiple dimensions to determine whether the knowledge snippet can support the answer to the user's question Q, and the high-quality knowledge snippet Snippets-pro that is most relevant to the user's question is obtained. Assume Snippets = {S 1 , S 2 , …, S K} are the Top K knowledge snippets obtained through retrieval in the previous step. Then this process can be expressed as:

[0128] Snippets-pro = {S i ∣ Filter(Q, S i ) = True for i = 1, 2, …, K}

[0129] The difference between the filter proposed in this embodiment and the existing method of obtaining a relevance score through semantic analysis lies in:

[0130] Let the fine-tuned language model be used as the filter to screen medical knowledge snippets from dimensions such as the deep semantics of medical texts and the relationships between medical entities, and retain the knowledge snippets with the screening result of True. In the dimension of the deep semantics of medical texts, the relevance between medical knowledge snippets and medical problems is captured through deep semantic matching; in the dimension of the relationships between medical entities, it focuses on whether the medical entities in the medical problem and the knowledge snippet are consistent or mutually related; the filter can also judge from other aspects whether the knowledge snippet contains supplementary medical information that helps answer the medical problem.

[0131] In the method of this embodiment, the fine-tuned language model acts as a high-level filter. Its core lies in transforming the complex medical text screening problem into a binary decision-making process, retaining the knowledge snippets with the screening result of True, rather than relying on traditional continuous relevance scores.

[0132] In one implementation, in the dimension of the deep semantics of medical texts, capturing the relevance between medical knowledge snippets and medical problems through deep semantic matching includes:

[0133] The filter pre-learns the clear labels of "whether relevant" between medical knowledge snippets and medical problems, judges the surface matching between medical problems and knowledge snippets based on the binary decision-making method, and mines out potential semantic commonalities.

[0134] In one embodiment, the method further includes constructing a multi-dimensional screening Prompt template for the filter, where the multi-dimensional screening Prompt template includes a system prompt and a standardized input template. The system prompt serves as a guiding instruction for task execution, specifying the behavior pattern and response format of the filter. The standardized input template defines a unified input format for the model's input, including medical question text, knowledge fragment text, and format requirements for the response.

[0135] As Figure 7 shown, the present invention enables the filter to achieve refined judgment through the following dimensions by means of a specially designed multi-dimensional screening Prompt for medical knowledge:

[0136] (1) Deep semantic matching of medical knowledge. The filter not only identifies the surface match between the medical question and the knowledge fragment but also can uncover potential semantic commonalities. Specifically, deep semantic matching models the context information such as medical questions and knowledge fragments using a fine-tuned Transformer-based model (such as Qwen, Deepseek, etc.), and utilizes a multi-level self-attention mechanism to capture the complex semantic associations between words and syntactic structures. Through in-depth learning of medical terms, synonyms, and professional expressions, the model can go beyond the surface text match and discover potential semantic similarities. At the same time, the advantage of the binary decision method is that the filter model can more precisely define the critical point of semantic matching by learning the explicit label of "whether relevant", avoiding the ambiguity and uncertainty of similarity scores near the critical value. The filter will learn how to judge the relationship between the question and the knowledge fragment in different contexts during the training phase, thereby achieving high-precision semantic matching.

[0137] (2) Recognition of medical entity relationships. The filter also pays attention to the consistency or mutual relevance of medical entities in the question and the knowledge fragment. In practical applications, medical questions and knowledge fragments often contain multiple medical entities, such as diseases, symptoms, treatment methods, drugs, etc. Identifying these entities and their relationships is crucial for improving the accuracy of question answering. The filter first extracts medical entities in the text through named entity recognition technology, and then further constructs a relationship network between entities based on a graph neural network. This network can capture the semantic connections and reasoning logic between medical entities.

[0138] The following are several specific examples of the network between medical entities:

[0139] Relationship between drug and disease: Amoxicillin treats bacterial infections;

[0140] Relationship between symptom and disease: Persistent cough and dyspnea are common in pneumonia or chronic obstructive pulmonary disease;

[0141] Relationship between diseases: Diabetes can cause complications such as kidney disease and hypertension;

[0142] Relationship between diseases and treatment methods: Breast cancer can be treated by various methods such as chemotherapy, radiotherapy, and surgery;

[0143] Through the entity relationship network, the filter can identify those text fragments that truly have the value of knowledge supplementation, thereby improving the relevance and accuracy of the retrieval results. The construction of the relationship network not only considers the explicit semantics but also combines the context information of the entities, so as to achieve more accurate relationship recognition and reasoning.

[0144] (3) Other medical information. The filter evaluates whether the knowledge fragment also contains other information helpful for answering the question.

[0145] The binary decision-making method implemented by fine-tuning the language model can not only fully capture the deep semantics of medical texts and the complex relationships between entities but also provide an efficient, interpretable, and robust filtering mechanism through the explicit True / False output.

[0146] Figure 7 The multi-dimensional screening Prompt template for medical knowledge shown is mainly composed of a system prompt and a standardized input template, aiming to guide the filter to complete the refined screening of medical knowledge fragments through structured instructions. The design of this template fully considers the complexity and diversity of medical domain knowledge. Through multi-dimensional feature extraction and evaluation, it significantly improves the model's ability to capture the relevance between medical questions and knowledge fragments. The system prompt, as the guiding instruction for task execution, clearly stipulates the behavior pattern and answer format of the filter. The standardized input template defines a unified input format for the model's input, including the medical question text, the knowledge fragment text, and the format requirements for the answer, ensuring the repeatability and consistency of the knowledge fragment screening process.

[0147] Finally, through step 6, the high-quality knowledge fragments obtained by screening are used as input and provided to the generation model Generator. Based on these knowledge fragments, the generation model Generator combines its own language generation ability to generate an answer Answer to the user's medical question. The generated answer will try to be accurate, clear, and easy to understand, and at the same time conform to medical professional knowledge and norms, providing valuable health guidance and information to users.

[0148] The Generator will, based on the semantic information of the question Q and the knowledge snippets Snippets-pro, and combined with its own language generation ability, output an answer that conforms to medical professional knowledge, is clear and easy to understand, and has high value. The Generator can use a variety of techniques, usually adopting pre-trained base models (such as GPT, Qwen, etc.). During the generation process, the model not only relies on the input knowledge snippets but also needs to maintain an understanding of the context of the user's question to ensure the coherence and logic of the generated answer.

[0149] The beneficial effects of the present invention include:

[0150] 1. Improve the quality of knowledge snippets in the medical knowledge base

[0151] By means of knowledge snippet topic clustering, internal knowledge graph construction of knowledge snippets, etc., it is ensured that relevant medical knowledge is integrated into the same medical knowledge snippet, so that the retrieved knowledge snippet is more relevant to the question. Through the dynamic update of the medical knowledge base, the timeliness and high quality of the knowledge snippets contained in the medical knowledge base are further guaranteed. 2. Improve the quality of knowledge snippets in the medical knowledge base

[0152] 2. Reduce the probability of the answer containing hallucinations and thus improve the quality of the answer

[0153] Using a model fine-tuned with medical knowledge as a filter can further screen and evaluate the retrieved knowledge snippets from multiple dimensions, ensuring that only high-quality and most relevant knowledge snippets to the user's question are used to assist in the generation of the answer. Irrelevant knowledge snippets are completely filtered out, so that the Generator can better capture key information, reduce the probability of the answer containing hallucinations, and improve the quality of the answer. Generate the answer. This significantly improves the professionalism and reliability of the generated answer.

[0154] 3. Be applicable to the medical field and facilitate cross-field migration

[0155] The present invention is specifically designed for the characteristics and needs of the medical field and can provide more accurate and professional medical consultation services for patients and doctors. Patients can obtain accurate health information faster, and doctors can also use this system to assist in diagnosis and treatment, improving work efficiency. In addition, by adjusting the fine-tuning process for the base model, it can also be quickly adapted to other fields, such as the legal, financial, etc. fields.

[0156] 4. Improve user satisfaction

[0157] By providing more accurate and professional medical consultation services, the present invention can significantly improve user satisfaction. Patients can obtain accurate health information faster, and doctors can also use the system more efficiently to assist in their work, thus improving the quality and efficiency of the overall medical service.

[0158] Generally speaking, compared with the existing methods, the present invention innovatively uses a fine-tuned medical model as a filter to more accurately screen knowledge fragments related to the user's medical problems and improve the retrieval quality. Traditional methods often have weak relevance and poor quality of retrieval results due to insufficient understanding of medical knowledge. The filter of the present invention is based on in-depth learning of medical knowledge, evaluates knowledge fragments from multiple dimensions, ensures its high relevance to the problem, provides strong support for generating professional and accurate medical answers, and better meets the needs of users for precise medical consultation.

[0159] Embodiment 2

[0160] Based on the same inventive concept, this embodiment discloses a retrieval enhanced generation device based on fine-tuning a language model with medical knowledge. Please refer to Figure 8 , including:

[0161] A knowledge base construction module 101, configured to construct a knowledge base based on an open-source knowledge base and common medical problems by using a semantic chunking method. Among them, constructing the knowledge base by using the semantic chunking method includes: classifying and organizing medical knowledge, and dividing the knowledge base into multiple knowledge fragments, where each knowledge fragment semantic block contains relevant medical concepts, terms, and knowledge points;

[0162] A medical language model fine-tuning module 102, configured to fine-tune a base model by using professional corpus in the medical field to obtain a fine-tuned model as a filter;

[0163] A medical problem acquisition module 103, configured to acquire a medical problem input by a user;

[0164] A knowledge base retrieval module 104, configured to return the top K knowledge fragments related to the medical problem input by the user from the knowledge base according to the relationship between the medical problem input by the user and the knowledge fragments in the knowledge base;

[0165] A knowledge fragment screening module 105, configured to screen the top K knowledge fragments by using the filter to obtain target knowledge fragments;

[0166] An answer generation module 106, configured to input the target knowledge fragments into a generation model to generate an answer to the user's medical problem.

[0167] Since the device introduced in Embodiment 2 of the present invention is the device adopted for implementing the retrieval enhanced generation method based on fine-tuning a language model with medical knowledge in Embodiment 1 of the present invention, based on the method introduced in Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the device, so it will not be elaborated here. Any device adopted for the method in Embodiment 1 of the present invention belongs to the scope protected by the present invention.

[0168] Embodiment III

[0169] Based on the same inventive concept, the present invention further provides a computer-readable storage medium 300. Please refer to Figure 9 , on which a computer program 311 is stored, and when the program is executed by a processor, it implements the method described in Embodiment I.

[0170] Since the computer-readable storage medium introduced in Embodiment III of the present invention is the computer-readable storage medium used to implement the retrieval enhanced generation method for fine-tuning a language model based on medical knowledge in Embodiment I of the present invention, based on the method described in Embodiment I of the present invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium, so it will not be elaborated here. Any computer-readable storage medium used in the method of Embodiment I of the present invention falls within the scope of protection of the present invention.

[0171] Embodiment IV

[0172] The present invention further provides a computer device. Please refer to Figure 10 , including a memory 401, a processor 402, and a computer program 403 stored on the memory and executable on the processor. When the processor executes the program, it implements the method described in Embodiment I.

[0173] Since the computer device introduced in Embodiment IV of the present invention is the computer device used to implement the retrieval enhanced generation method for fine-tuning a language model based on medical knowledge in Embodiment I of the present invention, based on the method described in Embodiment I of the present invention, those skilled in the art can understand the specific structure and variations of this computer device, so it will not be elaborated here. Any computer device used in the method of Embodiment I of the present invention falls within the scope of protection of the present invention.

[0174] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0175] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general purpose computers, special purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices create means for implementing the functions specified in one flow Figure 1 one flow or more flows and / or blocks Figure 1 or means for implementing the functions specified in one block or more blocks.

[0176] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art can make various changes and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A retrieval enhancement generation method based on a medical knowledge fine-tuned language model, characterized in that: include: Based on the open source knowledge base and common medical problems, the knowledge base is constructed by using the semantic block method, wherein the knowledge base is constructed by using the semantic block method, including: classifying and organizing medical knowledge, dividing the knowledge base into multiple knowledge fragments, wherein each knowledge fragment semantic block contains relevant medical concepts, terms and knowledge points; Fine-tune the base model using professional corpus in the medical field to obtain a fine-tuned model as a filter; Get medical questions input by users; According to the matching degree between the medical question input by the user and the knowledge fragments in the knowledge base, the top K knowledge fragments related to the medical question input by the user are returned from the knowledge base; Use the filter to filter the Top K knowledge fragments to obtain the target knowledge fragments; The target knowledge fragment is input into the generative model to generate answers to the user's medical questions.

2. The retrieval enhancement generation method based on the medical knowledge fine-tuning language model according to claim 1, characterized in that: Classify and organize medical knowledge and divide the knowledge base into multiple knowledge fragments, including: Topic clustering of medical texts using natural language processing techniques; Medical knowledge or concepts with similar semantic depth are grouped into the same knowledge segment, where semantic depth similarity includes similarity in text semantics and medical entity relationships.

3. The retrieval enhancement generation method based on the medical knowledge fine-tuning language model as claimed in claim 2, characterized in that: The method further comprises: Within each knowledge fragment, a knowledge graph is constructed to visualize the relationships between medical concepts, terms, and knowledge points.

4. The retrieval enhancement generation method based on the medical knowledge fine-tuning language model according to claim 1, characterized in that: According to the relationship between the medical question input by the user and the knowledge fragments in the knowledge base, the top K knowledge fragments related to the medical question input by the user are returned from the knowledge base, including: Calculate the cosine similarity between the medical question input by the user and the knowledge fragments in the knowledge base; Sort the knowledge fragments from high to low according to the cosine similarity, and select the top K knowledge fragments as the returned Top K knowledge fragments.

5. The retrieval enhancement generation method based on the medical knowledge fine-tuning language model according to claim 1, characterized in that: Use the filter to filter the Top K knowledge fragments to obtain the target knowledge fragments, including: Medical knowledge fragments are screened from the dimensions of deep semantics of medical text and medical entity relationships through the filter, and knowledge fragments with True screening results are retained. In the dimension of deep semantics of medical text, the correlation between medical knowledge fragments and medical problems is captured through deep semantic matching; in the dimension of medical entity relationships, the consistency or correlation between medical problems and medical entities in knowledge fragments is judged by constructing an inter-entity relationship network.

6. The retrieval enhancement generation method based on the medical knowledge fine-tuning language model according to claim 5, characterized in that: In the deep semantic dimension of medical text, the relevance of medical knowledge fragments and medical problems is captured through deep semantic matching, including: The filter pre-learns clear labels on whether medical knowledge fragments are "related" to medical questions, judges the surface match between medical questions and knowledge fragments based on a binary decision method, and mines out potential semantic commonalities.

7. The retrieval enhancement generation method based on the medical knowledge fine-tuning language model according to claim 5, characterized in that: The method also includes constructing a multi-dimensional screening prompt template for the filter, wherein the multi-dimensional screening prompt template includes a system prompt word and a standardized input template, the system prompt word is used as a guiding instruction for task execution to specify the behavior mode and answer format of the filter, and the standardized input template defines a unified input format for the input of the model, including the format requirements of the medical question text, the knowledge fragment text and the answer.

8. A retrieval enhancement generation device based on a medical knowledge fine-tuned language model, characterized in that: include: A knowledge base construction module is used to construct a knowledge base based on an open source knowledge base and common medical problems by adopting a semantic block method, wherein the method of constructing a knowledge base by adopting a semantic block method includes: classifying and organizing medical knowledge, dividing the knowledge base into multiple knowledge fragments, wherein each knowledge fragment semantic block contains relevant medical concepts, terms and knowledge points; The medical language model fine-tuning module is used to fine-tune the base model using professional corpus in the medical field to obtain a fine-tuned model as a filter; A medical question acquisition module is used to acquire the medical question input by the user; The knowledge base retrieval module is used to return the top K knowledge fragments related to the medical question input by the user from the knowledge base according to the matching degree between the medical question input by the user and the knowledge fragments in the knowledge base; The knowledge fragment screening module is used to use the filter to screen the top K knowledge fragments to obtain the target knowledge fragment; The answer generation module is used to input the target knowledge fragment into the generation model to generate answers to the user's medical questions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the retrieval enhancement generation method based on the medical knowledge fine-tuning language model as described in any one of claims 1 to 7 is implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the retrieval enhancement generation method based on the medical knowledge fine-tuning language model as described in any one of claims 1 to 7 is implemented.

Images