Method for generating diversified instruction data in medical field based on large language model

By applying natural language processing tools and TF-IDF algorithms on large language models, combining advanced prompt word evolution strategies and natural language inference technology, the accuracy and diversity problems in generating instruction data in the medical field are solved, and the model is improved Performance and application value.

CN119069138APending Publication Date: 2024-12-03BEIJING UNIV OF TECH
View PDF 0 Cites 7 Cited by

Patent Information

Application Number
CN202411184816.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

When the existing large language model generates instruction data in the medical field, it lacks accuracy, diversity and human preferences, resulting in the lack of change and innovation in expression of the generated Q&A pairs, which cannot effectively cover complex scenarios in clinical practice, affecting the generalization of the model. and practical application value.

Method used

Using a multi-step method based on a large language model, including data preprocessing of natural language processing tools, keyword extraction of TF-IDF algorithms, advanced prompt word evolution strategies, strict screening and filtering of questions and answers, combined with few-shot design and hyperparameter adjustments ensure the diversity and accuracy of problem generation, and verify the accuracy and relevance of answers through natural language reasoning techniques.

Benefits of technology

Significantly improve the quality and diversity of generated data, enhance the performance of the model in dealing with complex medical documents and diversified problems, and provides strong support for intelligent applications in the medical field, ensuring high quality and high availability of data. Reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119069138A_ABST
    Figure CN119069138A_ABST
Patent Text Reader

Abstract

The invention discloses a method for generating diversified instruction data in the medical field based on a large language model. Data preprocessing is performed on medical documents by utilizing a natural language processing tool. And a TF-IDF algorithm is used to extract keywords with the most informative and unique characteristics in the medical document. And an advanced cue word evolution strategy is designed. And strictly screening and filtering the generated questions by adopting a large language model. A standardized cue word template is created, the model is prompted to generate answers in combination with the chunk content of the specific document, it is ensured that the model can generate accurate, related and high-quality answers based on the document content, and therefore the answering ability of the model is improved. And the generated answers are strictly filtered to ensure the accuracy and correlation, so that the authenticity and specialty of question and answer data are improved. According to the method, a large amount of diversified instruction data is created through a large language model instead of manual work, so that the burden of manual data annotation is relieved, and meanwhile, the problems of data quality, diversity, accuracy and the like are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a method for generating diversified instruction data in the medical field based on a large language model. Background Art

[0002] With the rapid development of AI technology, large language models such as InstructGPT, ChatGPT, and GPT-4 have reshaped the boundaries of natural language processing (NLP) with their excellent language understanding and creative text generation capabilities. These models have not only attracted widespread attention in academia, but also promoted innovations in intelligent interaction and customer service in the commercial and industrial fields. However, there are many challenges in building large language models in specific fields. A key bottleneck has gradually emerged, that is, the scarcity of high-quality data in specific fields. This problem not only increases the manpower and time cost of model training, but also puts higher requirements on the quality and diversity of data. The scarcity of data has become a major challenge that restricts the further development and widespread application of large language models.

[0003] At present, most of the medical data in China is stored in medical institutions at all levels. The business systems are relatively independent, and data sharing is difficult. There is an obvious "data island" phenomenon. The real-scene data sets available for training are limited, and the open source Chinese medical question-and-answer data sets have problems such as single and simple question forms. The large language models trained in this way have weak ability to answer complex questions in the medical field. The resource constraints and unbalanced and insufficient resource allocation for training large language models limit the further development of large language models in the medical field. Creating supervised data for instruction adjustment faces huge challenges. This work requires a lot of manpower to annotate the data and work hard to solve problems related to data quality, diversity, accuracy and other related factors. Therefore, methods for using large language models instead of humans to create large amounts of domain-specific instruction data with different levels of complexity have been widely studied.

[0004] There are three main methods for generating domain-specific instruction data based on large language models. The first is to design common question types and prompt templates based on basic facts in a specific field, such as literature and textbooks, to prompt the large language model to use multiple templates and multiple factual information to generate a large number of question-answer pairs. The second is to design a diversification strategy based on existing question-answer pairs in a specific field. Without changing the original question-answer pair information, the large language model is prompted to produce diversified question-answer pairs by changing the wording, adding details, or re-structuring questions and answers. The third is based on the powerful generation ability of the large language model, by manually writing seed question-answer pairs and designing prompt word templates, prompting the model to randomly expand based on the seed question-answer pairs. The seed question-answer pairs should cover key questions and answers in the field to ensure the basic quality of the generated dialogue. The prompt word template can include question types, contextual prompts, and specific instructions, thereby guiding the model to generate diversified dialogue content related to the seed question-answer pairs.

[0005] In the medical field, most methods based on large language models to generate domain-specific instruction data often fail to fully consider the accuracy, diversity, and human preferences of the generated instruction data. Although large language models can generate seemingly reasonable answers using extensive medical knowledge, due to the lack of real-time supervision and proofreading by domain experts, the generated content may contain factual errors or misleading information. This is particularly dangerous in the medical field and may have a negative impact on patient health and weaken the practical application value of the model. In addition, large language models usually rely on fixed templates and limited input information, which makes the generated question-answer pairs lack variation and innovation in expression and cannot cover the complex and changing scenarios in clinical practice. Limited diversity not only affects the generalization ability of the model, but may also cause the model to perform poorly when facing diverse problems in the real world. At the same time, although the question-answer pairs generated by large language models are reasonable in language structure, they may not conform to the communication habits and preferences of doctors and patients in actual conversations. For example, doctors may use more understandable language when answering patients' questions, or provide personalized advice based on the patient's specific situation. However, these human details are often difficult to fully reproduce through model generation. These limitations lead to a large gap between synthetic data and real data, resulting in poor performance of the trained model, which seriously restricts the widespread application and development of large language models in the medical field. Summary of the invention

[0006] In view of the shortcomings of the existing technology, the present invention provides a method for generating diversified instruction data in the medical field based on a large language model, aiming to solve the limitations of the existing technology in data accuracy, diversity and expression of human preferences.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A method for generating diversified instruction data in the medical field based on a large language model comprises the following steps:

[0009] Step 1: Use natural language processing (NLP) tools to preprocess medical documents, including but not limited to text standardization, format unification, and information integrity verification. Use a text segmentation model based on sequence modeling to intelligently split medical documents, decompose long documents into small paragraphs with clear logic and complete information, and carefully design prompt words and data samples to guide large language models to accurately judge the potential for question-answer pair generation of medical document slices, laying a solid foundation for subsequent keyword extraction and question-answer generation.

[0010] Step 2: Use the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm, which combines term frequency and inverse document frequency, to measure the importance of a word in a document, effectively filtering out words that appear commonly in all documents but have low information content, while highlighting words that appear frequently in specific documents, thereby extracting the most informative and unique keywords in medical documents.

[0011] Step 3: An advanced cue word evolution strategy is designed, which not only combines cue word templates to ensure the diversity and richness of questions, but also incorporates operations such as few-shot design and hyperparameter adjustment to further improve the quality of question generation. Through few-shot design guided by a small number of examples, the language model can quickly understand the context of the task, while the fine tuning of hyperparameters ensures the accuracy and efficiency of the model in generating questions. Predefined templates are used to guide the model to generate various types of questions including information extraction, content summarization, calculation and reasoning, and few-shot examples and hyperparameters are continuously adjusted through iterative optimization and feedback loops to adapt to different medical documents and keywords, thus forming a closed-loop optimization process to ensure the adaptability and relevance of question generation.

[0012] Step 4: Use the large language model to strictly screen and filter the generated questions. First, we use the embedding model to encode each question to form a feature vector. Then, by calculating the cosine similarity between questions, we exclude those questions whose similarity exceeds the preset threshold to enhance the diversity of questions. At the same time, we conduct a detailed review of the questions to exclude questions with unclear reference, missing subjects, unclear expressions, and ambiguous meanings, ensuring that the questions that are retained have clear language expression, rigorous logical structure, and clear semantics. This series of operations not only improves the comprehension and accuracy of the questions, but also effectively increases the diversity of questions through similarity screening, laying a solid foundation for the subsequent answer generation and verification links.

[0013] Step 5: Create a standardized prompt word template and use it to prompt the model to generate answers based on the content of the specific document chunk, ensuring that the model can generate accurate, relevant, and high-quality answers based on the document content, thereby improving the model's answering ability.

[0014] Step 6: Strictly filter the generated answers to ensure their accuracy and relevance, thereby improving the authenticity and professionalism of the question-answering data. First, we use natural language inference (NLI) technology to perform attribution filtering to determine whether the answer is correctly based on the document content. This process compares the semantic consistency of the answer and the document content to ensure that the generated answer is indeed from the provided document and not fabricated out of thin air, thereby filtering out answers that do not conform to the document information. Secondly, we perform rejection filtering on the answers to check whether the answers contain sentences with the same meaning as "cannot answer", such as "cannot answer", which indicates that the model failed to provide a valid answer. By filtering rejection sentences, we retain valid answers that clearly answer the question, ensuring the practicality and informativeness of the answers.

[0015] Further, the process of step 1 is as follows:

[0016] Step 1.1: Use regular expressions to remove garbled characters, emoticons (non-text content), extra blank lines, etc. in the target medical document. This can be done by following the steps below:

[0017] (1) Identify the problem: Identify specific problems in the document, such as garbled characters, emoticons, and extra blank lines.

[0018] (2) Design rules: Design corresponding regular expression rules based on the identified problems. For example:

[0019] Garbled characters: may appear as non-standard characters or unrecognizable characters.

[0020] Emojis: Usually Unicode characters, you can search for specific Unicode ranges.

[0021] ·Extra blank lines: consecutive blank lines or more than a certain number of blank characters.

[0022] (3) Apply rules: Use the regular expression library in the programming language to apply the designed rules to the document text.

[0023] (4) Test and adjust: Test the document to ensure that the rules effectively remove unnecessary content while retaining the required information. Adjust the rules based on the test results.

[0024] (5) Final processing: Apply the final regular expression rules to process the entire document.

[0025] By designing and applying appropriate regular expressions, garbled characters, emojis, and extra blank lines in medical documents can be effectively removed. This process ensures the neatness and readability of the text, laying a solid foundation for subsequent analysis and processing.

[0026] Step 1.2: Intelligently slice the medical document based on the nlp_bert_document-segmentation_chinese-base model. Specifically, it includes the following steps:

[0027] (1) Each sentence in the medical document is tokenized by the WordPiece tokenizer.

[0028] (2) These sentences are encoded through an input representation layer consisting of "token embeddings, position embeddings, and segment embeddings".

[0029] (3) These embeddings are fed into the Transformer encoder, and the hidden states corresponding to k tokens are output. And the mean pooling method is used to obtain the sentence encoding.

[0030] (4) All sentences of the sentence encodings are input into a softmax binary classifier to determine whether each sentence is a paragraph boundary, minimizing the softmax cross-entropy loss, ensuring that the model can effectively reorganize the long text content into well-structured paragraphs, improving the readability and processing efficiency of the text.

[0031] Furthermore, the process of Step 2 is as follows:

[0032] (1) Preprocess the document, which includes steps such as tokenization, stop word removal, and lemmatization. Tokenization is to split the text into independent words or tokens, which can convert continuous text into a more easily processed form; stop word removal is to remove those common but low-information words, such as "de", "shi", etc., to reduce noise and improve the analysis accuracy; lemmatization reduces words to their basic or dictionary form, which helps to unify different forms of words and ensure the consistency of processing. Through these preprocessing steps, the accuracy and effectiveness of subsequent TF-IDF algorithms for keyword extraction can be significantly improved.

[0033] (2) Calculate the term frequency (TF). The term frequency refers to the frequency of a certain word in the document. Its formula is: where t represents the word and d represents the document.

[0034] (3) Calculate the Inverse Document Frequency (IDF). The IDF is a measure of the prevalence of a word in all documents. Its formula is: Where t represents a word and D represents a document set

[0035] (4) Calculate the TF-IDF value. TF-IDF is the product of term frequency and inverse document frequency, which is used to measure the importance of a term in a specific document. Its formula is: TF-IDF(t,d,D) = TF(t,d) × IDF(t,D)

[0036] The TF-IDF algorithm can effectively filter out words that are common in medical documents but not very informative, while highlighting words that appear frequently in medical documents, thereby extracting the most informative and unique keywords in medical documents. The higher the TF-IDF value, the greater the importance of the word in the document. This method is widely used in fields such as text analysis, information retrieval, and natural language processing.

[0037] Further, the process of step 3 is as follows:

[0038] Step 3.1: The content generated by LLM is affected by itself, and hyperparameters such as temperature, top-k, and top-p are crucial. In order to maximize the diversity of the dataset, these hyperparameters are adjusted, especially the temperatur setting.

[0039] Temperature: Adjust the temperature parameter to control the randomness of the generated content. Set a higher temperature parameter (such as 0.7-1.0) to increase the randomness and diversity of the generated questions.

[0040] Top-k sampling: Set the top-k parameter to sample only the k words with the highest probability, thus controlling the diversity of generated content.

[0041] Top-p (core) sampling: Set the top-p parameter to select words with cumulative probability exceeding p for sampling, thereby enhancing the diversity of generated content.

[0042] Duplicate penalty: Adjust duplicate penalty parameters to avoid generating duplicate content and increase question diversity.

[0043] Step 3.2: When performing Few-Shot design and sample selection, the main steps include data encoding, cluster analysis, sample selection, and construction of Few-Shot prompts. This process can be summarized into the following key steps.

[0044] Embedding encoding: Use the Embedding model to encode all data items and create an embedding list E = {e1, e2, ...., e m}, where m is the total number of examples, each e i is an embedding vector.

[0045] Clustering algorithm application: Use K-means clustering algorithm to cluster the embedding list. The goal of K-means algorithm is to divide the data points into n clusters {C1, C2, C3...C n}, so that the sum of the distances between each data point and the center point of the cluster to which it belongs is minimized.

[0046] Sample selection: From each cluster C j Randomly select an example from , generating a set of n different examples that represent different categories of questions, which helps the model learn diverse question generation methods.

[0047] Few-Shot Hint: During actual question generation, we provide these few-shot example questions and their answers as input to the model. This helps the model better understand the task requirements and generate questions that are similar to the examples but diverse.

[0048] Step 3.3: Design prompt evolution strategies to classify prompt words into different types. Design multiple prompt word templates for each type of question. Three evolution strategies are designed, as shown below:

[0049] Deep Evolution Strategy

[0050] Reasoning Evolutionary Strategies

[0051] Specific evolution strategy

[0052] In real doctor-patient dialogue scenarios, patients usually ask in-depth, more specific, and more complex questions about their own symptoms, rather than just asking questions such as "What medicine can be used to treat xx disease?", "What are the symptoms of xx disease?", etc. We imitate this process and further design specific prompt words to make the given instructions more complex. For example, we can diversify the simple questions mentioned above into "I feel very uncomfortable recently. I think I may have xx disease. Please recommend medicine for me!", "I have xx symptoms recently. I suspect it is xx disease. What do you think?". With this discovery, we can iteratively evolve the initial instruction dataset, increase its difficulty level, and expand its richness and diversity.

[0053] Further, the process of step 4 is as follows:

[0054] Step 4.1: Filter questions based on similarity to increase question diversity

[0055] Question encoding: Use the Embedding model to encode each generated question q i Encode and get its feature vector e i

[0056] Calculate similarity: Use the cosine similarity formula to calculate the similarity between any two questions q i and q j Similarity between

[0057] Screening questions: Set a similarity threshold θ, and keep only one question with a similarity greater than θ, thereby excluding questions with high similarity and improving diversity

[0058] Step 4.2: Design prompts to prompt LLM to conduct a detailed review of the questions and eliminate problems such as unclear references. The main checks are as follows:

[0059] Reference check: Make sure each question has a clear reference, i.e. there are no ambiguous questions.

[0060] Language clarity check: Make sure the question is clearly expressed and structured.

[0061] Semantic clarity check: Make sure the question is semantically clear and there is no ambiguity or multiple interpretations.

[0062] Further, the process of step 5 is as follows:

[0063] Design structured templates to guide the model to deeply understand the key information of the document, thereby ensuring that the generated answers are not only accurate, relevant, but also of high quality. In this process, the template acts as a bridge, closely connecting specific parts of the document with the expected form of the answer and providing the model with the necessary contextual information. In this way, the model can learn how to identify important elements in the document and transform them into clear and coherent answers.

[0064] Further, the process of step 6 is as follows:

[0065] Step 6.1: NLI natural language inference, based on the currently open source NLI dataset, fine-tune the qwen_1.8b_chat model. Through this fine-tuning, the output of the model can be standardized so that it can determine whether the answer is truly derived from the document content. If the answer is truly derived from the document content, then "entailment" is output; if the answer is irrelevant to the document content or contradictory, then "neutral" or "contradiction" is output. This process ensures that when the model generates answers, it can accurately identify and exclude answers that are inconsistent with the document content, thereby improving the accuracy and reliability of the answers.

[0066] Step 6.2: Design prompt words to determine whether the answer contains words with meanings such as "cannot be determined" or "cannot be known" based on the large language model. By setting these prompt words for the model, answers containing such meanings can be filtered. Specifically, when the model generates an answer, if it detects that the answer contains expressions such as "cannot be determined" or "cannot be known", it considers the answer to be unqualified and filters it out. This strategy further ensures that the answers output by the model are not only accurate and relevant, but also exclude those ambiguous or uncertain answers, thereby improving the overall answer quality.

[0067] The beneficial effects of adopting the above technical solution are:

[0068] The present invention can ensure the high quality and credibility of data while improving the richness and practicality of data sets through a complete set of data production technical solutions including document verification, keyword extraction, question generation, question verification, answer generation, answer verification and question diversification. Document verification ensures the validity of input documents, keyword extraction identifies the core information in documents, question generation guides the model to generate diversified questions, question and answer verification ensures their accuracy and relevance, and question diversification enhances the comprehensiveness of data sets. These steps work together to significantly improve the performance of large language models in processing complex medical documents and diversified questions, providing strong support for intelligent applications in the medical field. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flow chart of the present invention; DETAILED DESCRIPTION

[0070] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but the implementation of the present invention is not limited thereto.

[0071] Embodiment 1:

[0072] Combined with Figure 1 As shown, a method for generating diversified instruction data in the medical field based on a large language model includes:

[0073] Step 1: Use natural language processing (NLP) tools to preprocess medical documents, including text standardization, format unification, and information integrity verification. Use a text segmentation model based on sequence modeling to split long documents into small paragraphs with clear logic and complete information, and design prompt words and data samples to guide the large language model to judge the potential of document slices for question-answer pair generation, laying the foundation for keyword extraction and question-answer generation.

[0074] Step 2: Use the TF-IDF algorithm to combine word frequency and inverse document frequency to measure the importance of words in the document. Filter out commonly occurring words that are not very informative, highlight words that appear frequently in specific documents, and extract the most informative and unique keywords.

[0075] Step 3: Design an advanced prompt word evolution strategy, combining prompt word templates, few-shot design, and hyperparameter adjustment to improve the quality of question generation. Through few-shot design guided by a small number of examples, the language model quickly understands the task context, predefined templates guide the generation of various types of questions, and adjusts examples and hyperparameters through iterative optimization and feedback loops to ensure the adaptability and relevance of question generation.

[0076] Step 4: Use the large language model to screen and filter the generated questions. First, embed the model to encode the feature vector, calculate the cosine similarity between the questions, and exclude questions with too high similarity. Then, carefully review the questions to exclude those with unclear reference, missing subject, unclear expression, and unclear meaning, and ensure that the questions retained are clearly expressed in language, rigorous in logical structure, and clear in semantics, so as to improve comprehension and accuracy, and increase question diversity.

[0077] Step 5: Create a standardized prompt word template and use it to prompt the model to generate answers based on the specific document content. This ensures that the answers are accurate, relevant, and high-quality, and improves the model's ability to answer questions.

[0078] Step 6: Strictly filter the generated answers to ensure their accuracy and relevance. First, use natural language inference (NLI) technology to perform attribution filtering to determine whether the answer is based on the document content and ensure that the generated answer comes from the provided document. Secondly, perform rejection filtering to check whether the answer contains statements such as "cannot answer" and retain valid answers that clearly answer the question to ensure the practicality and informativeness of the answer.

[0079] Embodiment 2:

[0080] Based on Example 1, Figure 1 As shown, the step 1 specifically includes:

[0081] Step 1.1: In the field of medical document processing, ensuring the accuracy and readability of text data is of vital importance. To this end, a standardized process was developed to remove garbled characters, emoticons, and redundant blank lines from documents. The process first involves comprehensive problem identification of documents to identify non-standard characters, emoticons, and non-compliant blank line patterns. Subsequently, a series of regular expression rules are developed based on the identified problems. These rules are specifically designed to identify and remove non-compliant character sets, emoticons within a specific Unicode range, and redundant white space in documents. In the programming implementation phase, regular expression libraries are used to develop automated scripts that are responsible for performing content replacement or deletion operations. Before applying the rules to the entire document, small-scale testing is necessary. This step ensures the accuracy of the regular expression and avoids the accidental deletion of important content in the document. The test results will be fed back to the rule adjustment phase for necessary optimization to improve the accuracy of matching and the efficiency of execution.

[0082] After the rules are optimized, they are applied to the entire document to perform a comprehensive cleanup. After the cleanup is complete, the document is reviewed again to verify the cleanup effect and ensure that the document's readability and information integrity are maintained. The cleaned document is then saved as a new file version to maintain the integrity of the original data.

[0083] Step 1.2: The process of intelligent slicing of medical documents based on the nlp_bert_document-segmentation_chinese-base model involves a series of refined natural language processing techniques.

[0084] First, each sentence in the document is accurately segmented by the WordPiece tokenizer, which is a word segmentation method based on lexical statistics that can decompose sentences into sub-units that are easier for the model to process.

[0085] Subsequently, the representation of the sentence is encoded through a comprehensive embedding layer, which consists of token embedding, position embedding, and paragraph embedding. Token embedding maps each token after word segmentation to a vector in a high-dimensional space; position embedding provides the model with information about the position of each token in the sentence; and paragraph embedding further provides the model with paragraph-level contextual information. Among them, the i-th token embedding in the sentence represents: i =w i +p i +s i , w i Indicates token embedding, p i represents the position embedding, s i Represents a paragraph embedding.

[0086] Next, these embeddings are fed into the Transformer encoder, which uses a self-attention mechanism to encode each token in the sentence and outputs a hidden state corresponding to each token. These hidden states are then aggregated using the mean pooling method to obtain the encoding of the sentence. Mean pooling is achieved by calculating the average of all token hidden states. Among them C s is the encoding of the sentence, h i is the hidden state of the ith token, and N is the total number of tokens in the sentence.

[0087] Finally, the sentence encodings of all sentences are input into the softmax binary classifier to determine whether each sentence is a paragraph boundary. The training goal of the model is to minimize the softmax cross entropy loss function. S represents the total number of sentences, y s is the true label (0 or 1) of sentence s, is the label probability predicted by the model.

[0088] Through this process, the model can effectively reorganize long text content into well-structured paragraphs, thereby improving the readability and processing efficiency of the text. This not only enhances the structuring of the document, but also facilitates subsequent text analysis and information extraction tasks.

[0089] Embodiment 3:

[0090] Based on Example 2, Figure 1 As shown, the step 2 specifically includes: for text analysis of medical documents, the TF-IDF algorithm is applied to keyword extraction, and its process involves a series of detailed preprocessing steps and calculation methods.

[0091] First, the document needs to be preprocessed, which includes tokenization, stop word removal, and lemmatization. Tokenization is the process of breaking down continuous text into individual words or tokens, which converts the text into a form that is easier for machines to process. Next, stop words, which are words that appear frequently in the text but have low information value, such as particles and articles, are removed to reduce data noise and improve the accuracy of the analysis. Finally, lemmatization converts the words in the text into their base or dictionary form, ensuring that different forms of the same word are treated uniformly, enhancing the consistency of the process.

[0092] Based on the preprocessing, the next step is to calculate the term frequency (TF), which is an indicator that measures the frequency of a word in a specific document. Subsequently, Inverse Document Frequency (IDF) was introduced to measure the rarity of a word in a collection of all documents. Finally, the TF-IDF value is obtained by multiplying TF and IDF, TF-IDF(t,d,D)=TF(t,d)×IDF(t,D), which quantifies the importance of a term in a specific document.

[0093] The TF-IDF algorithm effectively identifies and filters out common words that appear widely in medical document collections but do not contain much information by calculating the TF-IDF value of each word, while highlighting words that appear frequently in specific documents and are relatively rare in the document collection. This method not only enhances the quality of keyword extraction, but also provides a powerful tool for fields such as text analysis, information retrieval, and natural language processing. The importance of a word in a document can be judged by the TF-IDF value, providing a basis for in-depth analysis and knowledge discovery of medical documents.

[0094] Embodiment 4:

[0095] Based on Example 3, Figure 1 As shown, the step 3 specifically includes:

[0096] Step 3.1: The Temperature parameter controls the randomness of the generated content; higher temperature settings (such as 0.7-1.0) increase the randomness and diversity of the generated questions. Top-k sampling controls the concentration of the generated content by selecting from the k words with the highest probability. Top-p (nucleus) sampling further enhances the diversity of the generated content by selecting words with cumulative probability exceeding the p threshold for sampling. In addition, adjusting the duplicate penalty parameter helps avoid generating duplicate content and further improves the diversity of questions.

[0097] Step 3.2: In terms of Few-Shot design and example selection, all data items are first encoded using the Embedding model to form an embedding list. Subsequently, the K-means clustering algorithm is applied to cluster the embedding list, with the goal of dividing the data points into n clusters and minimizing the sum of the distances from each data point to the center point of the cluster to which it belongs. An example is randomly selected from each cluster to form a set of diverse examples that help the model learn how to generate different categories of questions. When generating actual questions, these few-shot example questions and their answers are provided as input to the model to help the model better understand the task requirements and generate questions that are similar to the examples but diverse.

[0098] Step 3.3: When designing prompt evolution strategies, divide prompt words into different types and design multiple prompt word templates for each type. Through deep evolution strategies, reasoning evolution strategies, and specific evolution strategies, the initial instruction data set can be iteratively evolved to increase its difficulty level and expand its richness and diversity. In real doctor-patient dialogue scenarios, the questions asked by patients are often deep, specific, and complex. By imitating this process, specific prompt words can be designed to make a given instruction more complex, such as evolving from a simple "What medicine can be used to treat xx disease?" to "I feel very uncomfortable recently. I think I may have xx disease. Please recommend medicine for me!" or "I have xx symptoms recently. I suspect it is xx disease. What do you think?".

[0099] Embodiment 5:

[0100] Based on Example 4, Figure 1 As shown, the step 4 specifically includes:

[0101] Step 4.1: Apply the Embedding model to encode each generated question and convert it into a feature vector, which provides a numerical representation for the subsequent similarity calculation. The cosine similarity formula is used to quantify the similarity between any two questions. Cosine similarity evaluates their similarity by measuring the cosine value of the angle between two vectors. Its value range is [-1,1]. The closer the value is to 1, the more similar the questions are. The calculation formula is: Among them, q i and q j Represent the feature vectors of the two questions respectively, · represents the dot product, ||·|| represents the Euclidean normal form, and a threshold θ is set to filter questions based on the calculated similarity. If the similarity between two questions exceeds this threshold, only one of them is retained to exclude highly similar questions and ensure the diversity of the question set.

[0102] Step 4.2: When designing prompts, each question needs to be carefully reviewed to eliminate unclear references, unclear language, or ambiguous semantics. This includes ensuring that each question has a clear reference object to avoid unclear references; checking that the language of the question is clear to avoid confusing structures; and ensuring that the semantics of the question are clear and there is no possibility of ambiguity or multiple interpretations.

[0103] Embodiment 6:

[0104] Based on Example 5, Figure 1As shown, step 5 specifically includes: designing a structured prompt template to guide the large language model to deeply understand the key information of the medical document and generate accurate, relevant and high-quality answers. This method involves creating templates that can capture the core concepts and relationships of the document. These templates contain clear instructions and logical structures to ensure that the model can focus on relevant information and generate coherent answers. By combining the template with specific parts of the document, the model can extract key information and transform it into clear and organized answers. Ultimately, the use of templates provides a systematic way for the model to generate answers that meet high accuracy and professionalism.

[0105] Embodiment 7:

[0106] Based on Example 6, Figure 1 As shown, the step 6 specifically includes:

[0107] Step 6.1: Fine-tune the qwen_1.8b_chat model on the open source Natural Language Inference (NLI) dataset to enhance its ability to judge the consistency of the generated answer with the document content. The fine-tuning process involves fine-tuning the weights and biases of the model so that the model can distinguish the logical relationship between the answer and the document content. Specifically, the model learns to classify the output as "entailment", "neutral", or "contradiction", corresponding to the situation where the answer actually comes from the document, is irrelevant to the document, or contradicts the document content, where the model predicts that the relationship between answer A and document D is R: R = NLIMODEL(A,D), and possible R values ​​include "entailment", "neutral", or "contradiction"

[0108] Step 6.2: In order to further improve the quality of answers, a series of prompt words are designed to guide the model to identify and filter out answers containing uncertainty words. This strategy is based on a predefined set of uncertainty words, such as "cannot be determined" and "cannot be known". In the process of the model generating answers, if these words appear in the answer, the model will identify them as unqualified answers based on the guidance of the prompt words and filter them out. This filtering mechanism can be represented by the following pseudo code: Filter is a Boolean value. If the answer A contains any word w in the uncertain word set UncertainWords, Filter is true, indicating that the answer needs to be filtered.

[0109] In order to verify the effectiveness of the solution of the present invention, further explanation is given below in combination with experimental data:

[0110] The present invention uses public disease encyclopedia information captured from the medical consultation website Xunyiwenyao. Data collection is completed through automated web crawler technology, which ensures the breadth and efficiency of data collection. The collected data set contains 8808 Chinese disease encyclopedia information texts, which provides a rich corpus for evaluating the proposed method. Based on the crawled Chinese disease encyclopedia information texts, the data set is uniformly preprocessed, and on this basis, the commonly used data generation method SELF-QA is compared with the method of the present invention. The two methods are evaluated for performance under the same evaluation indicators, including diversity and human preferences. Through statistical analysis, the performance differences between the two methods are compared, and the statistical significance of these differences is explored. The scale of the data set generated by the two methods is shown in Table 1.

[0111] Table 1 Dataset size

[0112] Number of documents Number of document slices Number of question and answer pairs SELF-QA 8808 15792 34551 The present invention 8808 15792 24094

[0113] In order to evaluate the diversity of questions generated by the two methods, we can use multiple data diversity evaluation indicators: BLEU-1, BLEU-2, BLEU-3, BLEU-4, GLEU, ROUGE-1, ROUGE-2 and ROUGE. Each indicator evaluates the quality and diversity of generated text from different perspectives. The calculation formula is as follows:

[0114] 1. BLEU score: where p n is the exact match of the n-gram, w n is the weight, BP is the penalty factor, which takes into account the ratio of the length of the generated text to the length of the reference text.

[0115] 2. GLEU score: Where n is the number of n-grams in the generated text, and m is the number of n-grams in the reference text.

[0116] 3. ROUGLE-N score: Where Countmatch(gramn) is the number of matches of n-grams in the reference text and the generated text, and RefCount(gramn) is the total number of n-grams in the reference text.

[0117] 100 texts were randomly selected from Chinese disease encyclopedia information texts, and the question-answer pairs generated by the method of the present invention and other methods based on these texts were comprehensively evaluated, and the diversity and complexity of the questions were compared. The experimental results are shown in Table 2. The results show that the present invention is superior to other comparison methods and can generate more complex and diverse questions in the medical field.

[0118] Table 2 Comparison of the diversity of problems generated by the two methods

[0119] SELF-QA The present invention BLEU-1 0.2150 0.2129 BLEU-2 0.0441 0.0326 BLEU-3 0.0223 0.0098 BLEU-4 0.0048 0.0060 GLEU 0.0769 0.0678 ROUGE-1 0.2325 0.2082 ROUGE-2 0.0369 0.0201 ROUGE 0.2201 0.1859 AvgLength 19.85 29.45

[0120] In order to prove that the present invention is more in line with human preferences, a dual method of GPT-4 automatic scoring and manual evaluation was used to evaluate the question-answer pairs generated by the two different data production methods. 100 Chinese disease encyclopedia information texts were randomly selected. By comprehensively evaluating the question-answer pairs generated by the two methods based on these texts, we were able to comprehensively analyze the quality of the question-answer pairs from both automated and manual dimensions. The scoring rules are as follows:

[0121] 1. Relevance

[0122] Scoring criteria: The relevance of the question and answer in the question-answer pair. Whether the question and answer content match, and whether the answer directly responds to the question.

[0123] 1 point: The answer is completely irrelevant to the question.

[0124] 10 points: The answer fully addresses the question and provides detailed information.

[0125] 2. Accuracy

[0126] Scoring criteria: Whether the information in the answer is accurate and whether it is based on reliable data or facts.

[0127] 1 point: The answer contains false or misleading information.

[0128] 10 points: The answer is completely accurate and based on solid data and facts.

[0129] 3. Completeness

[0130] Scoring criteria: Whether the answer fully answers the question and does not omit key information.

[0131] 1 point: The answer omits a significant amount of key information and does not fully answer the question.

[0132] 10 points: The answer is comprehensive and addresses all parts of the question in detail.

[0133] The experimental results show that the question-answer pairs generated by the present invention perform better in satisfying human preferences, proving its effectiveness and superiority in practical applications. This evaluation not only enhances the user experience of the product, but also provides valuable data support for future technical improvements and research directions.

[0134] Table 3 Experimental comparison of the dual method of GPT-4 and manual evaluation to evaluate two different data production methods

[0135]

[0136] The embodiments of the present invention are described in detail above, and the above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Those skilled in the art can design many other modifications and implementation methods based on the present invention, and these modifications and implementation methods will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A method for generating diversified instruction data in the medical field based on a large language model, characterized in that: The following steps are involved: Step 1: Use natural language processing (NLP) tools to preprocess medical documents, including text standardization, format unification, and information integrity verification; use a text segmentation model based on sequence modeling to split long documents into small paragraphs with clear logic and complete information, and design prompt words and data samples to guide the large language model to determine the potential of document slices for question-answer pair generation, laying the foundation for keyword extraction and question-answer generation; Step 2: Use the TF-IDF algorithm to combine word frequency and inverse document frequency to measure the importance of words in the document; Filter out commonly occurring but low-information words, highlight frequently occurring words in specific documents, and extract the most informative and unique keywords; Step 3: Design an advanced prompt word evolution strategy, combining prompt word templates, few-shot design, and hyperparameter adjustment to improve the quality of question generation; Through a few-shot design guided by a small number of examples, the language model understands the task context, predefined templates guide the generation of various types of questions, and examples and hyperparameters are adjusted through iterative optimization and feedback loops to ensure the adaptability and relevance of question generation; Step 4: Use the large language model to screen and filter the generated questions. First, encode the questions using the embedding model to form feature vectors, calculate the cosine similarity between questions, and exclude questions with too high similarity. Then, review the questions to exclude questions with unclear reference, missing subject, unclear expression, and ambiguous meaning. Ensure that the questions retained have clear language expression, rigorous logical structure, and clear semantics, improve comprehension and accuracy, and increase question diversity. Step 5: Create a standardized prompt word template and use it to generate answers based on the specific document content to ensure that the answers are accurate, relevant, and high-quality, thus improving the model's ability to answer questions. Step 6: Strictly filter the generated answers to ensure their accuracy and relevance. First, use natural language inference (NLI) technology to perform attribution filtering to determine whether the answer is based on the document content and ensure that the generated answer comes from the provided document. Secondly, filter out rejected answers, check whether the answers contain "cannot answer" statements, retain valid answers that clearly answer the questions, and ensure the practicality and informativeness of the answers.

2. A method for generating diversified instruction data in the medical field based on a large language model according to claim 1, characterized in that: The step 1 specifically includes: Step 1.1: Developed a standardized process to remove garbled characters, emoticons, and extra blank lines in documents; the standardized process includes comprehensive identification of non-standard characters, emoticons, and non-compliant blank line patterns in documents, formulation of regular expression rules for identification and cleaning, development of automated scripts to perform replacement or deletion operations, and verification of the accuracy and effectiveness of the rules through small-scale testing; after testing and optimization, these rules were applied to the entire document for comprehensive cleaning, and finally the cleaning effect was reviewed to ensure the readability and information integrity of the document, and saved as a new file version to maintain the original data; Step 1.2: Intelligently slice the medical document based on the nlp_bert_document-segmentation_chinese-base model, segment each sentence in the document through the WordPiece tokenizer, decompose the sentence into more manageable sub-units, encode the sentence through the comprehensive embedding layer of token embedding, position embedding and paragraph embedding, and feed it into the Transformer encoder, encode each token using the self-attention mechanism, and output the hidden state, where the i-th token in the sentence is embedded as: e i =w i +p i +s i , w i Indicates token embedding, p i represents the position embedding, s i Represents paragraph embedding; the hidden states are aggregated through the mean pooling method to obtain the encoding of the sentence, and the average value of all token hidden states is calculated; the encoding of all sentences is input into the softmax binary classifier to determine whether each sentence is a paragraph boundary; the model reorganizes long text content into well-structured paragraphs, improves the readability and processing efficiency of the text, and facilitates subsequent text analysis and information extraction tasks.

3. The method for generating diversified instruction data in the medical field based on a large language model according to claim 1, characterized in that: Through the preprocessing steps of the TF-IDF algorithm, such as word segmentation, stop word removal, and word form restoration, medical text is converted into a form that is easy for machine processing; by calculating the term frequency TF and the inverse document frequency IDF, the importance of words in a specific document is quantified, and the TF-IDF value is obtained by multiplying the two, that is, TF-IDF(t,d,D)=TF(t,d)×IDF(t,D).

4. The method for generating diversified instruction data in the medical field based on a large language model according to claim 3, characterized in that: The step 3 specifically includes: Step 3.1: Adjust the Temperature parameters of the large language model, Top-k sampling, and Top-p sampling to improve the randomness and diversity of medical instruction data generation; adjust the Temperature parameters to increase the uncertainty and diversity of generated content; Top-k sampling controls the concentration of content by limiting the selection to the k words with the highest probability; Top-p sampling enhances the diversity of content by selecting words whose cumulative probability exceeds a specific threshold; Step 3.2: Few-shot can affect the content generated by the model. Designing a few-shot selection strategy is conducive to improving the diversity of medical instruction data. When selecting few-shot, first encode the data items through the Embedding model to obtain the embedding list E = {e1, e2, ...., e m }, where m is the total number of examples, each e i is an embedding vector; secondly, the K-means clustering algorithm is used to cluster E and divide the data points into n clusters. The objective function is to minimize the sum of the Euclidean distances from all data points to the center of their clusters, that is, in is the data point e i The center point of the cluster to which it belongs, d is the distance metric function; a representative example is randomly selected from each cluster to build a diverse set of examples; Step 3.3: When designing the prompt evolution strategy, the prompt words are divided into different types, and multiple prompt word templates are designed for each type of question, so as to improve the diversity and depth of medical question generation through different evolution directions; specifically, three main evolution strategies are designed: deep evolution strategy, reasoning evolution strategy and specific evolution strategy; the deep evolution strategy focuses on exploring the deep content of the problem, and improves the depth of the problem by guiding the model to generate more complex and detailed questions; the reasoning evolution strategy focuses on the exploration of logical reasoning and causal relationships, and designs prompt word templates to guide the model to generate questions that require reasoning and analytical capabilities; the specific evolution strategy aims to generate specific and clear questions, and helps the model generate more targeted and specific questions through detailed prompt word templates.

5. The method for generating diversified instruction data in the medical field based on a large language model according to claim 1, characterized in that: Described step 4 specifically comprises: Step 4.1: To further improve the diversity of medical question generation, a similarity screening method can be used. First, use the Embedding model to encode each generated question and obtain its feature vector. Secondly, the cosine similarity formula is used to calculate the similarity between any two questions. The formula is as follows: where e i 、e j Represent the feature vectors of problems i and j respectively; Finally, a similarity threshold θ is set. For any pair of questions whose similarity exceeds the threshold, only one of them is retained, thereby excluding questions with high similarity; Step 4.2: In order to improve the quality of generated medical questions, prompt words are designed to guide the large language model to conduct a detailed review of the generated questions to eliminate problems of unclear reference, unclear language expression, and unclear semantics; first, a reference check is performed to ensure that each question has a clear reference object and there is no unclear reference; second, a language expression clarity check is performed to ensure that the language expression of the question is clear, the structure is not chaotic, and the use of vague expressions is avoided; finally, a semantic clarity check is performed to ensure that the semantics of the question is clear and there is no ambiguity or multiple understandings.

6. The method for generating diversified instruction data in the medical field based on a large language model according to claim 1, characterized in that: Design structured templates to guide the model to deeply understand the key information of medical documents, thereby ensuring accurate, relevant and high-quality answers.

7. The method for generating diversified instruction data in the medical field based on a large language model according to claim 1, characterized in that: The step 6 specifically includes: Step 6.1: In order to check the accuracy and reliability of the answers generated by the model, the qwen_1.8b_chat model is fine-tuned based on the existing natural language inference NLI dataset to standardize the model's output so that it can determine whether the answer is truly derived from the document content; during the fine-tuning process, the model will learn three labels: if the answer is truly derived from the document content, it will output "entailment"; if the answer is irrelevant to the document content or contradicts each other, it will output "neutral" or "contradiction"; Step 6.2: Design prompt words to determine whether the answer contains words with the meanings of "cannot be determined" or "cannot be known" based on the large language model; filter out answers containing such meanings by setting these prompt words for the model; when generating answers, if it is detected that the answer contains expressions such as "cannot be determined" or "cannot be known", the answer is considered to be unsatisfactory and is filtered out.

Citation Information

Cited By

  • Medical text structured processing method based on Prompt technology

    CN120015344A

  • Medical file conversion method, device and equipment

    CN120316083A

  • Industrial instruction data construction method and system based on large model

    CN120409690A

  • Fine tuning data set construction method for large question and answer model in aerospace control software field

    CN121029926A

  • Method and device for evaluating speech expression ability of neurological and mental diseases

    CN121483566A