A traditional Chinese medicine question-answering method, device, equipment and medium based on a large language model

Abstract

The present application discloses a method, device, equipment and medium for Chinese medicine question and answer based on a large language model, and relates to the field of medical question and answer. The method obtains multi-source Chinese medicine knowledge data from books and documents, network platforms and Chinese medicine data sets, constructs diversified instruction data suitable for different scenarios, overcomes the defect that non-professional or inaccurate information may be introduced through the ChatGPT API collection, and affects the performance and reliability of the model. The model for Chinese medicine question and answer is obtained by training based on the Baichuan2‑7B‑Chat model, and the process from PT to SFT is realized. Different verification indicators are set according to different applicable scenarios, and the model is verified, overcoming the defect that the single or subjective evaluation indicator limits the accuracy of model evaluation. The present application improves the accuracy and reliability of Chinese medicine question and answer.

Description

Technical Field

[0001] The present application relates to the field of medical question-answering, and in particular to a method, device, equipment and medium for answering questions about traditional Chinese medicine based on a large language model. Background Art

[0002] Large Language Models (LLMs) are deep learning models trained with large amounts of text data, designed to understand and generate natural language text. These models are usually based on the Transformer architecture, which can capture the complexity and diversity of language. They are increasingly used in the field of biomedicine, especially in the field of traditional Chinese medicine. These models have brought many innovations to the research, application and dissemination of traditional Chinese medicine by understanding and generating natural language text. At present, the most common application is to use large models to analyze massive amounts of text data such as traditional Chinese medicine literature, ancient books, and modern research papers, and extract key information such as the name, nature, flavor, meridian, efficacy, dosage, and compatibility of traditional Chinese medicine from them to build a knowledge base of traditional Chinese medicine. Recently, some researchers have developed a large model called CMLM-ZhongJing for traditional Chinese medicine knowledge question and answer and auxiliary diagnosis and treatment. The large model is based on the data of the table of traditional Chinese medicine gynecology prescriptions, and generates instruction data for 15 scenarios by setting a specific prompt template, and finally fine-tuning the large model. However, the training data usually relies on ChatGPT API (Chat Application Learning Platform) to collect and build an aligned dataset. Relying on the ChatGPT API may introduce non-professional or inaccurate information, affecting the performance and reliability of the model. In addition, the existing evaluation indicators of the TCM large model are too single, and most of them use subjective evaluation as a verification method, which seriously limits the evaluation accuracy of the TCM large model. Summary of the invention

[0003] The purpose of this application is to provide a method, device, equipment and medium for Chinese medicine question answering based on a large language model, which can improve the accuracy and reliability of Chinese medicine question answering.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a traditional Chinese medicine question-answering method based on a large language model, and the traditional Chinese medicine question-answering method based on a large language model comprises:

[0006] Constructing a traditional Chinese medicine knowledge database; the traditional Chinese medicine knowledge database includes multi-source traditional Chinese medicine knowledge data obtained from books and documents, network platforms and traditional Chinese medicine data sets;

[0007] Generate unsupervised data and instruction data based on the traditional Chinese medicine knowledge database; the applicable scenarios of the instruction data include traditional Chinese medicine knowledge base, multiple-choice questions, reading comprehension, entity extraction, medical case diagnosis and traditional Chinese medicine or prescription recommendation;

[0008] Constructing a Baichuan2-7B-Chat model; the Baichuan2-7B-Chat model is obtained by improving the Transformer decoder, wherein the improvement includes: replacing the LayerNorm layer in the Transformer decoder with the RMSNorm layer, replacing the absolute position encoding layer in the Transformer decoder with the rotation position encoding layer, and replacing the ReLu activation function in the Transformer decoder with the SwiGLU activation function;

[0009] Use unsupervised data to perform unsupervised pre-training on the Baichuan2-7B-Chat model to obtain a pre-trained Baichuan2-7B-Chat model;

[0010] Use unsupervised data and instruction data to perform supervised training on the pre-trained Baichuan2-7B-Chat model to obtain a trained Baichuan2-7B-Chat model;

[0011] Verification indicators are set for different applicable scenarios, and the trained Baichuan2-7B-Chat model is verified using the verification indicators;

[0012] Use the verified Baichuan2-7B-Chat model to conduct traditional Chinese medicine question-answering.

[0013] In a second aspect, the present application provides a traditional Chinese medicine question-answering device based on a large language model, the traditional Chinese medicine question-answering device comprising a backend and a frontend;

[0014] The front end is used to interact with customers;

[0015] The backend is used to perform question answering using the above-mentioned traditional Chinese medicine question answering method based on a large language model.

[0016] In a third aspect, a computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned traditional Chinese medicine question-answering method based on a large language model.

[0017] In a fourth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned traditional Chinese medicine question-answering method based on a large language model.

[0018] In a fifth aspect, a computer program product includes a computer program, which, when executed by a processor, implements the above-mentioned traditional Chinese medicine question-answering method based on a large language model.

[0019] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0020] The present application provides a method, device, equipment and medium for Chinese medicine question and answer based on a large language model. The present application obtains multi-source Chinese medicine knowledge data from books and documents, network platforms and Chinese medicine data sets, and constructs diversified instruction data suitable for different scenarios, overcoming the defect that non-professional or inaccurate information may be introduced by directly collecting through ChatGPT API (large model), affecting the performance and reliability of the model. Based on Baichuan2-7B-Chat (the base model of Baichuan 7B parameters), training is performed to obtain a model for Chinese medicine question and answer, realizing the process from pre-training to instruction fine-tuning, setting different verification indicators according to different applicable scenarios, and verifying the model, overcoming the defect that a single or subjective evaluation indicator limits the evaluation accuracy of the model. The present application improves the accuracy and reliability of Chinese medicine question and answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 This is a flow chart of a traditional Chinese medicine question-answering method based on a large language model in one embodiment of the present application;

[0023] Figure 2 This is a schematic diagram of the training process of a large language model in one embodiment of the present application;

[0024] Figure 3 This is a comparison result diagram of Example 1 in one embodiment of the present application;

[0025] Figure 4 This is a comparison result diagram of Example 2 in an embodiment of the present application;

[0026] Figure 5 This is a comparison result diagram of Example 3 in an embodiment of the present application;

[0027] Figure 6 This is a comparison result diagram of Example 4 in an embodiment of the present application;

[0028] Figure 7 This is a comparison result diagram of Example 5 in an embodiment of the present application;

[0029] Figure 8 This is a comparison result diagram of Example 6 in an embodiment of the present application;

[0030] Fig. 9 This is a comparison result diagram of Example 7 in an embodiment of the present application;

[0031] Fig.10 This is a comparison result diagram of Example 8 in an embodiment of the present application;

[0032] Fig.11 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0035] In an exemplary embodiment, Figure 1 As shown, a Chinese medicine question-answering method based on a large language model is provided, which is executed by a computer device, specifically, it can be executed by a computer device such as a terminal or a server alone, or it can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to a server as an example for explanation, including the following steps 101 to 107. Among them:

[0036] Step 101, constructing a traditional Chinese medicine knowledge database; the traditional Chinese medicine knowledge database includes multi-source traditional Chinese medicine knowledge data obtained from books and documents, network platforms and traditional Chinese medicine data sets.

[0037] Step 102, generating unsupervised data and instruction data based on the traditional Chinese medicine knowledge database; the applicable scenarios of the instruction data include traditional Chinese medicine knowledge base, multiple-choice questions, reading comprehension, entity extraction, medical case diagnosis and traditional Chinese medicine or prescription recommendation.

[0038] Step 103, constructing a Baichuan2-7B-Chat model; the Baichuan2-7B-Chat model is obtained by improving the Transformer decoder, and the improvement method includes: replacing the LayerNorm layer in the Transformer decoder with the RMSNorm layer, replacing the absolute position encoding layer in the Transformer decoder with the rotational position encoding layer, and replacing the ReLu activation function in the Transformer decoder with the SwigLU activation function.

[0039] Step 104, use unsupervised data to perform unsupervised pre-training on the Baichuan2-7B-Chat model to obtain the pre-trained Baichuan2-7B-Chat model.

[0040] Step 105, using unsupervised data and instruction data to perform supervised training on the pre-trained Baichuan2-7B-Chat model to obtain a trained Baichuan2-7B-Chat model.

[0041] Step 106, setting verification indicators for different applicable scenarios, and using the verification indicators to verify the trained Baichuan2-7B-Chat model.

[0042] Step 107, using the verified Baichuan2-7B-Chat model to conduct traditional Chinese medicine question and answer.

[0043] By implementing the above steps 101 to 107, multi-source TCM knowledge data is obtained from books and documents, online platforms and TCM data sets, and diversified instruction data suitable for different scenarios is constructed, which overcomes the defect that non-professional or inaccurate information may be introduced when directly collecting through the ChatGPT API (large model), affecting the performance and reliability of the model. Training is performed based on Baichuan2-7B-Chat (the base model of Baichuan 7B parameters) to obtain a model for TCM question and answer, realizing the process from pre-training to instruction fine-tuning, setting different verification indicators according to different applicable scenarios, and verifying the model, overcoming the defect that a single or subjective evaluation indicator limits the evaluation accuracy of the model. This application improves the accuracy and reliability of TCM question and answer.

[0044] The construction of a large model of traditional Chinese medicine relies heavily on the fine-tuning of instruction data. This process is mainly achieved by training and optimizing the model on a large number of data sets related to traditional Chinese medicine, which can enhance the model's understanding and application capabilities in the field of traditional Chinese medicine. However, although fine-tuning is an important step in the training of large language models (LLMs), model pre-training, high-quality instruction data construction, and multi-dimensional model evaluation also play an indispensable role. To this end, the embodiment of the present application uses a diversity of instruction data sets to train the base model in two stages from pre-training to fine-tuning, and performs a multi-dimensional evaluation to construct a TCMChat system. The main goal of the TCMChat system is to answer applications such as basic knowledge of traditional Chinese medicine, intelligent recommendations, and medical case diagnosis. Therefore, structured and unstructured data are collected from different sources. The data is organized into unsupervised data and supervised instruction data through certain strategies, and trained on a cluster server. The trained model is compared with other models to quantify the accuracy of the model under different test sets.

[0045] The construction of instruction data is an indispensable step before fine-tuning the model. Recently, a large number of literatures have reported methods for constructing a set of Chinese medicine instructions using raw data. For example, BenTso constructed an instruction data set based on the knowledge graph and then fine-tuned it on Chinese-LLaMA. CMLM-ZhongJing uses professional table data and strictly sets a specific prompt template to generate instruction data for 15 scenarios. The fine-tuned model has the reasoning ability of Chinese medicine prescription data and diagnostic thinking logic. TCMChat has adopted a more comprehensive and diversified strategy in collection. In addition to Chinese medicine books and open source data, it also introduces multi-source data such as literature knowledge, Web platform data, and databases, and trains these multi-source data in a full-parameter manner. Secondly, in order to more scientifically evaluate the performance of TCMChat, the embodiment of this application has established a set of multi-dimensional, objective and comprehensive evaluation systems. These indicators include but are not limited to basic indicators such as model accuracy, recall rate, F1 score, and also introduce advanced indicators such as MRR, BERTSocre, and nDCG. Through the comprehensive evaluation of these indicators, the actual value of the model can be more comprehensively reflected.

[0046] In another exemplary embodiment, the data sources for constructing the Chinese medicine knowledge database in step 101 above include: books and documents, network platforms and Chinese medicine data sets. The Chinese medicine knowledge database constructed in step 101 above widely covers multiple dimensions such as books and documents, network crawler information, professional literature and open source data sets, aiming to comprehensively and deeply explore the rich knowledge in the field of Chinese medicine. Specifically, it covers the following aspects:

[0047] Books and materials: Chinese medicine books are an important part of the field of Chinese medicine. They not only carry theoretical knowledge of Chinese medicine, but also record rich clinical practice experience and drug information. Including national standards, medical textbooks and medical records, these materials provide a solid foundation for the theory, practice and historical research of Chinese medicine.

[0048] Disease and syndrome information: The information sources of the China Medical Information Query Platform are extensive and authoritative, including Chinese medicine information released by authoritative institutions such as the State Administration of Traditional Chinese Medicine and the China Academy of Chinese Medical Sciences. The information on the platform has been strictly reviewed and certified to ensure the accuracy and reliability of the information. Through the China Medical Information Query Platform (TCM-DaYi), detailed disease and syndrome records can be obtained, which greatly enriches the database of clinical application of traditional Chinese medicine.

[0049] Herbal medicine and prescription data: The Encyclopedia of Traditional Chinese Medicine (ETCM) website is a comprehensive resource database of traditional Chinese medicine, which aims to integrate the pharmacological information of traditional Chinese medicine and provide the associated data of traditional Chinese medicine components and targets to help researchers understand the pharmacological mechanism of traditional Chinese medicine. Using the ETCM website, detailed data on herbs and prescriptions can be collected, providing valuable resources for the research of traditional Chinese medicine prescriptions and herbal medicine.

[0050] Academic Literature Abstracts: CNKI (China National Knowledge Infrastructure) is the largest comprehensive academic resource database in China. It integrates various types of academic resources such as journals, dissertations, conference papers, newspapers, yearbooks, patents, standards, scientific and technological achievements, etc., and provides convenient academic resource retrieval and download services for scholars, researchers, educators and students. Abstracts of literature on traditional Chinese medicine and related fields can be downloaded through CNKI, which provides important support for the deepening and expansion of traditional Chinese medicine theory.

[0051] Baidu Encyclopedia: Baidu Encyclopedia contains a wealth of information on traditional Chinese medicine, which covers the basic concepts, classification, origin, collection, processing, medicinal properties theory, compatibility, contraindications, dosage and usage, and naming of traditional Chinese medicine. Baidu Encyclopedia text data (BaiduBaiKe) was obtained from the GitHub platform. These data contain popular content on traditional Chinese medicine knowledge.

[0052] TCM reading comprehension and named entity recognition: The Ali Tianchi platform dataset is an open scientific research data platform of Alibaba Group, jointly provided by Alibaba Group business teams and external research institutions, covering more than ten industries such as e-commerce, entertainment, logistics, medical health, transportation, industry, natural sciences, energy, and classic artificial intelligence technology fields such as data mining, machine learning, computer vision, natural language processing, and decision intelligence. Download the TCM reading comprehension dataset (TCM-RC) and the TCM instruction manual named entity recognition dataset (TCM-NER) on the Ali Tianchi platform. These data are crucial to improving the performance of natural language processing models in the field of traditional Chinese medicine.

[0053] TCM Syndrome: TCM Syndrome Differentiation Dataset (TCM-SD) is the first public, real-world TCM syndrome differentiation dataset, containing a large number of real-world clinical records, covering a variety of TCM syndromes. Through GitHub (TCM-SD), the TCM syndrome dataset was compiled, providing strong support for syndrome diagnosis and classification research.

[0054] Herb and prescription recommendation: ShenNong_TCM_Dataset is a large dataset focusing on the field of traditional Chinese medicine. It is based on the open source knowledge graph of traditional Chinese medicine and generates instruction data around traditional Chinese medicine by calling models such as ChatGPT. In order to enhance the capabilities of the traditional Chinese medicine recommendation system, the ShenNong dataset (ShenNong_TCM_Dataset) is introduced, which provides a rich data foundation for personalized recommendations of herbs or prescriptions.

[0055] In another exemplary embodiment, in the above step 102, the unsupervised data mainly comes from books, BaiduBaiKe, TCM-DaYi, HowNet documents and ShenNong_TCM_Dataset. First, the optical character recognition method is used to extract text information from the PDF format of the book, and the extracted text is manually proofread and edited, including correcting spelling errors, adding or deleting punctuation marks and paragraph formats. When processing document abstracts, text mining is applied to delete HTML tags that may be contained in the content and correct symbol errors to ensure the accuracy and readability of the data. For TCM-DaYi and ShenNong_TCM_Dataset, a simple segmentation method is used to process them. The method for constructing supervised instruction data is a systematic process that ensures that TCMChat can perform specific tasks. Specifically, the following different construction strategies are used.

[0056] 1) Human-computer interaction instruction creation, 2) Template conversion to text format, 3) Open source data set collection. The data obtained by these strategies are manually verified and screened to generate six basic scenarios: TCM knowledge base, multiple-choice questions, reading comprehension, entity extraction, medical case diagnosis, and TCM or prescription recommendation.

[0057] In another exemplary embodiment, the Baichuan2-7B-Chat model in the above step 103 adopts the Transformer decoder architecture, and makes the following improvements based on previous work: 1) replace the normalization layer LayerNorm with RMSNorm; 2) replace the absolute position encoding with the rotational position encoding; 3) replace the activation function ReLu with SwiGLU.

[0058] In another exemplary embodiment, Figure 2 As shown, Figure 2The left part is the network structure of the large language model (LLM) Baichuan2-7B-Chat, which includes the input layer (Input Embedding), which is used to convert text strings into text vectors. The multi-head attention layer (Multi Head Attention) is used to enhance the model's capabilities and capture more levels of information. The normalization layer (RMSNorm) is used to solve problems such as parameters that are too large or too small and the gradient explosion caused by them. The feedforward layer (FFN (SwiGLU)) is used to perform nonlinear transformation and mapping on the vector representation of each position. Figure 2 Outputs in the figure represents the output of the multi-head attention layer, SoftMax is the activation function, RoPE is the rotational position encoding, Q, K, and V are the query, key, and value in the attention mechanism, respectively. Figure 2 The right part is the model training process, including pre-training and fine-tuning. The two use the same network structure. Pre-training mainly uses unsupervised data. After pre-training, instruction data is used for fine-tuning.

[0059] Specifically, the above steps 104 and 105 specifically include:

[0060] In the pre-training stage (corresponding to step 104), a causal language model (CLM) is used. For a given input sequence, the model only uses the previous information of the sequence to predict the probability of the next word or character, which is expressed in the mathematical representation of the loss function as follows:

[0061] (1)

[0062] in, is the loss function, represents a sentence sequence in unsupervised data, represents the length of the sentence sequence, The parameters of the Baichuan2-7B-Chat model are When using The word before the word Predicted Words probability.

[0063] For the SFT (Supervised Fine-Tuning) stage, the generative language model is fine-tuned using the same loss function as pre-training. The significant difference is that the question and answer texts are concatenated into a long sequence and special delimiters are added to mark the boundaries between questions and answers as input. Therefore, instruction data is used to enhance the adaptability and generalization capabilities of deep learning models, and the risk of overfitting the model on specific tasks is reduced by reducing the learning rate and larger batches.

[0064] In step 104 and step 105, since LLM usually requires a large amount of GPU memory to store various parameters (such as intermediate activations and weights) in training, using only one GPU cannot meet the training requirements. Conduct parallel experiments. At the same time, in order to increase the batch size during model training and prevent GPU video memory overflow (OOM, Out of Memory), the embodiment of the present application reduces the usage ratio of video memory through Deepspeed technology. At present, Deepspeed parallelism methods include ZeRO-1, ZeRO-2, ZeRO-3, ZeRO-Offload and ZeRO-Infinity. Among them, ZeRO-1 divides the optimizer parameters to multiple GPUs, and the process on each GPU is only responsible for updating its own part of the parameters. ZeRO-2 is based on ZeRO-1, and the gradient information is also split. Only a part of the gradient information is needed to update the model weight, and the model parameters are still saved by each computing device. After the gradient of a Layer is calculated, it will be aggregated through collective communication. The aggregated gradient information will only be saved by the required devices, and other unnecessary devices will be released. ZeRO-3 is based on ZeRO-1 and ZeRO-2, and the model weights are also split and distributed on different devices. Synchronize through collective communication only when necessary, and release immediately after the calculation is completed. ZeRO-Offload distributes model parameters on the CPU and GPU, and uses the CPU to calculate part of the gradient, thereby reducing video memory usage, but it also brings certain computing overhead. ZeRO-Infinity is an extension of ZeRO-3. It allows large models to be trained by using NVMe solid-state drives to expand GPU and CPU memory. Therefore, Deepspeed can efficiently train models with parameters ranging from 1.5B to hundreds of B, and provide significant acceleration for these models.

[0065] For example, TCMChat has different parameters for the pre-training and SFT (Supervised Fine-Tuning) stages. For the pre-training process, the learning rate is 2e-4, the batch size is 32 per GPU, and the maximum context length is 1024 tokens. For the SFT process, full parameter fine-tuning is used. The learning rate is adjusted to 2e-5, the batch size is 16 per GPU, and the maximum context length is limited to 1024 tokens. The optimizer of the model uses AdamW and specifies a learning rate of 1e-4. The training experiments used 8 NVIDIA A100 GPUs and utilized DeepSpeed's ZeRO-2 and ZeRO-3 methods to enhance training, minimize memory usage and speed up the process. To ensure training stability, we mitigate the problems of gradient explosion and learning rate decay by reducing the loss by half.

[0066] In another exemplary embodiment, in order to ensure the scientificity and accuracy of the research, instruction data in five of the scenarios were randomly selected. At the same time, in order to compare the quality of model training, different closed-source and open-source models were selected as baselines, and different parameter sizes can externally reflect the generation ability of the model. Among them, GPT-3.5-Turbo is an optimized version of GPT-3.5 (175 billion parameters), but its exact number of parameters has not yet been disclosed. Gemini-Pro is a large language model with 100 billion parameters. Bentsao-med, Bentsao-literature, HuatuoGPT, and CMLM-zhongjing all have 7 billion parameters, as does TCMChat. The smallest model, BenQue2, also has 6 billion parameters. Finally, TCMChat was benchmarked with other large language modeling algorithms to evaluate its performance in response to different scenarios. Specifically, in order to objectively compare the performance of TCMChat in the TCM question-and-answer application scenario, five different types of evaluation data sets are constructed, including multiple-choice questions, reading comprehension, entity extraction, medical case diagnosis, and TCM or prescription recommendation. In different application scenarios, the verification indicators in the above step 106 are as follows.

[0067] Multiple-choice questions: Multiple-choice questions cover knowledge of traditional Chinese medicine and prescriptions, from which 500 data are extracted as test sets. For each question-answer response in the test data set, the accuracy of the selected answer is calculated using the following formula:

[0068] (2)

[0069] in, is the accuracy of the selected answer, , , and are the number of true positives, false positives, false negatives, and true negatives, respectively.

[0070] Reading comprehension: 500 data sets were randomly selected as the test set for the reading comprehension scenario, and their similarity was evaluated using BLEU (Bilingual Evaluation Understudy), Meteor (Metric for Evaluation of Translation with Explicit Ordering), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), and BertScore.

[0071] Among them, the bilingual evaluation index is used to quantify the similarity between the generated sentence and the reference sentence, and the sentence-level fluency is evaluated by evaluating the n-gram overlap. The specific formula is as follows.

[0072] (3)

[0073] in, is a bilingual evaluation indicator, is the penalty factor, is the window size for capturing consecutive words, To capture the first word in the window of consecutive words The weight of a phrase set, For the The accuracy of the phrase set; The accuracy of a phrase set is obtained by comparing each phrase in the nth phrase set obtained by cutting the generated text and the reference text.

[0074] Secondly, the explicit ranking translation evaluation metric is used as a software metric to evaluate the quality and complexity of texts. The main formula is as follows.

[0075] (4)

[0076] (5)

[0077] (6)

[0078] in, Translate evaluation metrics for explicit ranking, is the fragmentation penalty factor, is the harmonic mean of precision and recall, To determine the accuracy of the generated text, is the recall rate of the generated text, To adjust and The weight parameter, To generate the number of phrases that can be matched in the text, is the number of phrases in the reference text, is the scale parameter, is the power parameter.

[0079] Then, the recall-oriented summary evaluation metric measures the n-gram overlap between the generated output and the reference summary. The principle is to evaluate the quality of the model response based on the word matching in the longest common subsequence. There are mainly ROUGE-N and ROUGE-L. The main formula is as follows:

[0080] (7)

[0081] in, The window size for capturing consecutive words The length of the summary is used as an indicator for evaluation. represents a reference text in a reference text set, represents a reference text set, To capture the first word in the window of consecutive words A collection of phrases, express The number of phrases that successfully matched express The total number of phrases contained in .

[0082] (8)

[0083] in, is a metric for summarizing the length of the longest common subsequence between the reference text and the generated text. To generate a text set, yes and The length of the longest common subsequence of is the length calculation formula, To adjust the parameters.

[0084] BertScore is calculated based on the cosine similarity between the word vectors output by the BERT model, and includes two core indicators: precision and recall (in the following introduction, recall refers to the recall of the embedding of the generated text set). Precision measures the degree of match between the words in the generated text and the words in the reference text, while recall measures the degree of match between the words in the reference text and the words in the generated text. The main formula is as follows:

[0085] (9)

[0086] (10)

[0087] (11)

[0088] in, is the accuracy, is the embedding of the reference text set, is the embedding of the generated text set, To generate the embedding of the text set The sentence embedding, is the embedding of the reference text set The embedding of a sentence, the superscript T represents transposition, To generate the recall of the embedding of the text set, Represents the BertScore indicator.

[0089] Entity extraction: In the entity extraction scenario, 480 data were randomly selected as the test set and evaluated using precision, recall, and BertScore indicators. For formulas, refer to formula (9), formula (10), and formula (11).

[0090] Medical case diagnosis: 500 data are randomly selected as the test set according to the medical case diagnosis scenario, and the accuracy (in the following introduction, accuracy refers to the accuracy of the selected answer), precision, recall and BertScore indicators are used for evaluation. The formulas refer to formula (1), formula (9), formula (10) and formula (11).

[0091] Traditional Chinese medicine or prescription recommendation scenario: 500 pieces of data are randomly selected as the test set, and MRR (Mean Reciprocal Rank), precision@K, and nDCG (Normalized Discounted Cumulative Gain) are used to evaluate them. Among them, mean reciprocal rank is a measure of the performance of the information retrieval system. It is used to evaluate the ranking quality of relevant documents in the system results. The formula is as follows.

[0092] (12)

[0093] in, is the average reciprocal ranking, is the recommended amount, It is Suggestions.

[0094] Secondly, Refers to the system in front The accuracy of providing correct or relevant information in a recommendation or search result is called accuracy@K, and the formula is as follows:

[0095]

[0096] in, represents the precision @K, Before The number of true positive examples in the recommendations, Before The number of true negative examples in the recommendations.

[0097] Normalized discounted cumulative gain is a metric that ranks the quality of search results and is used to measure the quality of search results or recommendation lists. It considers the relevance and position (rank) of each item in the search results and reduces the position because users are more likely to pay attention to the top-ranked results. The formula is as follows:

[0098] (14)

[0099] in, is the normalized discounted cumulative gain, is a list of recommendations for generating text, Middle The relevance score of the recommendation to the question, A recommended list of reference texts, yes Middle The relevance score of each recommendation to the question.

[0100] Based on the same inventive concept, the embodiment of the present application also provides a Chinese medicine question-answering device based on a large language model for implementing the Chinese medicine question-answering method based on a large language model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the Chinese medicine question-answering device based on a large language model provided below can refer to the limitations of the Chinese medicine question-answering method based on a large language model above, and will not be repeated here.

[0101] In order to intuitively demonstrate the effect of TCM Hat reasoning, the device embodiment designs and deploys a model. Deploying a large model usually involves a backend (such as Flask) and a frontend (such as Vue). Flask is mainly used to load model weights and develop external APIs, while Vue is mainly used to create user interaction pages.

[0102] In an exemplary embodiment, a traditional Chinese medicine question-and-answer device based on a large language model is provided, including: a back end and a front end, wherein the front end is used to interact with customers, and the back end is used to conduct question-and-answer using the traditional Chinese medicine question-and-answer method based on a large language model of the above embodiment.

[0103] When deploying the TCMChat model, the device embodiment not only builds an efficient backend service through Flask for quickly loading and running model weights, but also develops an easily extensible API interface to ensure that the front end can smoothly request and process data. At the same time, the Vue.js framework is used to build the front-end page, which provides a rich component library and responsive data binding, making the user interaction interface both beautiful and smooth, and can intuitively display the real-time results and interactive feedback of TCMChat reasoning.

[0104] In another exemplary embodiment, in order to verify the effects of the above method and device embodiments, the following example is also provided.

[0105] Example 1:

[0106] In order to evaluate the performance of important models on multiple-choice questions, this example selected 500 pieces of data on traditional Chinese medicine and prescriptions as test sets and input them into each model using the few-shot mode. The model inference process does not rely on the output text response, but directly uses the maximum probability value of the A, B, C, D, and E indicators in the last layer of embedding of the model as the predicted answer. Finally, the key indicator of accuracy is used to measure the performance of each model. The results show that Figure 3 As shown, Figure 3 The models GPT-3.5-turbo, Gemini-pro, Bentsao-med, Bentsao-literature, BianQue2, HuatuoGPT, CMLM-zhongjing, Baichuan2-7B-chat, and the TCMChat model obtained by supervised and unsupervised training of the Baichuan2-7B-Chat model in this application ( Figure 3 Compared with Ours in Figure 2, TCMChat outperforms other methods, with the accuracy of Chinese medicine and prescriptions being 64.0% and 77.0% respectively, while the accuracy of other models is less than 60%.

[0107] Example 2:

[0108] This example also evaluates the TCMChat model obtained by supervised and unsupervised training of the Baichuan2-7B-Chat model on a 500-item reading comprehension test dataset. Figure 4 Ours in ) and other large models ( Figure 4 The performance of GPT-3.5-turbo, Gemini-pro, Bentsao-med, Bentsao-literature, BianQue2, HuatuoGPT, CMLM-zhongjing, Baichuan2-7B-chat in the task. This task requires the model to extract or infer relevant information from the text and give answers to questions asked in a given text (usually called "context" or "article"). The scores are BLEU (bilingual evaluation index), METEOR (explicit ranking translation evaluation index), ROUGE (summary evaluation index) and BertScore (BertScore index). The larger the value, the better the performance. The results show that Figure 4 As shown, Figure 4 The gray closed lines in the figure represent the index scores, which are 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9 from the inside to the outside. The TCMChat model of this application is superior to other models in BLEU (0.602), Meteor (0.749), ROUGE-1 (0.789), ROUGE-2 (0.753), ROUGE-L (0.785), and BertScore (0.894). It is worth noting that the BLEU of the models Bentsao-med, Bentsao-literature, BianQue2, HuatuoGPT, and CMLM-zhongjing are all lower than 0.2, indicating that the text fluency generated by these models is low. Overall, the quality of the text generated by these models is not as good as the TCMChat model of this application.

[0109] Example 3:

[0110] Medical case diagnosis has many meanings and values ​​in the field of traditional Chinese medicine, and the judgment of evidence type can evaluate the processing ability of each model in medical case diagnosis tasks. Figure 5 As shown, in this example, in the 500 medical case diagnosis test data set, the TCMChat model of this application ( Figure 5The performance of Ours in the above dataset (Accuracy: 0.850, Precision: 0.593, Recall (the recall rate of the generated text set embedding): 0.565, F1-score (the harmonic mean of precision and recall): 0.569) far exceeds that of other models, and the evaluations of models Bentsao-med, Bentsao-literature, BianQue2, and HuatuoGPT are all 0, indicating that these models did not consider the diagnosis cases or medical records in actual medical practice when building or training this model. Although GPT-3.5-turbo, Gemini-pro, and CMLM-zhongjing can determine the type of medical record evidence, their performance values ​​are all relatively small and can hardly meet the capacity of medical record diagnosis. Therefore, TCM Hat medical record diagnosis has more advantages in algorithm.

[0111] Example 4:

[0112] This example evaluates entity extraction and randomly extracts 480 data from the TCM instruction manual dataset as a test set containing 13 entity types. The results are consistent, such as Figure 6 As shown in Figure 2, Bentsao-med, Bentsao-literature, BianQue2, HuatuoGPT, and CMLM-zhongjing cannot extract entities, while the TCMChat model of this application ( Figure 6 Ours in the above table all performed well in entity extraction (Precision: 0.996, Recall (recall of the embedded text set): 0.905, F1-score (harmonic mean of precision and recall): 0.941, model GPT-3.5-turbo (Precision: 0.988, Recall: 0.876, F1-score: 0.914), and model Gemini-pro performed well in entity extraction (Precision: 0.981, Recall: 0.852, F1-score: 0.894). It can be seen that the performance of the TCM Hat model of this application is slightly better than that of GPT-3.5-turbo and Gemini-pro.

[0113] Example 5:

[0114] Given the importance of intelligent recommendation in traditional Chinese medicine, this example randomly extracts 500 pieces of data as a test set, mainly involving traditional Chinese medicine and prescriptions based on diseases, symptoms, and efficacy. In addition, since it is impossible to evaluate the consistency of the output results of each model, this example parses the output response text, extracts the traditional Chinese medicine and prescription keywords in the text, and sorts them according to the order of text output. The evaluation results are as follows: Figure 7 As shown. The TCMChat model of this application ( Figure 7 The performance of the Ours (MRR (mean reciprocal rank): 0.55, precision@1 (precision @1): 0.286, precision@3 (precision @3): 0.530, nDCG (normalized discounted cumulative gain): 0.322) is significantly better than other large models (GPT-3.5-turbo, Gemini-pro, Bentsao-med, Bentsao-literature, BianQue2, HuatuoGPT, CMLM-zhongjing, Baichuan2-7B-chat). Although the performance of the TCMChat model can only reach a shallow level, after pre-training and SFT, the model has certain feasibility in intelligent recommendation of traditional Chinese medicine.

[0115] In another exemplary embodiment, in order to more intuitively illustrate the performance of TCM Hat, a comparative analysis of cases in different scenarios was conducted. These data mainly cover three scenarios of medical case diagnosis, reading comprehension, and recommendation of Chinese medicine or prescriptions, and the source is a test instruction set. Among them, the input content of medical case diagnosis data mainly includes patient information, medical information, symptom description, etc., and the input of reading comprehension is mainly contextual information and response questions. The input of Chinese medicine or prescription recommendation data is the attributes related to Chinese medicine or prescriptions. All model outputs are in text format, and the following examples are provided.

[0116] Example 6

[0117] This example involves a case in which the TCM medical record diagnosed as "asthma", such as Figure 8 As shown. Figure 8As can be seen from the figure, the responses of Bentsao-literature, Bentsao-med, and CMLM-zhongjing are relatively direct, without recognizing that the case information is related to TCM diagnosis and treatment. Their model responses do not contain relevant content or keywords to meet the needs of users, while the response of the Bianque2 model gives some treatment suggestions but repeatedly generates the same content. HuotuoGPT does not provide substantive TCM diagnosis suggestions; the response of Gemini-pro gives a very detailed prescription combination but cannot correctly judge the case type. In comparison, GPT-3.5-turbo's judgment on the case type is close to correct because it believes that the treatment is "resolving phlegm and relieving asthma, clearing heat and detoxifying." However, the best treatment is not "clearing heat and detoxifying" but "relieving cough." In contrast, the answer of the TCMChat model includes (1) syndrome type, (2) diagnosis, (3) treatment, and (4) suggestion, showing superior answer logic and comprehension compared with the baseline model.

[0118] Example 7

[0119] This example is a case of a reading comprehension task for extracting information from traditional Chinese medicine. Fig. 9 As shown in the figure, the response results of Bentsao-med and Bentsao-literature have large dialogue errors or repeatedly generate repeated texts, which cannot meet the needs of users. HuatuoGPT, CMLM-Chongqing and BianQue2 did not find the cause of "mouth cleft, pain" from the context, but only explained the condition based on model cognition. At the same time, Gemini-pro can only partially see the content in the context. In contrast, the GPT-3.5-turbo and TCMChat models are able to find explanations related to the condition.

[0120] Example 8

[0121] This example involves the case of intelligent recommendation of "similar compounds", such as Fig.10As shown, the responses of Bentsao-med and Bentsao-literature are not relevant to the question and do not provide suggestions related to similar chemical composition profiles. HuatuoGPT attempts to give some prescription suggestions, but they do not seem to be based on consideration of similar chemical composition profiles and lack factual basis and relevance. These suggestions do not seem to be based on consideration of similar chemical composition profiles and lack factual basis and relevance. The suggestions of BianQue2 and CMLM-zhongjing also appear to be disorganized, without a clear recommendation logic based on similar chemical composition profiles. Although Gemini-pro attempts to recommend prescriptions based on efficacy or pharmacological effects, this is not related to the topic of the question. "GPT-3.5-turbo's suggestions also tend to be based on efficacy or pharmacological effects rather than directly on chemical composition profiles." This once again reflects that the model has a poor understanding of the concept of "similar chemical composition profiles." In contrast, TCMChat performed the best. It was able to recommend prescriptions on the ETCM website whose chemical composition profiles are similar to those of a given compound or formula group. This demonstrates the high accuracy and reliability of TCMChat in understanding and handling such questions.

[0122] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data determined by complementarity. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a traditional Chinese medicine question-and-answer method based on a large language model is implemented.

[0123] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0124] In an exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps in the above-mentioned embodiment of the traditional Chinese medicine question-and-answer method based on a large language model are implemented.

[0125] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which, when executed by a processor, implements the steps in the above-mentioned embodiment of the traditional Chinese medicine question-and-answer method based on a large language model.

[0126] In an exemplary embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps in the above-mentioned embodiment of the traditional Chinese medicine question-answering method based on a large language model.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0128] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, etc., but is not limited thereto.

[0129] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A traditional Chinese medicine question-answering method based on a large language model, characterized in that: The traditional Chinese medicine question answering method based on the large language model includes: Constructing a traditional Chinese medicine knowledge database; the traditional Chinese medicine knowledge database includes multi-source traditional Chinese medicine knowledge data obtained from books and documents, network platforms and traditional Chinese medicine data sets; Generate unsupervised data and instruction data based on the traditional Chinese medicine knowledge database; the applicable scenarios of the instruction data include traditional Chinese medicine knowledge base, multiple-choice questions, reading comprehension, entity extraction, medical case diagnosis and traditional Chinese medicine or prescription recommendation; Constructing a Baichuan2-7B-Chat model; the Baichuan2-7B-Chat model is obtained by improving the Transformer decoder, wherein the improvement includes: replacing the LayerNorm layer in the Transformer decoder with the RMSNorm layer, replacing the absolute position encoding layer in the Transformer decoder with the rotation position encoding layer, and replacing the ReLu activation function in the Transformer decoder with the SwiGLU activation function; Use unsupervised data to perform unsupervised pre-training on the Baichuan2-7B-Chat model to obtain a pre-trained Baichuan2-7B-Chat model; Use unsupervised data and instruction data to perform supervised training on the pre-trained Baichuan2-7B-Chat model to obtain a trained Baichuan2-7B-Chat model; Verification indicators are set for different applicable scenarios to verify the trained Baichuan2-7B-Chat model; Use the verified Baichuan2-7B-Chat model to conduct traditional Chinese medicine question-answering; In the scenario of multiple-choice questions, the verification indicator is the accuracy of the selected answers; In the reading comprehension application scenario, the verification indicators include bilingual evaluation indicators, explicit sorting translation evaluation indicators, summary evaluation indicators, precision, recall rate of embedding of generated text sets, and BertScore indicators; In the applicable scenario of entity extraction, the verification indicators include: precision, recall rate of embedding of generated text set and BertScore indicator; In the medical case diagnosis application scenario, the verification indicators include: the accuracy of the selected answer, the precision, the embedding recall rate of the generated text set, and the BertScore indicator; In the application scenario of traditional Chinese medicine or prescription recommendation, the verification indicators include: average reciprocal ranking, precision @K and normalized discounted cumulative gain; The formula for calculating the accuracy of the selected answer is: ； in, is the accuracy of the selected answer, , , and are the number of true positives, false positives, false negatives, and true negatives, respectively; The calculation formula for the bilingual evaluation index is: ； in, is a bilingual evaluation indicator, is the penalty factor, is the window size for capturing consecutive words, To capture the first word in the window of consecutive words The weight of a phrase set, For the The accuracy of the phrase set; The accuracy of the phrase set is obtained by cutting the generated text and the reference text. The phrases in the phrase set are compared to obtain; The calculation formula of the explicit ranking translation evaluation index is: ； ； ； in, Translate evaluation metrics for explicit ranking, is the fragmentation penalty factor, is the harmonic mean of precision and recall, To determine the accuracy of the generated text, is the recall rate of the generated text, To adjust and The weight parameter, To generate the number of phrases that can be matched in the text, is the number of phrases in the reference text, is the scale parameter, is the power parameter; The calculation formula of the summary evaluation index is: ； ； in, The window size for capturing consecutive words The length of the summary is used as an indicator for evaluation. is a metric for summarizing the length of the longest common subsequence between the reference text and the generated text. represents a reference text in a reference text set, represents a reference text set, To capture the first word in the window of consecutive words A collection of phrases, express The number of phrases that successfully matched express The total number of phrases contained in To generate a text set, yes and The length of the longest common subsequence of is the length calculation formula, To adjust the parameters; The calculation formula for accuracy is: ； in, is the accuracy, is the embedding of the reference text set, is the embedding of the generated text set, To generate the embedding of the text set The sentence embedding, is the embedding of the reference text set The embedding of a sentence, the superscript T represents transposition; The formula for calculating the recall of the generated text set embedding is: ； in, The recall of the embedding for the generated text set; The calculation formula of the BertScore indicator is: ； in, Represents the BertScore indicator; The formula for calculating the average reciprocal ranking is: ； in, is the average reciprocal ranking, is the recommended amount, It is suggestions; The calculation formula of precision @K is: ； in, represents the precision @K, Before The number of true positive examples in the recommendations, Before The number of true negatives in the recommendations; The normalized discounted cumulative gain is calculated as: ； in, is the normalized discounted cumulative gain, is a list of recommendations for generating text, yes Middle The relevance score of the recommendation to the question, A recommended list of reference texts, yes Middle The relevance score of each recommendation to the question.

2. The Chinese medicine question-answering method based on a large language model according to claim 1, characterized in that: The loss function used in the process of unsupervised pre-training of the Baichuan2-7B-Chat model using unsupervised data and supervised training of the pre-trained Baichuan2-7B-Chat model using unsupervised data and instruction data is: ； in, is the loss function, Represents a sentence sequence in unsupervised data represents the length of the sentence sequence, The parameters of the Baichuan2-7B-Chat model are When using The word before the word Predicted Words probability.

3. The Chinese medicine question-answering method based on a large language model according to claim 1, characterized in that: The process of unsupervised pre-training of the Baichuan2-7B-Chat model using unsupervised data and the process of supervised training of the pre-trained Baichuan2-7B-Chat model using unsupervised data and instruction data both adopt a distributed training method based on Deepspeed.

4. A Chinese medicine question-answering device based on a large language model, characterized in that: The traditional Chinese medicine question-answering device comprises a back end and a front end; The front end is used to interact with customers; The backend is used to perform question answering using the traditional Chinese medicine question answering method based on a large language model as described in any one of claims 1 to 3.

5. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the traditional Chinese medicine question-answering method based on a large language model as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the traditional Chinese medicine question-answering method based on a large language model described in any one of claims 1 to 3 is implemented.

Images