Multi-task large language model training method for biomedical field

Abstract

The multi-task large language model training method for the biomedical field belongs to the field of large language model training. In order to solve the problem of single large language model realizing multi-task processing, the technical key points are as follows: constructing a training set of a first instruction data set for a medical intelligent question answering task; constructing a training set of a second instruction data set for a medical report generation task; constructing a training set of a third instruction data set for a biomedical information extraction task; training the large language model using the training sets, and the effect is that the single large language model can realize multi-task accurate and efficient processing.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of large language model training, and relates to a multi-task large language model training method for the biomedical field. BACKGROUND

[0002] In recent years, large language models (LLMs) have attracted much attention due to their outstanding performance in various general natural language processing tasks. These models, with large-scale parameters and massive training data, exhibit strong generation, understanding, and few-shot / zero-shot learning capabilities, achieving breakthroughs in text generation, question answering, translation, and other tasks. However, the biomedical field has unique text characteristics and stringent application requirements, which result in significant domain differences and technical bottlenecks in the direct application of LLMs, becoming an important obstacle to their development in this field.

[0003] Firstly, the biomedical field involves a large number of professional terms, proper nouns, and their abbreviations, and these terms have a high dependence on professional background knowledge. In addition, the complex syntactic structure and uncommon vocabulary in biomedical texts further increase the difficulty of text processing. Secondly, due to the rapid development of biomedical research, a large number of new terms and knowledge points are constantly emerging, leading to obvious lag in traditional pre-training models when dealing with new domain knowledge. This highly dynamic domain characteristic poses higher requirements on the generalization ability and adaptability of the model. More importantly, the biomedical field has extremely high requirements for the accuracy and reliability of text processing. In clinical diagnosis, medical report generation, and intelligent question answering tasks, model errors can directly affect medical decisions and thus threaten the safety of patients. In addition, the model needs to balance efficiency when processing biomedical data to meet the demand of large-scale data processing, while providing sufficient explainability to ensure the credibility and traceability of the prediction results.

[0004] In addition, although general LLMs such as GPT-4, PaLM, and LLaMA perform well in a wide range of natural language processing tasks, they often fail to meet the field-specific requirements when directly applied to the biomedical field. This is because general models are mainly trained based on open-domain data, which has significant differences in corpus distribution, language patterns, and domain knowledge from biomedical data. Therefore, general models are prone to inaccurate semantic understanding, term translation errors, and knowledge gaps when processing biomedical text, resulting in a significant decline in model performance. Currently, research on this problem focuses on two directions of optimization: domain fine-tuning and specialized model development. Some studies fine-tune general models (such as PubMedBERT and BioBERT) on biomedical domain data, significantly improving model performance on specific tasks. However, these methods mostly optimize for single tasks (such as question answering or text classification), and the multi-task generalization ability of the model has not been fully explored. In addition, model development focusing on a single language (mainly English) has limited performance in multilingual tasks, while biomedical research and practice often require cross-language processing capabilities, posing higher challenges for model development.

[0005] However, current application scenarios mostly correspond to a separate model for each task or each language, which not only consumes a large amount of resources but also has many inconveniences in deployment and use. Users need to frequently switch models between multiple needs, increasing operational complexity. In addition, small parameter models have a significant bottleneck in performance and poor reliability.

[0006] Therefore, developing a multi-task large language model suitable for the biomedical field is not only an important direction for current technological development but also an urgent need to solve practical problems in the field of biomedical information processing. Such a model not only needs to be fine-tuned on professional domain data but also needs to fully consider the specific needs of diverse tasks to improve the model's versatility and adaptability. In addition, by constructing a bilingual or multilingual model with cross-language support, the demand for multilingual communication in international medical research and practice can be effectively met. SUMMARY

[0007] To solve the problem of single large language model implementing multi-task processing, in a first aspect, according to the multi-task large language model training method for the biomedical field in some embodiments of the present application, the model is used to automatically select and execute medical intelligent question answering, medical report generation, or biomedical information extraction tasks according to the task type of the input question.

[0008] The method comprises

[0009] constructing a training set of an instruction data set for a medical intelligent question answering task, the training set of the instruction data set having a first instruction format, the first instruction format including a data instruction format based on context learning and a data instruction format based on a thinking chain;

[0010] constructing a training set of a second instruction data set for a medical report generation task, the training set of the second instruction data set having a second instruction format, the second instruction format including a data instruction format based on entity information embedding enhancement and a data instruction format based on spelling correction enhancement;

[0011] constructing a training set of a third instruction data set for a biomedical information extraction task, the training set of the third instruction data set having a third instruction format, the third instruction format including a data instruction format based on task description and type definition and a data instruction format based on special symbol marking;

[0012] training the large language model using the training set.

[0013] According to the multi-task large language model training method for the biomedical field in some embodiments of the present application, a MOSS data set for general field dialogue is constructed, which is used to automatically select and execute general field dialogue according to the task type of the input question.

[0014] According to the multi-task large language model training method for the biomedical field in some embodiments of the present application, a model self-cognition data set is constructed, wherein the model self-cognition data is inserted in each of the training sets.

[0015] According to the multi-task large language model training method for the biomedical field in some embodiments of the present application, the large language base model is a GLM-4-9B model, and the hyperparameters of the large language model are as follows:

[0016] Learning rate: 1e-05

[0017] Gradient accumulation steps: 16

[0018] Batch size: 1

[0019] Data type (dtype): bf16

[0020] Random seed: 42.

[0021] According to the multi-task large language model training method for the biomedical field in some embodiments of the present application, the training set of the instruction data set for the medical intelligent question and answer task is constructed, including

[0022] The first instruction, a piece of data in the training set of the first data set, or the first instruction, a piece of data in the training set of the first data set, and the selected example from the few samples in the first data set are constructed as a first training data in the training set. Each piece of first training data corresponding to each piece of data in the training set of the first data set is obtained. The first training data is data in a context-based data instruction format.

[0023] Using the large language model, a reasoning corresponding to a piece of data in the training set of the first data set is generated through a second instruction. A piece of data in the training set of the first data set is constructed as a second training data in the training set through the second instruction and the reasoning. Each piece of second training data corresponding to each piece of data in the training set of the first data set is obtained. The second training data is data in a data instruction format based on a thinking chain.

[0024] According to the multi-task large language model training method for the biomedical field in some embodiments of the present application, wherein:

[0025] The content of the first instruction includes "answer in the format of 'answer: option'".

[0026] The content of the second instruction includes "show the reasoning process".

[0027] According to the multi-task large language model training method for the biomedical field in some embodiments of the present application, the training set of the second instruction data set for the medical report generation task is constructed, including

[0028] An entity category in a piece of data in the training set of the second data set is generated. The piece of data and the entity category are constructed as a third training data in the training set. Each piece of third training data corresponding to each piece of data in the training set of the second data set is obtained. The third training data is data in a data instruction format based on entity information embedding.

[0029] Using the large language model, a piece of data after spelling error correction of a piece of data in the training set of the second data set is generated through a fourth instruction. The fourth instruction, the piece of data, and the piece of data after spelling error correction are constructed as a fourth training data in the training set. Each piece of fourth training data corresponding to each piece of data in the training set of the second data set is obtained. The fourth training data is data in a data instruction format based on spelling correction enhancement.

[0030] According to the multi-task large language model training method for the biomedical field in some embodiments of the present application, wherein:

[0031] The entity category in the data in the training set of the second data set is generated, including extracting the entity category using a medical entity extraction tool.

[0032] The content of the fourth indication includes "spelling correction".

[0033] According to the multi-task large language model training method for the biomedical field in some embodiments of the present application, wherein the training set of the third instruction data set for the biomedical information extraction task is constructed, including

[0034] The fifth indication and the data in the training set of the third data set are constructed as a piece of data in the training set, and each piece of data in the training set is constructed as a piece of fifth training data, and each piece of data in the training set corresponds to each piece of fifth training data, and the fifth training data is data in a data instruction format based on task description and type definition; wherein the content of the fifth indication includes task description and type definition.

[0035] A plurality of text sequences of a piece of data in the training set of the third data set are generated, and each text sequence has an entity marked by a special symbol, and the piece of data and the text sequence are constructed as a piece of sixth training data in the training set, and each piece of data in the training set corresponds to each piece of sixth training data, and the sixth training data is data in a data instruction format based on a special symbol mark.

[0036] In a second aspect, the embodiments of the present application also provide an electronic device, which includes one or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions, which, when executed by the electronic device, cause the electronic device to perform the first aspect and any possible technical solutions of the first aspect.

[0037] Beneficial effects: through in-depth research on the key technology of large language model for the biomedical field, the present application has achieved remarkable results, successfully constructed a multi-lingual, multi-task biomedical large model, and showed excellent performance, covering multiple tasks such as multi-lingual biomedical intelligent question and answer, doctor-patient dialogue, medical report generation, biomedical information extraction, etc.

[0038] Through the technical means such as context learning, instruction fine-tuning, and thinking chain method, the application effectively improves the answer quality and generalization ability of the model in the Chinese medical question and answer task. In the medical report generation task, the application adopts optimization methods such as named entity information embedding and spelling correction, which significantly improves the summary generation performance of the model. At the same time, in the biomedical information extraction task, different instructions and decoding strategies based on large models are explored, which not only can accurately predict entity mentions, but also can mark the specific position of the entity in the text.

[0039] Based on the above, the application integrates a large number of English open source data sets, covers multiple tasks, and further enhances the generalization ability and entity position recognition ability of the model through the carefully designed instruction templates and decoding strategies. In addition, the application also constructs role-playing data, multi-task question and answer data, and self-cognition data, which further improves the comprehensive ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 Overall framework.

[0041] Figure 2 Zero-shot example.

[0042] Figure 3 Few-shot example.

[0043] Figure 4 Example of instruction format of MedQACoT training set.

[0044] Figure 5 Comparison of answers generated by two models.

[0045] Figure 6 Data sample after entity information embedding enhancement processing.

[0046] Figure 7 Example of spelling errors existing in HQS data set.

[0047] Figure 8 Example of spelling error detection and correction in HQS task.

[0048] Figure 9 Simple instruction.

[0049] Figure 10 Task description and type definition instruction.

[0050] Figure 11 Task description, type definition, and example instruction.

[0051] Figure 12 HTML decoding scheme.

[0052] Figure 13 Index decoding scheme.

[0053] Figure 14 Special symbol marking decoding scheme.

[0054] Figure 15 Instruction data format example.

[0055] Figure 16 Medical intelligent answering instruction data format example.

[0056] Figure 17 Self-awareness, general field dialogue instruction data format example.

[0057] Figure 18 Self-awareness and dialogue ability test. DETAILED DESCRIPTION

[0058] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should belong to the scope of protection of the present application.

[0059] I. SUMMARYWith the rapid development of large language model technology, these models have made significant progress in various general field tasks, showing strong versatility and task adaptability. However, in the biological field, there are still unique challenges such as strong professional nature of biomedical text, rapid dynamic changes, and large data scale. Therefore, the present application aims to address the special needs of the biomedical field and proposes a multi-task fine-tuning method based on large language models to build a large language model system that meets the high standards of the industry.

[0060] The core goal of the present application is to develop a multi-task large language model specifically applicable to the biomedical field by integrating natural language processing technology, large language models, and research results in biomedical informatics. This model can efficiently handle multiple task scenarios including biomedical intelligent question answering, doctor-patient dialogue, medical report generation, and information extraction.

[0061] The present application covers three core tasks: medical question answering, medical report generation, and biomedical information extraction. In addition, considering that the present application also needs to have certain self-awareness and general field dialogue capabilities, on the basis of task optimization, it will also explore how to enable the model to explicitly identify itself and have certain basic dialogue capabilities.

[0062] To ensure the best performance of each task, the present application will prioritize independent optimization of medical question answering, medical report generation and biomedical information extraction tasks, develop independent optimization strategies for each task, and explore the optimal training method for each core task. After completing the optimization of core tasks, a series of methods will be explored to further enhance the self-awareness of the present application and give it the ability to communicate in general fields, enabling it to communicate naturally with users. After completing all explorations, summarize these innovative task optimization methods and integrate training using existing field resources to obtain the final biomedical large language model with multi-task bilingual capabilities. The overall framework of the present application is as follows Figure 1 as shown.

[0063] II. Explanation of technical terms

[0064] Large Language Models (LLM) are artificial intelligence models based on deep learning technology, designed to understand and generate human language. These models typically contain hundreds of billions or even thousands of billions of parameters, trained on large-scale text data to learn the patterns, grammar and semantics of language. The core architecture of large language models is mostly based on Transformer, which can capture the complexity and diversity of language and generate natural and fluent text with contextual understanding.

[0065] III. Medical intelligent question answering task

[0066] The medical intelligent question answering task module uses context learning, instruction fine-tuning and thought chain construction methods to analyze and experiment on the performance of domestic large language models in Chinese medical question answering tasks. First, the Chat version of the model base is guided by prompt words to answer questions. Since the Chat version of the model has certain general capabilities, it can effectively answer Chinese medical related questions through zero-shot learning or few-shot learning. Then, the QLoRA method is also used to fine-tune multiple large language models efficiently to select the most suitable model for subsequent experiments. Further, to improve the performance of models with relatively small parameter quantities in medical question answering, the present application uses the advanced large language model GLM-4-9B to expand the MedQA dataset with its powerful generation capabilities, constructs instruction datasets with thought chains by generating medical explanations corresponding to questions, and effectively improves the performance of models in Chinese medical question answering. The specific implementation technical solutions and experimental results of this task module will be introduced in detail below.

[0067] 3.1 Context learning

[0068] Context learning of large language models refers to providing a certain number of example samples before inputting the question, so that the model can understand the pattern of the current task according to these examples and stimulate the knowledge possessed by the model itself, thereby more effectively answering the question. This method is also known as few-shot prompt learning, and the key is the design of prompt words and the selection of example data. Prompt words play a guiding role, and they can clearly convey the goal and format of the task, thereby stimulating the model's processing ability for specific tasks. Example data plays the role of actual teaching, and by showing specific examples of the task, it helps the model better understand how to apply knowledge to solve problems. Just like students when solving math problems, looking at a few related examples in advance can clearly understand the steps and techniques for solving problems, thereby more smoothly completing similar problems.

[0069] When constructing the prompt words of few-shot learning, the first step is to clearly define and describe the natural language processing task to be solved. The model is positioned as a "medical expert" in this invention, and the task is to solve Chinese medical question and answer selection questions. Then, a small number of samples are carefully selected from the task-related dataset as examples. These samples should be representative and cover various situations and variants that may occur in the task.

[0070] The CMB dataset is used in the context learning task in this invention. CMB contains multiple categories of Chinese medical question and answer selection questions. This invention retains and medical-related data, including physician exams, nursing exams, pharmacist exams, medical technology exams, professional knowledge exams, and medical postgraduate exams. At the same time, it filters and removes data unrelated to medicine, such as subsets of political science for postgraduate exams.

[0071] To ensure the effectiveness of the prompt words and prevent the model from being confused by too many example samples, the number of selected samples is usually controlled between 3 and 5. After selecting the examples, the invention organizes the selected examples into prompt content according to the principle of simplicity and clarity according to a specific template. After several attempts, it is decided to use "Question: … Answer: …" as the template and arrange the examples in order to ensure the logical relationship between the examples is clear.

[0072] Finally, the test sample is added to the end of the prompt words and input into the large language model along with the examples for processing. The model generates a prediction result for the test sample based on the prompt words and examples. This invention uses 0-shot, 3-shot, and 5-shot (representing 0, 3, and 5 example samples, respectively) to test the medical question and answer ability of the Chat model. The corresponding prompt word examples are shown in Table 3.2. Figure 2 、 Figure 3 The experimental results are shown in Table 3.2.

[0073] 3.2 Instruction fine-tuning related exploration

[0074] The present application constructs instruction data based on existing data sets, and fine-tunes each model using the QLoRA method. The fine-tuned model can accurately output medical question and answer answers according to the instruction format, and finally generates the answer effect through script evaluation to explore the best instruction data set construction method.

[0075] Firstly, the present application fine-tunes the model on the MedQA training set, and evaluates the question and answer ability on the C-Eval medical related subsets (clinical medicine, veterinary medicine, junior high school biology, physician qualification, basic medicine, plant protection, and high school biology seven subsets) and MedQA test set, uses accuracy as the main evaluation index, explores the performance of different models in Chinese medical question and answer task, and the difference of different parameter scale models. The results are shown in Table 3.3.

[0076] Then, the present application fine-tunes the model using the training sets of the existing three data sets (MedQA, CMExam, and CMB), and evaluates the performance on each test set to test the generalization ability of the model. At the same time, in order to improve the data diversity, so that the model can learn more extensive patterns to improve its generalization ability in different tasks and scenarios, the present application fuses the training sets of the three data sets to create a new training set Mix3. In order to ensure the rigor of the experiment, the repeated data between the training sets is removed during the merging process, and the part repeated with the test set is also removed to avoid data leakage. Fusing multiple data sets can also avoid the model from over-relying on the characteristics of a specific data set, and enhance the universality of the model.

[0077] By fine-tuning using Mix3 and evaluating the model performance on C-Eval, CMExam, CMB, and MedQA test sets, the present application studies the influence of training data diversity on model performance, and further reveals the relationship between data diversity and model performance, providing an important reference for future model optimization. The results are shown in Table 3.4.

[0078] 3.3 Construction of thought chain data set

[0079] In order to fully exert the ability of large language model in reasoning generation, the present application combines the thought chain method and constructs instruction data containing medical explanation in the fine-tuning process, so that the model can follow the process of "reasoning first and then giving answer" when answering Chinese medical questions, thereby tapping its potential in medical question and answer.

[0080] To this end, the application constructs the thought chain data on the training set of MedQA. First, the application uses the system prompt word "As a medical expert, your task is to help users solve complex Chinese medical multiple-choice questions. For each question, show your reasoning process and choose the best option." consumes nearly 5 million tokens, and uses the GLM-4-9B model to generate the corresponding medical answer explanation for the 7106 data in the training set of MedQA. Then the application uses regular expressions to extract data answers, and manually identifies and modifies the answers generated by the model. Finally, compared with the correct answers, 6221 pieces of reasoning correct data are retained to form a new thought chain training set MedQACoT, whose format is as shown in Figure 4

[0081] 3.4 Experimental results

[0082] During the training process, the parameter setting takes ChatGLM3-6B-Base as an example, the training round is 5, the learning rate is 2e-4, and the preheating learning strategy is adopted, the preheating step number is 500, the rank of QLoRA matrix is 64, the scaling coefficient is 16, and fp16 hybrid precision is adopted. During the reasoning process, the temperature coefficient temperature is set to 0.35, and the sampling coefficient top_p is set to 0.9. Since the application uses all the data in the Chinese medical question and answer data set as multiple-choice questions, the accuracy is used as an indicator for experimental evaluation, and the calculation formula is: "Accuracy = Correct answer number / Total number of questions".

[0083] The data set used in the experiment of the application includes the following main data sets:

[0084] (1) C-Eval: The C-Eval data set consists of 13948 multiple-choice questions, which cover a wide range of content from humanities to science and engineering, covering 52 different disciplines, and also includes four difficulty levels: middle school, high school, university and professional examination. The questions in this data set are all single-choice questions. The application mainly uses the data subset related to Chinese medical question and answer in the Chinese LLM evaluation kit for model evaluation, including clinical medicine, veterinary medicine, junior high school biology, physician qualification, basic medicine, plant protection and high school biology related questions, and the number of each category is shown in Table 3.1.

[0085] ​(2) MedQA: MedQA dataset is the first open question answering dataset for medical questions in free-form multiple-choice format, containing 3 languages: English, Simplified Chinese and Traditional Chinese, with 12,723, 34,251 and 14,123 questions respectively. The questions in this dataset are single-choice questions, and the inventors use the MedQA dataset in Simplified Chinese for experiments.

[0086] (3) CMB: CMB dataset is a comprehensive and multi-level Chinese medical evaluation dataset, containing 280,839 multiple-choice questions and 74 complex case consultation questions, covering all clinical medical specialties and different professional levels. The goal of this dataset is to comprehensively evaluate the medical knowledge and clinical consultation ability of the model, covering multiple categories such as physician examination, nursing examination, pharmacist examination, medical technology examination, professional knowledge examination, and medical postgraduate examination, and also contains single and multiple choice questions.

[0087] (4) CMExam: The questions in the CMExam dataset come from the Chinese National Medical License Examination, containing more than 60,000 multiple-choice questions. This dataset is used for standardized and objective evaluation, and the test set also contains five additional annotation dimensions: disease group, clinical department, medical department, medical discipline, ability field and question difficulty level. 85.24% of the questions in the dataset also contain medical explanations for part of the questions, with difficulty levels ranging from 1 to 5, representing easy, manageable, medium, difficult and extremely difficult. The division of the training set, development set and test set of each dataset is shown in Table 3.1.

[0088] Table 3.1 Dataset usage information

[0089]

[0090] The large language model used in the research is mainly the open source Chinese large language model in China, including the Qwen series of Ali Cloud, the Baichuan series of Baichuan Intelligence, and the ChatGLM series of Zhi Spectrum.

[0091] (1) Comparison of question and answer performance based on context learning

[0092] In order to study the zero-shot learning ability of large language models and their performance in few-shot learning, this section tests the Chat version of the model on the test set of CMB. Zero-shot learning directly asks questions through prompt words, while few-shot learning adds data selected from the CMB training set to form a sample template based on the prompt words of zero-shot learning. Table 3.2 shows the performance comparison of the three models on the CMB test set.

[0093] Table 3.2 Performance comparison of few-shot learning and zero-shot learning (%)

[0094]

[0095] where the bold part is the highest value in the column, N-Shot represents the number of examples in the prompt, 0 means no example, and 3 and 5 represent 3 and 5 examples, respectively.

[0096] The results show that the performance of Qwen-7B-Chat is significantly better than the other two models, whether in zero-shot learning or few-shot learning, its accuracy is higher than that of other models. In particular, Qwen-7B-Chat shows strong instruction following ability in zero-shot learning, and the accuracy of zero-shot learning is even higher than that of few-shot learning. At the same time, the performance of ChatGLM3-6B and Qwen-1.8B-Chat in few-shot learning is better than that in zero-shot learning. This shows that few-shot learning can further improve the question and answer ability of the model by adding examples. Although 5-shot has two more examples in the prompt than 3-shot, it does not make the model accuracy improve.

[0097] (2) Comparison of question and answer performance of different large language model instruction fine-tuning

[0098] In order to test the performance of different large language models in Chinese medical question and answer tasks, this section uses domestic open source large language models to fine-tune on the MedQA training set, and then evaluates on the C-Eval medical related subset and MedQA test set respectively, and compares and analyzes. The test results of each model after fine-tuning training are shown in Table 3.3.

[0099] Analyzing the test results in Table 3.3, the following four conclusions can be drawn:

[0100] 1) In the same series of models, the model with more parameters generally performs better on the test set. For example, the 13B model of the Baichuan2 series is about 6% higher than the 7B model in the average accuracy of C-Eval, and about 5% higher on the MedQA test set; Similarly, the 7B model of the Qwen series is about 6% higher than the 1.8B model on the MedQA test set, and nearly 10% higher on the C-Eval. This shows that the more parameters a large language model has, the more content and knowledge it learns, and thus it can better utilize its vast knowledge to solve problems in testing, thereby improving performance.

[0101] Table 3.3 Fine-tuning test results of each model (%)

[0102]

[0103] The underlined part is the highest value of the domestic large language model in this column, and the bold part is the highest value of all models in this column considering GPT-3.5 and GPT-4.

[0104] 2) The performance of Base and Chat versions of different large language models varies greatly. For example, the Chat version of Qwen-1.8B outperforms the Base version on the C-Eval and MedQA test sets, with a difference of about 4%. For the ChatGLM3 series, the Base version generally outperforms the Chat version. The performance difference between the two versions of the Baichuan2-7B series is relatively small. This difference may be closely related to the strategies and data used during training.

[0105] 3) Small parameter large language models may not necessarily outperform large parameter large language models, which is not in conflict with conclusion (1). As can be seen from the results in Table 3.3, the performance of ChatGLM3-6B-Base is better than Baichuan2-13B, with an average accuracy on the C-Eval test set of nearly 7%. This phenomenon can be explained as follows: First, the size of the parameter is not always directly related to the performance of the model on a specific task. ChatGLM3-6B-Base may have better generalization ability and be able to better learn task-related key information, thus being able to more accurately answer questions in the test set after fine-tuning. In addition, although large parameter models may have stronger learning ability, they may also learn some noise or irrelevant information that is not related to the task, leading to overfitting or performance degradation. Small parameter models are usually more concise and can focus on key information related to the task, thus avoiding such interference. In addition, the initialization and training process of the model may also affect the performance after fine-tuning. If a large parameter model is not properly initialized or fine-tuned, it may not be able to fully realize its potential. In contrast, ChatGLM3-6B-Base may perform better on specific tasks through more optimal training strategies.

[0106] 4) Compared with GPT-3.5, the test results of ChatGLM3-6B-Base and Baichuan2-13B are generally better than GPT-3.5; however, compared with GPT-4, there is still a significant gap between domestic open source models. Except for the "high school biology" questions and the MedQA test set, the performance of ChatGLM3-6B-Base is slightly better than GPT-4, and the performance on other test sets is surpassed by GPT-4.

[0107] Table 3.4 Results of models fine-tuned on different training sets on different test sets (%)

[0108]

[0109] where the underlined part represents the highest value of this column without considering the test results of Mix3 and GPT-4 test results, and the bold part is the highest value of this column considering all test results.

[0110] Table 3.4 shows the results of the model after fine-tuning on different training sets and evaluating on different test sets. The experimental results show that the model has good generalization ability after fine-tuning training. For the results on the C-Eval test set, it can be found that ChatGLM3-6B-Base fine-tuned on the CMExam dataset performs best without considering GPT-4. By analyzing the specific data, it can be found that the content correlation of these two datasets is relatively high, and the training set of CMExam has a certain data size. The model obtains relevant useful knowledge to solve the problems in the C-Eval dataset during the fine-tuning process, thereby achieving good results.

[0111] Observing the underlined data, it can be analyzed that the fine-tuning on the CMB dataset has the best generalization effect, which is benefited to a certain extent by the large data size of CMB. The model fine-tuned on Mix3 achieves the highest value on most test sets, except for the C-Eval item which is lower than GPT-4, and other items are higher than GPT-4. This shows that the large-scale dataset after fusion contains more knowledge and enhances the diversity of data, which helps the model to better adapt to various situations during fine-tuning. Therefore, fine-tuning using large-scale datasets can improve the stability of the model and make it maintain good performance in different scenarios.

[0112] (3) Comparison of question and answer performance based on thought chain method

[0113] To further explore the reasoning ability of large language models, this section uses the CMExam dataset and the self-constructed MedQACoT dataset with medical explanations to construct instruction data with thought chains, i.e. <input, thought chain, output>, fine-tune the model, and test it on the CMExam test set. The results are shown in Table 3.5.

[0114] Table 3.5 CMExam dataset fine-tuning results with thought chain (%)

[0115]

[0116] where AO (answer only) represents fine-tuning without thought chain, CoT represents fine-tuning using instruction data with thought chain, and the bold part is the highest value of this column.

[0117] From the data in Table 3.5, it can be found that the performance of the model after adding the thinking chain does not improve significantly, and even the accuracy of some models decreases. The reason analysis shows that the parameter quantity of the model is small, and the fine-tuned model may have reached its upper limit of answering ability, and the thinking chain does not further stimulate its "emergence" ability. In the research of large language models, "emergence" refers to the qualitative leap in the ability of the model when its size reaches a certain threshold, showing excellent language understanding and reasoning ability. Usually, only when the parameter quantity of the model reaches 10B to 100B, can it exhibit the emergence ability.

[0118] By comparing the answering effect with the Chat version model, the application finds that the ChatGLM3-6B-Base model fine-tuned with thinking chain data has a reasoning process that is more in line with medical common sense, while the general dialogue model ChatGLM3-6B has a more serious "hallucination" problem in Chinese medical question answering. As shown in the following table, the correct answer should be option D, while ChatGLM3-6B in the answering process seems reasonable in analyzing options A and D, but compared with the answer of ChatGLM3-6B-Base model fine-tuned, it is found that these analyses do not follow medical common sense, and in fact are fictional explanations of the model. Figure 5

[0119] The reason for this phenomenon is that general dialogue models are usually pre-trained based on a large amount of cross-domain data, aiming to solve a wide range of natural language processing tasks. However, this wide generality leads to unsatisfactory performance of the model in specific domains (such as medical question answering). Through fine-tuning, the model can be trained on the basis of pre-training for specific domains, learning professional terms, context and rules within the domain, thereby significantly improving the performance of the model in that domain. This targeted fine-tuning training can help the model better solve specific tasks and avoid inaccurate answers due to lack of professional knowledge.

[0120] Four, medical report generation task

[0121] The medical report generation task module mainly focuses on the automatic summary generation of medical text, and carries out research on three task data of health problem summary, medical radiology report generation and doctor-patient dialogue summary. Medical text usually contains complex entity information and rich contextual relationships, and it is challenging to generate accurate summaries. Therefore, the application proposes to optimize the performance of the model in the summary generation task through entity information embedding, spelling error detection and correction, etc. The specific implementation technical scheme and experimental results are as follows.

[0122] 4.1 Entity information embedding enhanced generation method

[0123] ​In the medical report generation task, due to the length of the dialogue data, the model often loses the memory of part of the key information in the generation process, leading to the omission of important content or inaccurate information in the summary. Therefore, in the medical dialogue summary generation task, adding entity information is crucial, which helps to improve the model's ability to extract key information, reduce information forgetting, enhance the structure and readability of the summary, improve the model's understanding of the clinical background, and improve the interpretability of the summary. By effectively utilizing entity information, the model can generate more accurate, comprehensive and practical medical summaries to provide more valuable decision support for medical personnel.

[0124] The present application carries out entity information embedding for the doctor-patient dialogue summary task (IMC-V2 dataset) and the medical radiology report generation task (RRS dataset) and carries out related experiments to explore the effect of the method.

[0125] For the IMC-V2 dataset, each data in the dataset is a real doctor-patient dialogue, and the model needs to generate a medical summary according to the dialogue. The biological-related entities (symptoms, drugs, drug categories, examinations, operations, etc.) in the dialogue are extracted and added to the end of the dialogue in the format of "[entity category]: [entity content information]". For the RRS dataset, each data in the dataset is a radiology examination report, which is an English dataset. The present application extracts the medical entities in the report using the open-source medical entity extraction tool MEDCAT and adds them to the end of the report content in the format of "[Contain Medical Entitys]:...". The processed data sample is shown. Figure 6

[0126] For the doctor-patient dialogue summary task, the present application first fine-tunes the QLoRA on the IMC-V2 dataset on the Bio-ProphetNet, T5, ChatGLM3 three models and tests on their test sets, and the results are used for subsequent experimental comparison. The present application enhances the IMC-V2 dataset through the entity information embedding method and fine-tunes and tests ChatGLM3 again to explore the effectiveness of the method. At the same time, through multiple experiments by changing the number of embedded entity information, the best strategy is explored.

[0127] Similarly, for the medical radiology report generation task, the present application fine-tunes and tests the QLoRA on the RRS dataset on the allenai / biomed_roberta_base, BERT-base, Qwen2.5-7B-Instruct three models, and fine-tunes the Qwen2.5-7B-Instruct again through entity embedding to enhance the dataset, and compares the experimental results to prove the effectiveness of the method. ​

[0128] 4.2 Spelling Correction Augmentation Generation

[0129] In medical forums and online question-asking, users are often not experts, and thus spelling errors are common in the dataset. Un-corrected spelling errors not only lead to mismatches between source text and summary, but also produce incorrect outputs in the prediction process. Such errors are particularly significant in evaluation metrics based on string matching (e.g., Rouge), as they disrupt n-gram matching. In addition, certain spelling errors can cause the model to deviate in semantic understanding, as shown in Figure 7

[0130] For the above spelling correction scheme, the present application uses the GLM-4 model to perform spelling detection on the HQS training set, and returns the words that need to be modified by setting appropriate cue words, as shown in Figure 8 Subsequently, manual verification and correction are performed. This effectively reduces the spelling errors in the HQS dataset. The present application uses the original HQS dataset and the corrected HQS dataset to fine-tune the Llama-3.1 model for QLoRA, and tests it on the HQS test set to compare the results and explore the effectiveness of the spelling correction augmentation scheme.

[0131] 4.3 Experimental Results

[0132] The present application conducts experiments on three tasks: health question summary (HQS), medical radiology report generation (RRS), and doctor-patient conversation summary (IMCS-V2):

[0133] 1) Health Question Summary (HQS): This task aims to generate a concise summary of the question, covering the minimum information needed to find the correct answer from the original question. The challenge of this task lies in accurately extracting key information and refining it, avoiding oversimplification or missing critical information.

[0134] 2) Medical Radiology Report Generation (RRS): Radiology report summary aims to condense detailed examination analysis and results into a concise impression section, capturing the most prominent and actionable information in the study. The present application uses the MIMIC-III radiology report dataset, which includes 7 anatomical sites (head, abdomen, chest, spine, neck, sinuses, pelvis) and two modalities (magnetic resonance imaging MRI and computed tomography CT). The data comes from patients in the intensive care unit of the Beilinson Hospital in Israel between 2001 and 2012.

[0135] ​3) IMCS-V2: This task aims to automatically generate a medical report from a doctor-patient conversation. The present invention uses the IMCS-V2 dataset constructed by the Big Data Institute of Fudan University under the guidance of experts from the Fudan University School of Medicine. This dataset covers multi-level artificial labeling of real doctor-patient conversations and is suitable for tasks such as named entity recognition (NER), dialogue act analysis (DAC), and medical report generation (MRG). The generated medical report covers six main aspects: chief complaint, present illness history, auxiliary examination, past medical history, diagnosis, and recommendations. The statistical results of the dataset are shown in Table 4.1.

[0136] Table 4.1 Dataset statistics

[0137]

[0138] The experimental results are measured by traditional summary evaluation indicators to evaluate the quality of the generated summary. The following indicators are mainly used:

[0139] 1) Rouge-1: Based on word-level overlap, it evaluates the unigram overlap between the generated summary and the reference summary, focusing on precision and recall.

[0140] 2) Rouge-2: It evaluates the bigram overlap between the generated summary and the reference summary, calculating the matching degree of bigrams to measure text similarity.

[0141] 3) ROUGE-L: Based on the longest common subsequence (LCS), it evaluates text similarity, considering precision and recall.

[0142] 4) BERTScore: It uses BERT's contextual embeddings to calculate the semantic similarity between the generated text and the reference text, supplementing the syntactic level evaluation.

[0143] (1) Performance of different base fine-tuning

[0144] The present invention fine-tunes the model using the Adam optimizer with weight decay repair function. In the health problem summary task, the present invention selects three advanced open-source large models for comparison: Llama-3.1-7B-Instruct, Qwen2.5-7B-Instruct, and GLM-4-9B-Chat. The performance comparison results of the three models are shown in Table 4.2, which shows the performance differences of each model in the task.

[0145] Table 4.2 Performance comparison of different base models on the HQS task (Rouge-F1 as the main indicator)

[0146]

[0147] (2) Spelling error detection and correction results

[0148] The present application also uses the Adam optimizer with weight decay repair function and the QLoRA fine-tuning method to fine-tune the model for the health problem summary (HQS) task, and shows the spelling error detection and correction results. Table 4.3 shows the corrected model performance.

[0149] Table 4.3 Spelling check and correction in the HQS task

[0150]

[0151] From the results, it can be seen that after correcting spelling errors, various indicators have been significantly improved. The correction of spelling errors helps to reduce the penalty in string matching metrics, while enhancing the model's understanding of text semantics, making it more accurate in the generation process.

[0152] (3) Performance of adding entity information model

[0153] In the doctor-patient conversation summary task, the present application fine-tunes the model by embedding entity information, and compares the effects of adding different entities. Table 4.4 shows the performance results of ChatGLM3 model before and after embedding different entity information. The results show that the introduction of entity information improves various evaluation indicators, proving the effectiveness of entity information in improving model prediction accuracy. However, not the more entity information, the better the model performance. Too much entity information may cause the model to lose focus, affecting performance. Therefore, how to optimize the selection and use of entity information is still the key to further research.

[0154] In the medical radiology report generation (RRS) task, Table 4.5 shows the results of adding entity information in the report generation process. After introducing entity information, the recall rate is significantly improved, indicating that the model can pay attention to more key information and generate more comprehensive summaries. However, other indicators have declined, and after careful comparison, the study found that the model paid too much attention to non-core content, resulting in too long generated text and negatively affecting accuracy. Therefore, how to accurately select and embed core entity information to balance the comprehensiveness and accuracy of the summary is the key to improving model performance.

[0155] Table 4.4 Experimental results of IMC-V2 doctor-patient conversation summary task

[0156]

[0157] Table 4.5 Entity information embedding in RRS task

[0158]

[0159] V. Biomedical information extraction task

[0160] The biomedical information extraction task module, the present application mainly adopts the method based on large model fine-tuning to carry out Chinese electronic medical record named entity recognition (CNER), mainly including the following steps:

[0161] Instruction prompt design stage, design three different types of instruction prompts: simple instructions, task description and type definition, and task description, type definition and example.

[0162] Decoding scheme design stage, three decoding schemes are proposed: HTML scheme, index scheme and special symbol marking scheme.

[0163] Instruction fine-tuning stage, based on the designed instruction prompt and decoding scheme, construct the adaptive dataset, and fine-tune the large model, finally realize efficient named entity recognition on Chinese electronic medical record.

[0164] The specific implementation technical scheme of instruction prompt design and decoding scheme design stage and experimental results are as follows.

[0165] 5.1 Instruction prompt design

[0166] When fine-tuning large models, the instruction design in the data is crucial, as it directly affects the performance of the model and its understanding and execution ability for the task. As the bridge between the model and the user or environment, instructions carry task definition, expected output and related context information. This invention adopts a step-by-step progressive exploration step to explore the impact of different styles of instructions on the model's understanding ability, in order to find the best instruction format. The exploration steps are divided into three key stages: simple instructions; task description and type definition; task description, type definition and example. The complexity of these three design methods increases gradually.

[0167] First, simple instructions only contain the sentence to be predicted, such as Figure 9 As shown in the design, only the sentence to be predicted is input, and the purpose is to compare with the subsequent complex instruction design, so as to analyze whether different instructions have an impact on the performance of large model named entity recognition, and the degree of influence.

[0168] Further, the task description and type definition instruction design provides task requirements and entity type definition in addition to the input of the sentence to be predicted, and clearly defines the specific definition of the entity. This invention helps the model to clearly understand the focus of the task by this way, and guides it to focus on key information, so as to better understand the entity features and context relationship. As Figure 10 The instruction format for task description and type definition is shown.

[0169] Further, in addition to task requirements and entity type definitions, the present application also helps the model better understand the task by providing examples similar to the sentence to be predicted. Figure 11 The instruction design format is shown, which includes task description, type definition and example. These examples are designed through scripts and use vector databases to retrieve the most similar content from the data set to the input text to help the large model better identify entities and improve the accuracy and efficiency of named entity recognition.

[0170] 5.2 Decoding scheme design

[0171] In the Chinese Named Entity Recognition (CNER) task, designing a suitable decoding scheme is one of the key problems. The goal of the CNER task is to identify specified entity information (such as diseases, drugs, clinical manifestations, etc.) in electronic medical record texts, providing a foundation for medical information extraction and decision support. However, Chinese electronic medical records have varying text lengths, flexible and diverse language expressions, and contain a large number of professional terms and abbreviations, all of which affect the accuracy and robustness of entity recognition. At the same time, the particularity of the medical field requires the CNER task to understand the context and interrelationships of entities, thereby increasing the complexity of the decoding process. In addition, the present application not only focuses on identifying entities and their types, but also strives to accurately obtain the position information of these entities in the text, making the decoding task more challenging.

[0172] The present application explores three decoding schemes: HTML decoding scheme, index decoding scheme and special symbol marking decoding scheme. The inputs of these schemes remain consistent, all adopting the format of task description and type definition, to explore the effects of different decoding schemes on the CNER task.

[0173] First, the HTML decoding scheme imitates the format of an HTML file, as shown in Figure 12 This scheme instructs the model to use tags with class attributes representing entity types and entity names to highlight named entities in the sentence. Specifically, the <type: entity> tag represents the starting position of the entity, while the < / type: entity> tag represents the ending position of the entity. The marked form of entity type and entity name helps the model to clearly identify the start and end of the entity, thereby improving the accuracy of entity recognition.

[0174] In addition, the structure of the index decoding scheme is as follows Figure 13The output of this scheme first shows the original sequence of the sentence and the index of each character, and then outputs the entity, entity type, and the start and end positions of the entity. By outputting each character together with its corresponding index, the model can better understand the exact position of the entity in the sentence. In addition, considering that there may be multiple entity types in the dataset, and there may be nested relationships between different entities, when there are multiple entities of the same entity type, the entities will be separated by semicolons (;).

[0175] Finally, the structure of the special symbol tagging decoding scheme is as shown in Figure 14 The scheme converts the CNER task into a text generation task through special symbol tagging. Specifically, the model converts the task into generating a text sequence, where special markers (such as [ and ]) are used to mark entities. If the input text does not contain entities, the output text will be consistent with the input text; if the input text contains entities, special symbol markers are used to highlight these entities. In addition, in order to handle multiple entities and nested relationships between entities that may appear in the dataset, multiple entities of the same entity type in the design scheme will be separated by semicolons (;).

[0176] 5.3 Experimental results

[0177] The research content of this part uses the CMeEE-V2 dataset. This dataset is constructed according to the requirements of the fine-tuning framework and combined with different instruction schemes and decoding schemes.

[0178] CMeEE-V2 dataset: The core goal is to accurately identify and classify key entities in medical text. Based on the predefined schema, this task extracts entities closely related to medical clinics from pure medical text documents and classifies them into nine categories, including diseases, clinical manifestations, medical procedures, medical devices, drugs, bodies, departments, and medical test items. The CMeEE-V2 dataset not only has multiple entity categories, but also has a large number of nested entities, making it difficult to perform named entity recognition.

[0179] In named entity recognition, commonly used evaluation indicators mainly include accuracy (Precision), recall (Recall), and F1 value. This paper mainly uses Micro-F1 as the evaluation indicator.

[0180] (1) Performance impact of different instruction schemes

[0181] Table 5.1 presents the performance comparison of the three instruction schemes when the decoding scheme uses special symbol tagging, using the Chatglm3-6B-base base. From the strict evaluation index results, the task description and type definition design is the best, and the input only to be predicted sentence is the worst. The analysis may be due to the fact that a clear and accurate task description can help the model to clearly understand which key information it should focus on, and adjust its internal parameters and strategies accordingly. In the named entity recognition task, this guidance is particularly important, as it requires the model to accurately identify and classify entities in the text. Through the task description, the model can better understand the core requirements of this task, and thus more accurately identify entities in the subsequent processing process.

[0182] However, compared with task description and type definition, the instruction method of adding examples increases the workload but does not bring better results. This may be due to the particularity of the named entity recognition task. In the named entity recognition task, the model pays more attention to entity-level information rather than sentence-level input text. Therefore, the examples found at the sentence level may not completely match the entities in the task sentence, which may have a negative impact on the model's learning. In addition, too many examples may increase the computational burden of the model and reduce its processing speed.

[0183] Table 5.1 Performance comparison of different instruction schemes

[0184]

[0185] (2) Performance impact of different decoding schemes

[0186] On the basis of using task description and type definition as instruction prompts, the performance of three different decoding schemes on the CMeEE-V2 dataset is compared, and the results are presented in Table 5.2. Observing the results, we can find that from the perspective of strict evaluation index, the special symbol tagging decoding scheme stands out among the three methods, showing significant advantages. In contrast, the HTML decoding scheme is slightly inferior, and its performance is not satisfactory.

[0187] Table 5.2 Performance comparison of different decoding schemes

[0188]

[0189] In view of this finding, this paper analyzes that the HTML decoding scheme, when dealing with nested entities, due to the complexity of its output structure, may bring additional challenges to the model's learning process. When there is a nested relationship between entities, the HTML scheme tends to generate a more complex and complex output structure, which undoubtedly increases the difficulty of the model in capturing entity boundaries, and thus affects its overall labeling effect.

[0190] In contrast, the special symbol tagging decoding scheme demonstrates its unique advantages. By ingeniously introducing specific tagging symbols around entities, this scheme provides clearer guidance for large models when generating annotated sequences. This design not only effectively bridges the gap between sequence tagging tasks and text generation tasks, but also simplifies the annotation process to some extent. More importantly, the special symbol tagging decoding scheme performs well in handling nested entities. It adopts a strategy of marking only one entity per output sentence, and this concise and clear approach greatly alleviates the complexity problem brought by nested entities, making the overall annotation process more efficient and accurate.

[0191] The index decoding scheme has relatively moderate performance. Its advantage is that the length of the large model inference output is shorter, and the required time is less, so it can improve the efficiency of named entity recognition while maintaining relatively good performance.

[0192] (3) Comparison of performance of different bases

[0193] The difficulty of comparing the impact of different bases and different model sizes on the task of Chinese electronic medical record named entity recognition based on large models lies in finding the best balance point between different bases (such as Qwen, Baichuan2, Chatglm, etc.) and different model sizes to improve the accuracy and efficiency of entity recognition. The key lies in a comprehensive consideration of experimental design, data set selection and evaluation indicators. On the basis of controlling other influencing factors, the instruction adopts task description and type definition, and the decoding adopts special symbol tagging. The invention compares Qwen, Baichuan2 and Chatglm3 bases, and the results are shown in Table 5.3.

[0194] Table 5.3 Comparison of performance of different bases

[0195]

[0196] In the above table experiment, the performance of Qwen-7B-base, Baichuan2-7B-base and Chatglm3-6B-base in the CMeEE-V2 dataset is compared when other conditions are the same and only the base is changed. According to the inference results, it can be seen that Chatglm3-6B-base base performs better. In this paper, the performance of Chatglm3-6B base is selected from the base, and then the performance of base and chat version of Chatglm3-6B in the named entity recognition of CMeEE-V2 dataset is compared. It can be seen that the base model performs better and the reasons are as follows: first, the base model has not undergone fine-tuning for specific tasks before participating in this experiment. This "unprocessed" state allows the base model to have a relatively "pure" perspective when facing new named entity recognition tasks. Therefore, when the base model is fine-tuned on the CMeEE-V2 dataset, it can more easily capture key features in the data, thereby better adapting to the needs of the named entity recognition task.

[0197] Six, self-cognition and general domain dialogue ability

[0198] In the process of exploring the application of large language models in the biomedical field, although they perform excellently in specific tasks such as medical question answering, report generation, and information extraction, there is still a problem of insufficient self-cognition ability. For example, existing models often cannot accurately identify their own roles, ability boundaries, or information credibility during conversations, and may output incorrect or inappropriate content. In addition, trained biomedical large models focus on specific tasks but lack basic general dialogue ability, resulting in limited performance when dealing with cross-domain communication or comprehensive question answering. In summary, if users use standard task format data, the model can output standard answers, but if users engage in ordinary conversations, the model may struggle to return normal content.

[0199] To this end, the present invention explores a method to enhance the self-cognition and general domain dialogue ability of large language models, enabling them to accurately understand and limit their roles while performing biomedical tasks, improve the credibility of external output content, and support natural interaction with users.

[0200] 1. Self-cognition mechanism: By designing role positioning instruction data, through question and answer, constantly asking the model identity questions, guiding the model to think about identity, and setting answers in the data as the identity information set by the model. Disperse self-cognition data into all training data to ensure the accuracy of the model's role positioning throughout the process.

[0201] 2. General domain dialogue capability: The present application collects and collates the most authoritative general domain dialogue data set MOSS at present, and removes the noise data such as website, garbled code, non-standard pseudo code and the like contained in the data set, so as to ensure the stability during model training; at the same time, the self-cognition information in the data set is removed, the MOSS data set hopes to set the cognition of the model as "MOSS agent", if the self-cognition information is contained, it will affect the identity cognition ability of the present application. After screening, about 400000 high-quality data are reserved as the general domain dialogue capability training data of the model, so that the model can well reply to the dialogue even if it receives the inquiry of non-specific task.

[0202] The self-cognition data and the processed MOSS data format are shown below.

[0203] Seven, multi-task large language model fine-tuning for biomedical field

[0204] Through the fine-tuning method exploration of the above different biomedical tasks, the present application finally applies the method technology of different tasks to the existing biomedical data set, constructs the final version of the instruction data, and performs full-parameter instruction fine-tuning training on the GLM-4-9B base, and obtains the final biomedical large model.

[0205] 7.1 Training data

[0206] In order to maximize the use of existing biomedical text resources, the present application focuses on integrating comprehensive open source data sets covering English and Chinese. The data sources mainly include the following three aspects: existing English and Chinese biomedical shared task data sets, data resources that have been used to train existing biomedical large models, and exercise contents on traditional Chinese medicine examination websites. For specific data set information, see Table 7.1.

[0207] Table 7.1 Training set information statistics

[0208]

[0209] The data set of the application covers various tasks, such as biomedical information extraction tasks (including named entity recognition, relation extraction), medical intelligent question and answer tasks (fill-in-the-blank question and answer), and medical report generation tasks. Among them, the data set used for the named entity recognition task is bc5cdr-chem, bc5cdr-dis, CmeEE, chemNER, NCBIdis; the data set used for the relation extraction task is bc5cdr, biorelex, CMEie_v2, DDI_corpus; the data set of the medical intelligent question and answer task includes choice_med_qa_en, choice_pubmed_qa. Choice_medqa_zh; and the data set used for the medical report generation task is HQS, RRS, IMCS_V2.

[0210] In order to construct a data set suitable for instruction fine-tuning, the application designs a special instruction template for each task data according to the characteristics of each task and the strategies explored above.

[0211] The question and answer and dialogue task instruction template directly uses the original question as the input of the model and the corresponding answer as the output, aiming to help the model understand the core goal of the task.

[0212] The named entity recognition (NER) task instruction template uses various decoding strategies designed by the application. Not only the scheme of directly answering the entity is used, but also the strategy based on special markers is designed to explicitly mark the position of the specific entity in the input to improve the model's ability to identify the position of the entity.

[0213] Further, in order to enhance the comprehensive ability of the model, the application additionally generates role-playing data and multi-task question and answer data. For different tasks, the model plays a specific role, such as "an experienced natural language processing expert" or "an information extraction expert". At the same time, the application randomly selects one task ( ) to construct a multi-task question and answer data set to simulate task interaction in complex scenarios.

[0214] Further, in order to improve the performance of the model in self-awareness scenarios, the application artificially constructs about 1400 pieces of self-awareness data to ensure that when the model receives a question like "Who are you?", it can generate an accurate answer according to the set role. And in order to improve the general question and answer ability of the model, the application also adds a part of the MOSS data set as a supplement.

[0215] The format of all instruction data is as shown in Figures 15-17 .

[0216] 7.2 Model training

[0217] The present application aims to explore the potential of large language models (LLM) in bilingual biomedical natural language processing tasks, and selects the current advanced GLM-4-9B model as the base model for full-parameter training. The training of the model is mainly carried out using the ms-swift framework, and the link is as follows: https: / / github.com / modelscope / ms-swift?tab=readme-ov-file. The main reasons for choosing GLM-4-9B as the pre-training model include the following points: 1) excellent performance in bilingual tasks: GLM-4-9B performs well in both English and Chinese tasks, and can meet the diverse needs of bilingual biomedical tasks. 2) moderate model size, high training efficiency: GLM-4-9B has a moderate parameter size, which can guarantee task performance while having high training efficiency, and meets the existing limitations of computing resources. 3) wide coverage of pre-training data: GLM-4-9B's pre-training dataset covers the general data required for multi-task, providing a solid foundation for cross-task applications. Through training based on GLM-4-9B, it is expected to fully tap its potential in the bilingual biomedical field. For specific training parameters, please refer to Table 7.2.

[0218] Table 7.2 Training parameter settings

[0219]

[0220] The following is an explanation of the role of each hyperparameter:

[0221] 1. Learning_rate (learning rate): The learning rate determines the magnitude of the model's adjustment at each parameter update. A smaller learning rate (such as 1e-05) is usually used to avoid rapid convergence or oscillation, helping the model find the optimal solution more stably. Small learning rates are suitable for training processes that require fine-tuning.

[0222] 2. Gradient_accumulation_steps (gradient accumulation steps): Gradient accumulation refers to calculating gradients over multiple mini-batches, but not immediately updating model weights, but accumulating a certain number of mini-batches before updating the weights again. By setting it to 16, it means that the gradient is accumulated once every 16 mini-batches, and then the weight is updated again. This can effectively use the video memory, and also increase the batch size, improving the performance of the model.

[0223] 3. Batch_size (batch size): The batch size determines the number of training samples used in each iteration. A batch size of 1 is usually used for settings that require less video memory, or for tasks with long sequence lengths (such as long text in NLP tasks), although the training process will be slower, but it can reduce the requirements for hardware resources.

[0224] 4. dtype (Data Type): The data type specifies how the value is represented. bf16 (bfloat16) is a floating-point representation format, commonly used in deep learning to accelerate computation and reduce memory usage. Compared to the standard float32, bf16 can maintain higher computational precision while reducing GPU memory requirements, making it a commonly used setting when training large models.

[0225] 5. Seed (Random Seed): The random seed is used to initialize the random number generator, ensuring the repeatability of the experiment. By fixing the seed, initialization, data shuffling, and other operations during each training session will yield the same results, which helps in comparing and analyzing experimental results.

[0226] These hyperparameter settings take into account training stability, computational efficiency, and resource constraints, aiming to optimize performance and effectiveness during the training process.

[0227] The final model training tasks included biomedical information extraction (including Named Entity Recognition (NER) and Relation Extraction (RE), medical intelligent question answering (QA), and medical report generation. The experimental results on the medical information extraction and medical intelligent question answering tasks are as follows:

[0228]

[0229] The model's test results on the medical report generation task test set after training are as follows:

[0230]

[0231] The self-awareness and general-domain dialogue tasks are primarily designed to stimulate the model's cognitive and conversational abilities; therefore, quantitative performance analysis will not be conducted through experiments. Instead, qualitative performance analysis will be performed through human dialogue testing. Dialogue test samples are shown below. Figure 18 As shown, the present invention has excellent performance in both identity recognition and general domain dialogue capabilities.

[0232] VIII. Conclusion and Outlook

[0233] This invention has achieved remarkable results through in-depth research on key technologies of large language models for the biomedical field. It has successfully constructed a multilingual, multi-task biomedical large model with excellent performance, covering multiple tasks such as multilingual biomedical intelligent question answering, doctor-patient dialogue, medical report generation, and biomedical information extraction.

[0234] Through contextual learning, instruction fine-tuning, and thinking chain methods, the application effectively improves the response quality and generalization ability of the model in Chinese medical question and answer tasks. In the medical report generation task, the application uses named entity information embedding and spelling correction optimization methods to significantly improve the summary generation performance of the model. Meanwhile, in the biomedical information extraction task, different instructions and decoding strategies based on large models are explored, which not only can accurately predict entity mentions, but also can mark the specific position of entities in the text.

[0235] Finally, the application integrates a large number of English and Chinese open source datasets, covers multiple tasks, and further enhances the generalization ability and entity position recognition ability of the model through carefully designed instruction templates and decoding strategies. In addition, the application also constructs role-playing data, multi-task question and answer data, and self-cognition data, further improving the comprehensive ability of the model.

[0236] Looking forward to the future, the application will continue to optimize and expand the application range of biomedical large models, explore innovative technical methods to cope with the growing biomedical text data and increasingly complex application requirements. The application hopes to cooperate deeply with medical institutions to apply the model to the actual clinical environment, evaluate its effectiveness and feasibility in the real world. At the same time, in the process of model development and application, it will strictly follow the ethical norms and privacy protection standards to ensure the safety and privacy of patient data and strengthen the security of the model. Through continuous technological innovation and interdisciplinary cooperation, the application expects to promote the development of biomedical large model technology and provide stronger language processing support for biomedical research and clinical practice.

[0237] It should be noted that the terms "first", "second" and the like in the specification and claims of the application and in the above description of the drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

Claims

1. A method for training a multi-task large language model in the biomedical field, characterized in that, The model is used to automatically select and execute tasks such as medical intelligent question answering, medical report generation, or biomedical information extraction based on the task type of the input question. The method includes: A training set for an instruction dataset for a medical intelligent question answering task is constructed. The instruction dataset has a first instruction format, which includes a data instruction format based on context learning and a data instruction format based on thought chain. A training set for a second instruction dataset for medical report generation tasks is constructed. The training set data of the second instruction dataset has a second instruction format, which includes a data instruction format enhanced based on entity information embedding and a data instruction format enhanced based on spell correction. A training set for a third instruction dataset for biomedical information extraction tasks is constructed. The training set data of the third instruction dataset has a third instruction format, which includes a data instruction format based on task description and type definition, and a data instruction format based on special symbol marking. The large language model is trained using the training set. The large language base model is the GLM-4-9B model, and its hyperparameters are as follows: Learning rate: 1e-05, gradient accumulation steps: 16, batch size: 1, data type: bf16, random seed: 42; The training set for constructing the instruction dataset for medical intelligent question answering tasks includes: The first training data is constructed by taking one data point from the training set of the first instruction and the first dataset, or by taking one data point from the training set of the first instruction and the first dataset and a few selected samples from the first dataset as a first training data point in the training set. The first training data points are obtained corresponding to each data point in the training set of the first dataset. The first training data points are data in a context-based data instruction format. Using the large language model, the reasoning corresponding to a piece of data in the training set of the first dataset is generated through the second instruction. The second instruction, the piece of data, and the reasoning are constructed into a piece of data in the training set to form a second training data in the training set. Each piece of second training data corresponding to each piece of data in the training set of the first dataset is obtained. The second training data is data in the data instruction format based on the thought chain. Wherein: the first instruction includes "Answer in the format of "Answer: Options"; the second instruction includes "Show the reasoning process".

2. The method for training a multi-task large language model for the biomedical field according to claim 1, characterized in that, It also includes building a MOSS dataset for general domain dialogues, which automatically selects and executes general domain dialogues based on the task type of the input question.

3. The method for training a multi-task large language model for the biomedical field according to claim 1 or 2, characterized in that, It also includes constructing a model self-awareness dataset, wherein the model self-awareness data is distributed and inserted into each of the training sets.

4. The method for training a multi-task large language model for the biomedical field according to claim 1, characterized in that, in, The training set for constructing the second instruction dataset for the medical report generation task includes: Generate an entity category from a data point in the training set of the second dataset, construct a third training data point from the data point and the entity category, and obtain each third training data point corresponding to each data point in the training set of the second dataset. The third training data point is data in a data instruction format based on entity information embedding. Using the large language model, a data point with spelling error correction is generated from a data point in the training set of the second dataset via the fourth instruction. The fourth instruction, the data point, and the data point with spelling error correction are then used to construct a fourth training data point in the training set. Each fourth training data point corresponding to each data point in the training set of the second dataset is obtained. The fourth training data point is data in a data instruction format based on spelling correction enhancement.

5. The method for training a multi-task large language model for the biomedical field according to claim 4, characterized in that, in: The entity category in a single data point of the training set of the second dataset is generated, including extracting the entity category using a medical entity extraction tool; the content of the fourth instruction includes "spelling error correction".

6. The method for training a multi-task large language model for the biomedical field according to claim 1, characterized in that, in, The training set for constructing a third instruction dataset for biomedical information extraction tasks includes: The fifth instruction and a single data point from the training set of the third dataset are used to construct a fifth training data point in the training set. Each fifth training data point corresponding to each data point in the training set of the third dataset is obtained. The fifth training data is data in a data instruction format based on task description and type definition. The content of the fifth instruction includes task description and type definition. Multiple text sequences of a single data point in the training set of the third dataset are generated, and each text sequence has an entity marked with a special symbol. The single data point and the text sequences are constructed as a sixth training data point in the training set. Each sixth training data point corresponding to each data point in the training set of the third dataset is obtained. The sixth training data point is data in a data instruction format based on special symbol marking.

7. An electronic device, the electronic device comprising: One or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method of any one of claims 1-6.

Images