Dunhuang vertical field large model training method, system, medium and equipment

Abstract

The invention discloses a Dunhuang vertical field large model training method and system, a medium and equipment, and belongs to the field of artificial intelligence and the field of culture. According to the method, Dunhuang vertical domain knowledge such as Dunhuang dictionary, Dunhuang introduction and Dunhuang 18 speaking is organized to form Dunhuang instruction fine adjustment data and a dictionary type vector knowledge base of keyword-explanation, and the Dunhuang instruction fine adjustment data and the keyword-explanation dictionary type vector knowledge base are obtained through the steps of continuous pre-training, supervised fine adjustment, plug-in of the vector knowledge base and the like. And a big Dunhuang vertical field model oriented to cultural tourism and language experts is trained. And a PhiloCoP framework is further provided, so that application of a large model in the Dunhuang vertical field is facilitated. According to the method, a more convenient knowledge query and auxiliary research tool is provided for Dunhuang enthusiasts and archaeologists, the Dunhuang culture can be more conveniently propagandized in a larger range, and the method can be applied to the field of ancient linguistics research and tourism scenes.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and culture, and in particular relates to a Dunhuang vertical field large model training method, system, medium and equipment. Background Art

[0002] Recent developments in big language models have spawned many vertical big models customized for different downstream applications. In Chinese patent CN117253576A, it is introduced that big language models are used in the medical field for auxiliary diagnosis, answering health-related questions and psychological counseling; in Chinese patent CN117131173A, it is introduced that big language models are used in the legal field for legal question-answering, generating legal documents and predicting legal judgments; in Chinese patent CN117112760A, it is also introduced that big language models are used in other fields, such as finance and education.

[0003] There are already some projects that have successfully used language models pre-trained on ancient language corpora to perform specific tasks. In the prior art, Assael et al. studied the text restoration, chronological attribution, and geographical attribution problems of ancient Greek inscriptions. Yoo et al. trained a BERT model on ancient Korean corpora to complete chronological attribution, topic classification, named entity recognition, and summary retrieval tasks. However, these models were all developed before the development of large language models. There is currently no large language model based on the vertical field of ancient Chinese research. The limitations of training large models in the vertical field of Dunhuang are: the lack of high-quality training datasets, the complexity of ancient Chinese, and the fragmentation of ancient language research. Summary of the invention

[0004] The purpose of the present invention is to solve the problems existing in the prior art and provide a method for training a large model in the vertical field of Dunhuang. The present invention organizes the knowledge related to Dunhuang, forms a Dunhuang large model instruction fine-tuning data set and a "keyword-explanation" dictionary knowledge base; through pre-training, supervised fine-tuning, external vector knowledge base and other steps, a Dunhuang large model for cultural tourism and language experts is trained.

[0005] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a Dunhuang vertical domain large model training method, which comprises the following steps:

[0007] S1. Continue pre-training the base large language model on unlabeled texts in the general and Dunhuang fields to obtain a large language model after continued pre-training;

[0008] S2. According to the preset downstream tasks, Dunhuang instruction fine-tuning data of the corresponding field is constructed, and general instruction fine-tuning data is obtained. The large language model after continuing pre-training is fine-tuned on the Dunhuang instruction fine-tuning data and the general instruction fine-tuning data to obtain the large language model after instruction fine-tuning; among them, there are nine downstream tasks, namely text repair, splicing, age attribution, named entity recognition, topic modeling, judgment, question answering, analysis, and reasoning;

[0009] S3. Plug the pre-constructed vector knowledge base into the large language model after instruction fine-tuning to obtain the final trained Dunhuang vertical domain large model.

[0010] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0011] As a preferred embodiment of the above-mentioned first aspect, in step S1, a data set containing four types of documents, namely, classics, history, philosophy, and collections, is obtained, and the data set covers public data, original museum collections, research papers, academic publications, and professional documents. After the documents in the data set are converted into text format, the documents in the text format are cleaned and deduplicated, and then the documents after data cleaning and deduplication are merged into a text file to form unlabeled text in the Dunhuang field; wherein, data cleaning and deduplication include removing redundant spaces, line breaks, headers, footers, illustrations, tables, formulas, annotation symbols, references, and links in the documents in the text format.

[0012] As a preferred embodiment of the first aspect, in step S2, Dunhuang instruction fine-tuning data is generated by three methods including Self-QA, Self-Instruct and manual construction.

[0013] As a preferred embodiment of the above-mentioned first aspect, in step S2, the instruction fine-tuning adopts LoRA fine-tuning.

[0014] As a preferred embodiment of the first aspect, in step S3, the specific process of constructing the vector knowledge base is as follows:

[0015] S31, sorting out the dictionaries and materials related to Dunhuang studies to form a Dunhuang knowledge base, and dividing the texts in the Dunhuang knowledge base into blocks to obtain text blocks containing Dunhuang field information;

[0016] S32, using a word vector model to vectorize the text block to obtain a vector knowledge base;

[0017] S33. Through the LangChain framework, the vector knowledge base is connected to the large language model after instruction fine-tuning to obtain the Dunhuang vertical field large model.

[0018] In a second aspect, the present invention provides a question-answering system for a large model of a vertical domain of Dunhuang, comprising:

[0019] A model acquisition module, used to acquire a Dunhuang vertical domain large model trained according to a Dunhuang vertical domain large model training method described in any solution of the first aspect above;

[0020] The first processing module is used to input the user's question into the Dunhuang vertical domain big model, perform named entity recognition, extraction and classification on the user's question, and obtain various named entities appearing in the user's question;

[0021] The second processing module is used to input user questions and various named entities appearing in user questions into the Dunhuang vertical domain big model, infer the implicit relationship between the named entities, and obtain the relationship network;

[0022] The third processing module is used to input the user questions and various named entities and relationship networks appearing in the user questions into the Dunhuang vertical domain big model, translate the classical Chinese in the user questions, and obtain the modern Chinese translation of the classical Chinese in the user questions;

[0023] The result acquisition module is used to input user questions, various named entities appearing in user questions, relationship networks and modern Chinese translations of classical Chinese texts in user questions into the Dunhuang vertical domain big model to obtain answers to user questions.

[0024] As a preferred embodiment of the second aspect, the question-answering system is visualized to obtain a user interface of the Dunhuang vertical domain large model with the PhiloCoP framework added;

[0025] The user interface includes a clear history function and a retry function. The retry function is to regenerate answers to questions input by the user. If the user clicks the clear history function and does not continue to ask questions, the Dunhuang vertical field large model does not retain the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and does not continue to ask questions, the Dunhuang vertical field large model retains the user's historical questions and the answers to the historical questions; if the user clicks the clear history function and continues to ask questions, the Dunhuang vertical field large model does not retain the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions do not refer to the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and continues to ask questions, it is a multi-round dialogue mode, the Dunhuang vertical field large model retains the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions will refer to the user's historical questions and the answers to the historical questions.

[0026] Furthermore, the specific process of the visualization processing is as follows: the icon, name and corresponding instructions for use of the Dunhuang vertical field model are added to the original interactive interface in advance. When the user clicks the send function of the original interactive interface, the input box of the original interactive interface obtains the question entered by the user, visualizes the question entered by the user in the dialog box of the original interactive interface and outputs the answer.

[0027] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a large model training method for a vertical field of Dunhuang as described in any of the schemes of the first aspect is implemented.

[0028] In a fourth aspect, the present invention provides a computer electronic device, comprising a memory and a processor;

[0029] The memory is used to store computer programs;

[0030] The processor is used to implement a Dunhuang vertical field large model training method as described in any of the solutions of the first aspect above when executing the computer program.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention proposes a method, system, medium and equipment for training a large model of a vertical field of Dunhuang. The training method of the present invention can help Dunhuang enthusiasts and tourists, answer questions related to Dunhuang culture, and be put into use in the tourism industry; at the same time, it can help Dunhuang archaeologists in their research, facilitate and quickly find information, greatly save time and energy, and improve research efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the steps of the present invention;

[0034] Figure 2 It is a flow chart of the overall solution of the present invention;

[0035] Figure 3 is the Self-QA flow chart of the present invention;

[0036] Figure 4 is a Self-Instruct flow chart of the present invention;

[0037] Figure 5 It is a visualization interface diagram of the Dunhuang vertical domain large model of the present invention;

[0038] Figure 6The graphs are the results of the Dunhuang vertical domain large model of the present invention on different tasks; (a) is the result graph on the named entity recognition task; (b) is the result graph on the analysis task; (c) is the result graph on the reasoning task;

[0039] Figure 7 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0040] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0041] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0042] In a preferred embodiment of the present invention, a method for training a large model of Dunhuang vertical domain is provided, which belongs to the combination of artificial intelligence and cultural fields. In view of the current situation that there is no large model of Dunhuang vertical domain and the general large language base model has many deficiencies in answering most of the Dunhuang vertical domain questions due to the lack of sufficient training data, such as the inability to understand relevant proper nouns, insufficient professionalism in generated content, and serious hallucinations in answers, the present invention uses Qwen1.5-7B-Chat as the base large language model through pre-training, supervised fine-tuning, external vector knowledge base and other steps, provides a large amount of Dunhuang domain data, and trains a large model of Dunhuang vertical domain that can be used for both Dunhuang amateurs or tourists and Dunhuang researchers.

[0043] The overall solution process is as follows Figure 2As shown. First, in order to enable the Dunhuang vertical field large model to master more Dunhuang field knowledge, the present invention collects and organizes four types of data and uses Qwen1.5-7B-Chat as the base large language model, and continues pre-training on its basis. Then, instruction fine-tuning data is constructed for 9 downstream tasks, and instruction fine-tuning is performed to enable the Dunhuang vertical field large model to have the ability to answer relevant questions. The Dunhuang vertical field large model can consult relevant information through the external vector knowledge base. Among them, the implementation method of the base large language model Qwen1.5-7B-Chat belongs to the prior art. For details, please refer to the prior art literature: Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., T An, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., & Zhu, T. (2023). Qwen Technical Report. *arXiv preprint arXiv:2309.16609*.

[0044] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned Dunhuang vertical domain large model training method includes the following steps S1 to S3. The specific implementation process is described in detail below.

[0045] 1. Continue pre-training

[0046] S1. Continue pre-training the base large language model on unlabeled texts in the general and Dunhuang fields to obtain a large language model after continued pre-training.

[0047] It should be noted that in step S1 of the present invention, continuing pre-training is the first step in training the Dunhuang vertical domain large model, allowing the base large language model to continue learning on the unlabeled texts in the general and Dunhuang fields, so that the base large language model can better understand and adapt to the language structure and characteristics of the specific field. This process provides the Dunhuang vertical domain large model with specific aspects of the corpus that can be learned.

[0048] Specifically, in step S1 of the present invention, a data set containing four types of documents, namely classics, history, philosophy and collections, is obtained, wherein the data set covers public data, original museum collections, research papers, academic publications and professional documents, and after converting the documents in the data set into a text format, the documents in the text format are cleaned and deduplicated, and then the documents after data cleaning and deduplication are merged into a text file to form unlabeled text in the Dunhuang field; wherein data cleaning and deduplication include removing redundant spaces, line breaks, headers, footers, illustrations, tables, formulas, annotation symbols, references and links in the documents in the text format.

[0049] In this embodiment, it is first necessary to organize the Dunhuang field corpus for training the large model of the Dunhuang vertical field, put it into a text file to form the PhiloCorpus-ZH corpus, and use it as the unlabeled text in the Dunhuang field. The general unlabeled text and the PhiloCorpus-ZH corpus are then organized into a text file with a total of 1.23 million data, of which 1 million data are from general unlabeled text and 230,000 data are from the PhiloCorpus-ZH corpus. Then, the configuration file for continuing pre-training of the large model in the prior art Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. 2024. Llamafactory: Uniffed efffcient ffne-tuning of 100+language models.arXiv preprint arXiv: 2403.13372 is used to continue pre-training the base large language model to obtain a large language model after continued pre-training.

[0050] Further, in this embodiment, the PhiloCorpus-ZH corpus includes four categories of documents: classics, history, philosophy, and collections, provided by the School of Literature of Zhejiang University, covering public data, original museum collections, research papers, academic publications, and professional documents. These files are in PDF format, and the PaddleOCR tool in the prior art is used to convert these files into text format. Extensive data cleaning and deduplication were then carried out to ensure quality, including removing extra spaces, line breaks, headers, footers, illustrations, tables, formulas, annotation symbols, references, and links. Finally, all files were merged into a text file to form the PhiloCorpus-ZH corpus.

[0051] Furthermore, in this embodiment, in order to prevent the general ability of the base large language model from being significantly reduced, it is also necessary to organize general unlabeled texts. Specifically, the Shusheng Wanjuan corpus, the Wikipedia Chinese data set and SkyPile-150B in the prior art are used to obtain general unlabeled texts.

[0052] 2. Instruction fine-tuning

[0053] S2. Construct the Dunhuang instruction fine-tuning data of the corresponding field according to the preset downstream tasks, obtain the general instruction fine-tuning data, and continue to fine-tune the large language model after pre-training on the Dunhuang instruction fine-tuning data and the general instruction fine-tuning data to obtain the large language model after instruction fine-tuning; among them, there are nine downstream tasks, namely text repair, splicing, age attribution, named entity recognition, topic modeling, judgment, question and answer, analysis, and reasoning.

[0054] It should be noted that in step S2 of the present invention, instruction fine-tuning is to enable the large language model after continued pre-training to have the ability to answer relevant questions. The data used for instruction fine-tuning is divided into two parts, one part is the Dunhuang instruction fine-tuning data constructed according to the preset downstream task, and the other part is the general instruction fine-tuning data. Therefore, it is first necessary to construct the Dunhuang instruction fine-tuning data according to the downstream task.

[0055] In this embodiment, three methods, Self-QA, Self-Instruct and manual construction, are used to fine-tune the data. Specifically, Self-QA is a self-supervised learning method that does not require labeled data and can generate questions and corresponding answers based on the provided text. The implementation process of Self-QA is as follows: Figure 3 As shown in Figure 1, we can call GPT4’s API to generate instructions based on the Dunhuang Dictionary, then input the instructions and the corresponding documents and call GPT4 to generate the answer to the question. The process of self-instruction to generate instructions and fine-tune data is as follows: Figure 4 As shown, the instruction generation code provided by the BELLE tool in the prior art is used. First, seed data needs to be prepared. The quality of the generated data is closely related to the quality of the provided seed data, so first a high-quality seed data set with comprehensive content needs to be compiled. There are three sources of high-quality seed data sets, namely, manually written data, high-quality data in Self-QA, and high-quality data screened out after preliminary Self-Instruct. Finally, the Self-Instruct operation is performed with a high-quality seed data set to generate a large amount of instruction data. It should also be noted that the implementation methods of Self-QA and Self-Instruct belong to the prior art and are not described in detail in the present invention.

[0056] It should be noted that in step S2 of the present invention, LoRA fine-tuning is specifically adopted to perform instruction fine-tuning on the Dunhuang instruction fine-tuning data and the general instruction fine-tuning data for the large language model after continued pre-training. The implementation method thereof also belongs to the prior art and will not be described in detail.

[0057] In this embodiment, the ratio of the two parts of data used for instruction fine-tuning is 1:1, with a total of 48278 data. The general instruction fine-tuning data sources include Stanford Alpaca (zh), BELLE 2M (zh), and Fireffy 1.1M 791 (zh). The fine-tuning also uses the instruction fine-tuning file in LLaMA-Factory, and the specific parameter settings are shown in Table 1. Among them, the implementation method of StanfordAlpaca(zh) comes from the existing technology Cui, Y., & Yao, X. (2024). Rethinking LLM language adaptation: A case study on Chinese Mixtral. arXiv preprint arXiv: 2403.01851. https: / / arxiv.org / abs / 2403.01851; the implementation method of BELLE 2M(zh) comes from the existing technology BelleGroup. (2023). train_2M_CN dataset [Data set]. Hugging Face. https: / / huggingface.co / datasets / BelleGroup / train_2M_CN; the implementation method of Fireffy1.1M 791(zh) comes from the existing technology YeungNLP. (2023). firefly-train-1.1M dataset [Data set]. HuggingFace. https: / / huggingface.co / datasets / YeungNLP / firefly-train-1.1M.

[0058] Table 1. Instruction fine-tuning parameter settings

[0059] parameter set up Fine-tuning Type LoRA Cutoff Length 2048 Learning Rate 1e-4 Training Epoch 8 Batch Size 32 Optimizer adamw_torch LoRAγ 8 LoRA α 16 LoRA Dropout 0.1 LoRA Target All

[0060] 3. External Vector Knowledge Base

[0061] S3. Plug the pre-constructed vector knowledge base into the large language model after instruction fine-tuning to obtain the final trained Dunhuang vertical domain large model.

[0062] In the present invention, in order to improve the accuracy of generated content and reduce hallucinations, additional knowledge is provided for the Dunhuang vertical field large model by means of an external vector knowledge base. The principle of the external vector knowledge base is to use an additional database as input, and to retrieve knowledge in this knowledge base according to the user's questions, and to form prompt words according to the search content and the question splicing, and to input the large language model after the instruction is fine-tuned to generate answers. However, since the knowledge base is too large and exceeds the word limit that the large language model can read at one time, the data can be divided into several pieces, and only the items related to the question are read out each time, as the input of the large language model after the instruction is fine-tuned. Then you only need to find the most relevant fragments in the database to the question, and vector retrieval is used here to vectorize all documents and keywords and store them in the vector database. Finally, through the LangChain framework, the vector knowledge base is connected to the large language model after the instruction is fine-tuned. After adding the external vector knowledge base, the effect of the large language model after the instruction is fine-tuned has been significantly improved, reducing the situation of irrelevant answers and hallucinations.

[0063] It should be noted that, in step S3 of the present invention, the specific process of constructing the vector knowledge base is as follows:

[0064] S31, sorting out the dictionaries and materials related to Dunhuang studies to form a Dunhuang knowledge base, and dividing the texts in the Dunhuang knowledge base into blocks to obtain text blocks containing Dunhuang field information;

[0065] S32, using a word vector model to vectorize the text block to obtain a vector knowledge base;

[0066] S33. Through the LangChain framework, the vector knowledge base is connected to the large language model after instruction fine-tuning to obtain the Dunhuang vertical field large model.

[0067] It should be noted that, as an ideographic text, one Chinese character can represent multiple meanings according to the context and grammar. There are also some grammars that are not found in modern Chinese in ancient Chinese, such as inversion and part of speech conversion. In addition, religious and cultural factors cause significant differences in naming habits between ancient and modern Chinese. Therefore, linguists usually conduct a comprehensive analysis of the above three points before conducting subsequent research. In the present embodiment, in order to imitate the analysis process of linguists when studying ancient Chinese literature, a question-answering system of a Dunhuang vertical field large model including the PhiloCoP framework is also designed, and the Dunhuang vertical field large model is used to solve the problem step by step to help the Dunhuang vertical field large model complete the task.

[0068] like Figure 7 As shown, a question-answering system for a large model of the Dunhuang vertical domain specifically includes:

[0069] A model acquisition module, used to acquire a Dunhuang vertical domain large model trained by a Dunhuang vertical domain large model training method in the above embodiment;

[0070] The first processing module is used to input the user's question into the Dunhuang vertical domain big model, perform named entity recognition, extraction and classification on the user's question, and obtain various named entities appearing in the user's question;

[0071] The second processing module is used to input user questions and various named entities appearing in user questions into the Dunhuang vertical domain big model, infer the implicit relationship between the named entities, and obtain the relationship network;

[0072] The third processing module is used to input the user questions and various named entities and relationship networks appearing in the user questions into the Dunhuang vertical domain big model, translate the classical Chinese in the user questions, and obtain the modern Chinese translation of the classical Chinese in the user questions;

[0073] The result acquisition module is used to input user questions, various named entities appearing in user questions, relationship networks and modern Chinese translations of classical Chinese texts in user questions into the Dunhuang vertical domain big model to obtain answers to user questions.

[0074] It should be noted that, in the present invention, the above-mentioned question-answering system can be visualized to obtain a user interface of the Dunhuang vertical domain large model with the PhiloCoP framework added.

[0075] The above-mentioned user interface includes a clear history function and a retry function. The retry function is to regenerate the answers to the questions input by the user. If the user clicks the clear history function and does not continue to ask questions, the Dunhuang vertical field large model does not retain the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and does not continue to ask questions, the Dunhuang vertical field large model retains the user's historical questions and the answers to the historical questions; if the user clicks the clear history function and continues to ask questions, the Dunhuang vertical field large model does not retain the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions do not refer to the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and continues to ask questions, it is a multi-round dialogue mode, the Dunhuang vertical field large model retains the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions will refer to the user's historical questions and the answers to the historical questions.

[0076] The specific process of the above-mentioned visualization processing is as follows: add the icon, name and corresponding instructions for use of the Dunhuang vertical field model on the original interactive interface in advance. When the user clicks the send function of the original interactive interface, the input box of the original interactive interface obtains the question entered by the user, visualizes the question entered by the user in the dialog box of the original interactive interface and outputs the answer.

[0077] In the visualization processing of this embodiment, the original interactive interface of Qwen is used as the original interactive interface. After modification is made on the basis of the original interactive interface, the icon, name and corresponding instruction text of the Dunhuang vertical domain model are added, and then the visualized user interface of the Dunhuang vertical domain model with the PhiloCoP framework added can be obtained, such as Figure 5 As shown. After the user enters the question he wants to know the answer to in the input box and clicks send, the question will appear in the dialog box above, and then the answer will be output. In addition, the user interface designed by the present invention also has a history clearing function and a retry function. If you continue to ask questions without clearing the history, it will be a multi-round dialogue mode. The Dunhuang vertical field model will retain the information of previous user questions. After clearing the history, the memory of the previous question information will be deleted, so the same question may have different answers before and after clearing the history.

[0078] In order to better demonstrate the specific implementation and technical effects of the present invention, a Dunhuang vertical domain large model training method shown in steps S1 to S3 in the above preferred implementation method is applied to a specific example.

[0079] Example

[0080] The Dunhuang vertical domain big model shows significant performance improvements on various linguistic tasks. In Table 2, the Dunhuang vertical domain big model is benchmarked against other competitive big models using a benchmark designed based on downstream tasks. "N / A" indicates that the big model cannot solve the linguistic specific task, that is, it refuses to answer or randomly generates answers. For text repair and age attribution tasks, the smaller the score, the better the performance, while in other tasks, the larger the score, the better the performance. It can be seen that in the text repair and age attribution tasks, other open source big models are basically unable to complete the tasks, while the Dunhuang vertical domain big model of the present invention shows significant enhancements, especially in tasks centered on generation and retrieval. The PhilCoP architecture enables most large language models to achieve performance improvements on their respective tasks. The present invention uses PhilCoP to reconstruct the original document to eliminate the lexical confusion inherent in ancient Chinese grammar and explicitly extract key information through named entity recognition. This method can improve the performance of models that are not trained on domain datasets, except Baichuan2, because it is insufficient to handle classical Chinese named entity recognition and translation.

[0081] Table 2. Test results of Dunhuang vertical domain large model and other large models

[0082]

[0083] In this example, three base large language models are used respectively, and the GPT-4o judgment success rate is used as an evaluation indicator to evaluate the effect of the Dunhuang vertical domain large model on named entity recognition, analysis and reasoning tasks. The results are as follows: Figure 6 As shown, the higher the winning rate, the stronger the ability.

[0084] In summary, the present invention generates a large amount of high-quality data in the vertical field of Dunhuang by selecting general reading materials and professional documents in the vertical field of Dunhuang, and combines general corpus to form unlabeled text for continued pre-training of the base large language model; the present invention constructs Dunhuang instruction fine-tuning data covering professional scientific research and general cultural and tourism scenes, specifically covering the fields of Dunhuang history, stories, culture, economy, science, etc., for supervised instruction fine-tuning of the large language model after continued pre-training; the present invention constructs a vector knowledge base with "keyword-explanation" dictionary knowledge, documents and keywords, and through the plug-in vector knowledge base, keyword retrieval is performed on the questions input to the large language model, and the knowledge related to the questions is merged into prompt words as new inputs of the large language model, thereby enhancing the accuracy and professionalism of the content generated by the large language model; in addition, the present invention proposes the PhiloCoP framework, which follows the thinking chain of bibliographers and gradually helps the large language model to parse the vocabulary and grammar of ancient Chinese. This method shows significant improvements in multiple downstream tasks; finally, the present invention provides a user interface to facilitate the use of the large model in the vertical field of Dunhuang, display the input and output of the question, and can perform sending, clearing history and retry operations.

[0085] It should also be noted that the Dunhuang vertical field large model training method in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the Dunhuang vertical field large model training method provided in the above embodiment, which includes a memory and a processor;

[0086] The memory is used to store computer programs;

[0087] The processor is used to implement a Dunhuang vertical field large model training method in the above embodiment when executing the computer program.

[0088] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0089] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the Dunhuang vertical field large model training method provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, it can implement the Dunhuang vertical field large model training method in the above embodiment.

[0090] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.

[0091] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0092] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0093] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A Dunhuang vertical domain large model training method, characterized in that: The following steps are involved: S1. Continue pre-training the base large language model on unlabeled texts in the general and Dunhuang fields to obtain a large language model after continued pre-training; S2. According to the preset downstream tasks, Dunhuang instruction fine-tuning data of the corresponding field is constructed, and general instruction fine-tuning data is obtained. The large language model after continuing pre-training is fine-tuned on the Dunhuang instruction fine-tuning data and the general instruction fine-tuning data to obtain the large language model after instruction fine-tuning; among them, there are nine downstream tasks, namely text repair, splicing, age attribution, named entity recognition, topic modeling, judgment, question answering, analysis, and reasoning; S3. Plug the pre-constructed vector knowledge base into the large language model after instruction fine-tuning to obtain the final trained Dunhuang vertical domain large model.

2. The Dunhuang vertical domain large model training method according to claim 1, characterized in that: In step S1, a data set containing four types of documents, namely, classics, history, philosophy, and collections, is obtained, wherein the data set covers public data, original museum collections, research papers, academic publications, and professional documents. After converting the documents in the data set into a text format, the documents in the text format are cleaned and deduplicated, and then the documents after data cleaning and deduplication are merged into a text file to form an unlabeled text in the Dunhuang field; wherein, data cleaning and deduplication include removing redundant spaces, line breaks, headers, footers, illustrations, tables, formulas, annotation symbols, references, and links in the documents in the text format.

3. The Dunhuang vertical domain large model training method according to claim 1, characterized in that: In step S2, Dunhuang instruction fine-tuning data is generated by three methods: Self-QA, Self-Instruct, and manual construction.

4. The Dunhuang vertical domain large model training method according to claim 1, characterized in that: In step S2, the instruction fine-tuning adopts LoRA fine-tuning.

5. The Dunhuang vertical domain large model training method according to claim 1, characterized in that: In step S3, the specific process of constructing the vector knowledge base is as follows: S31, sorting out the dictionaries and materials related to Dunhuang studies to form a Dunhuang knowledge base, and dividing the texts in the Dunhuang knowledge base into blocks to obtain text blocks containing Dunhuang field information; S32, using a word vector model to vectorize the text block to obtain a vector knowledge base; S33. Through the LangChain framework, the vector knowledge base is connected to the large language model after instruction fine-tuning to obtain the Dunhuang vertical field large model.

6. A question-answering system for a large model of Dunhuang vertical domain, characterized in that: include: A model acquisition module, used to acquire a Dunhuang vertical domain large model trained according to a Dunhuang vertical domain large model training method according to any one of claims 1 to 5; The first processing module is used to input the user's question into the Dunhuang vertical domain big model, perform named entity recognition, extraction and classification on the user's question, and obtain various named entities appearing in the user's question; The second processing module is used to input user questions and various named entities appearing in user questions into the Dunhuang vertical domain big model, infer the implicit relationship between the named entities, and obtain the relationship network; The third processing module is used to input the user questions and various named entities and relationship networks appearing in the user questions into the Dunhuang vertical domain big model, translate the classical Chinese in the user questions, and obtain the modern Chinese translation of the classical Chinese in the user questions; The result acquisition module is used to input user questions, various named entities appearing in user questions, relationship networks and modern Chinese translations of classical Chinese texts in user questions into the Dunhuang vertical domain big model to obtain answers to user questions.

7. The question-answering system of the Dunhuang vertical domain large model as claimed in claim 6, characterized in that: Visualize the question-answering system to obtain a user interface of the Dunhuang vertical domain model with the PhiloCoP framework added; The user interface includes a clear history function and a retry function. The retry function is to regenerate answers to questions input by the user. If the user clicks the clear history function and does not continue to ask questions, the Dunhuang vertical field large model does not retain the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and does not continue to ask questions, the Dunhuang vertical field large model retains the user's historical questions and the answers to the historical questions; if the user clicks the clear history function and continues to ask questions, the Dunhuang vertical field large model does not retain the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions do not refer to the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and continues to ask questions, it is a multi-round dialogue mode, the Dunhuang vertical field large model retains the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions will refer to the user's historical questions and the answers to the historical questions.

8. The question-answering system of the Dunhuang vertical domain large model as claimed in claim 7, characterized in that: The specific process of the visualization processing is as follows: the icon, name and corresponding instructions for use of the Dunhuang vertical field model are added to the original interactive interface in advance. When the user clicks the send function of the original interactive interface, the input box of the original interactive interface obtains the question entered by the user, and the question entered by the user is visualized in the dialog box of the original interactive interface and the answer is output.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, a Dunhuang vertical field large model training method as described in any one of claims 1 to 5 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement a Dunhuang vertical field large model training method as described in any one of claims 1 to 5 when executing the computer program.

Images