A dunhuang vertical field large model training method, system, medium and equipment

Abstract

The application discloses a Dunhuang vertical field large model training method and system, medium and equipment, and belongs to the fields of artificial intelligence and culture. The application forms Dunhuang instruction fine-tuning data and a dictionary type vector knowledge base of 'key word-explanation' by arranging Dunhuang vertical field knowledge such as Dunhuang dictionary, Dunhuang outline and 18 Dunhuang lectures, and trains a Dunhuang vertical field large model for cultural tourism and language experts through steps of continuous pre-training, supervised fine-tuning and external vector knowledge base. Furthermore, a PhiloCoP framework is proposed to facilitate the application of the Dunhuang vertical field large model. The application provides a more convenient tool for Dunhuang lovers and archaeologists to query knowledge and assist in research, helps to more conveniently promote Dunhuang culture in a larger range, and can be applied in the field of ancient linguistics research and a tourism scene.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence and culture, and in particular relates to a large-scale model training method, system, medium and equipment for the Dunhuang vertical field. Background Art

[0002] Recent developments in big language models have spawned numerous vertically-specific big models tailored for diverse downstream applications. Chinese patent CN117253576A describes the use of big language models in medicine for assisting diagnosis, answering health questions, and providing psychological counseling. Chinese patent CN117131173A describes the use of big language models in the legal field for legal question-answering, generating legal documents, and predicting legal decisions. Chinese patent CN117112760A also describes the application of big language models in other fields, such as finance and education.

[0003] There are already some projects that have successfully used language models pre-trained on ancient language corpora to perform specific tasks. In the existing technology, Assael et al. studied the text restoration, chronological attribution, and geographical attribution problems of ancient Greek inscriptions. Yoo et al. trained a BERT model on ancient Korean corpora to complete chronological attribution, topic classification, named entity recognition, and summary retrieval tasks. However, these models were developed before the development of large language models. There are currently no large language models based on vertical fields of ancient Chinese research. The limitations of training large models in the Dunhuang vertical field include: the lack of high-quality training datasets, the complexity of ancient Chinese, and the fragmentation of ancient language research. Summary of the Invention

[0004] The present invention aims to address the existing challenges and provide a method for training a large model for the Dunhuang domain. This method compiles knowledge related to Dunhuang to create a dataset for fine-tuning instructions for the Dunhuang large model and a dictionary-like knowledge base of "keyword-explanation" relationships. Through pre-training, supervised fine-tuning, and the use of an external vector knowledge base, a large Dunhuang model tailored for cultural tourism and language experts is developed.

[0005] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a method for training a large model in a Dunhuang vertical domain, comprising the following steps:

[0007] S1. Pre-train the base large language model on unlabeled texts in the general and Dunhuang domains to obtain a large language model after further pre-training.

[0008] S2. Construct Dunhuang instruction fine-tuning data for the corresponding domain based on the preset downstream tasks, obtain general instruction fine-tuning data, and fine-tune the pre-trained large language model on the Dunhuang instruction fine-tuning data and general instruction fine-tuning data to obtain the fine-tuned large language model. There are nine downstream tasks: text restoration, splicing, chronological attribution, named entity recognition, topic modeling, judgment, question answering, analysis, and reasoning.

[0009] S3. Plug the pre-constructed vector knowledge base into the large language model after instruction fine-tuning to obtain the final trained Dunhuang vertical domain large model.

[0010] Based on the above solution, each step can be implemented in the following preferred specific manner.

[0011] As a preferred embodiment of the first aspect above, in step S1, a data set containing four categories of documents, namely classics, history, philosophy, and collections, is obtained, and the data set covers public data, original museum collections, research papers, academic publications, and professional documents. After converting the documents in the data set into text format, the documents in the text format are cleaned and deduplicated, and then the documents after data cleaning and deduplication are merged into a text file to form unlabeled text in the Dunhuang field; wherein, data cleaning and deduplication include removing redundant spaces, line breaks, headers, footers, illustrations, tables, formulas, annotation symbols, references, and links in the documents in the text format.

[0012] As a preferred embodiment of the first aspect, in step S2, Dunhuang instruction fine-tuning data is generated by three methods: Self-QA, Self-Instruct, and manual construction.

[0013] As a preferred embodiment of the first aspect above, in step S2, the instruction fine-tuning adopts LoRA fine-tuning.

[0014] As a preferred embodiment of the first aspect, in step S3, the specific process of constructing the vector knowledge base is as follows:

[0015] S31, organizing dictionaries and materials related to Dunhuang studies to form a Dunhuang knowledge base, and dividing the text in the Dunhuang knowledge base into blocks to obtain text blocks containing Dunhuang field information;

[0016] S32. Use a word vector model to vectorize the text block to obtain a vector knowledge base;

[0017] S33. Through the LangChain framework, the vector knowledge base is connected to the large language model after instruction fine-tuning to obtain the Dunhuang vertical field large model.

[0018] In a second aspect, the present invention provides a question-answering system for a large model of a vertical domain in Dunhuang, comprising:

[0019] A model acquisition module, configured to acquire a Dunhuang vertical domain large model trained according to a Dunhuang vertical domain large model training method described in any one of the solutions of the first aspect above;

[0020] The first processing module is used to input user questions into the Dunhuang vertical domain large model, perform named entity recognition, extraction and classification on user questions, and obtain various named entities appearing in user questions;

[0021] The second processing module is used to input user questions and various named entities appearing in user questions into the Dunhuang vertical domain large model, infer the implicit relationships between named entities, and obtain a relationship network;

[0022] The third processing module is used to input the user questions and various named entities and relationship networks appearing in the user questions into the Dunhuang vertical domain large model, translate the classical Chinese text in the user questions, and obtain the modern Chinese translation of the classical Chinese text in the user questions;

[0023] The result acquisition module is used to input user questions, various named entities appearing in user questions, relationship networks and modern Chinese translations of classical texts in user questions into the Dunhuang vertical field big model to obtain answers to user questions.

[0024] As a preferred embodiment of the second aspect, the question-answering system is visualized to obtain a user interface of the Dunhuang vertical domain model with the PhiloCoP framework added;

[0025] The user interface includes a clear history function and a retry function. The retry function is to regenerate the answers to the questions input by the user. If the user clicks the clear history function and does not continue to ask questions, the Dunhuang vertical field large model will not retain the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and does not continue to ask questions, the Dunhuang vertical field large model will retain the user's historical questions and the answers to the historical questions; if the user clicks the clear history function and continues to ask questions, the Dunhuang vertical field large model will not retain the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions will not refer to the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and continues to ask questions, it is a multi-round dialogue mode, the Dunhuang vertical field large model retains the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions will refer to the user's historical questions and the answers to the historical questions.

[0026] Furthermore, the specific process of the visualization processing is as follows: the icon, name and corresponding instructions for use of the Dunhuang vertical field model are added to the original interactive interface in advance. When the user clicks the send function of the original interactive interface, the input box of the original interactive interface obtains the question entered by the user, and visualizes the question entered by the user in the dialog box of the original interactive interface and outputs the answer.

[0027] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a large-scale model training method for a vertical field of Dunhuang as described in any of the schemes in the first aspect above is implemented.

[0028] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0029] The memory is used to store computer programs;

[0030] The processor is used to implement a Dunhuang vertical field large model training method as described in any of the solutions of the first aspect above when executing the computer program.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] This invention proposes a method, system, medium, and equipment for training large-scale models of Dunhuang vertical domains. This training method can help Dunhuang enthusiasts and tourists answer questions related to Dunhuang culture and be applied to the tourism industry. It can also assist Dunhuang archaeologists in their research, facilitating faster and more convenient data retrieval, significantly saving time and effort and improving research efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the steps of the present invention;

[0034] Figure 2 It is a flow chart of the overall solution of the present invention;

[0035] Figure 3 is the Self-QA flow chart of the present invention;

[0036] Figure 4 is a Self-Instruct flow chart of the present invention;

[0037] Figure 5 This is a visualization interface diagram of the Dunhuang vertical domain large model of the present invention;

[0038] Figure 6The results of the Dunhuang vertical domain large model of the present invention on different tasks are shown in Figure 2. (a) shows the result on the named entity recognition task; (b) shows the result on the analysis task; (c) shows the result on the reasoning task.

[0039] Figure 7 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0041] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0042] In a preferred embodiment of the present invention, a method for training a large model in the vertical field of Dunhuang is provided, which is a combination of artificial intelligence and cultural fields. In view of the current situation that there is no large model in the vertical field of Dunhuang and the general large language base model has many deficiencies in answering most of the questions in the vertical field of Dunhuang due to the lack of sufficient training data, such as the inability to understand relevant proper nouns, insufficient professionalism in generated content, and serious hallucinations in the answers, the present invention uses Qwen1.5-7B-Chat as the base large language model through pre-training, supervised fine-tuning, external vector knowledge base and other steps, provides a large amount of Dunhuang field data, and trains a large model of the vertical field of Dunhuang that can be used by both Dunhuang amateurs or tourists and Dunhuang researchers.

[0043] The overall solution process is as follows Figure 2As shown. First, in order to enable the Dunhuang vertical field large model to master more Dunhuang field knowledge, the present invention collected and sorted four types of data and used Qwen1.5-7B-Chat as the base large language model, and continued pre-training on it. Then, instruction fine-tuning data was constructed for 9 downstream tasks, and instruction fine-tuning was performed to enable the Dunhuang vertical field large model to have the ability to answer relevant questions. The Dunhuang vertical field large model can access relevant information through the external vector knowledge base. Among them, the implementation of the base large language model Qwen1.5-7B-Chat belongs to the existing technology. For details, please refer to the existing technical literature: Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., T an, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., & Zhu, T. (2023).QwenTechnical Report.*arXiv preprint arXiv:2309.16609*.

[0044] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned Dunhuang vertical domain large model training method includes the following steps S1 to S3. The specific implementation process is described below.

[0045] 1. Continue pre-training

[0046] S1. Continue pre-training the base large language model on unlabeled text in the general and Dunhuang fields to obtain a large language model after continued pre-training.

[0047] It should be noted that in step S1 of the present invention, continuing pre-training is the first step in training the Dunhuang vertical field large model, allowing the base large language model to continue learning on unlabeled texts in general and Dunhuang fields, so that the base large language model can better understand and adapt to the language structure and characteristics of specific fields. This process provides the Dunhuang vertical field large model with specific aspects of corpus that can be learned.

[0048] Specifically, in step S1 of the present invention, a data set containing four categories of documents, namely classics, history, philosophy, and collections, is obtained, and the data set covers public data, original museum collections, research papers, academic publications, and professional documents. After converting the documents in the data set into text format, the documents in the text format are cleaned and deduplicated, and then the documents after data cleaning and deduplication are merged into a text file to form unlabeled text in the Dunhuang field; wherein, data cleaning and deduplication include removing redundant spaces, line breaks, headers, footers, illustrations, tables, formulas, annotation symbols, references, and links in the documents in the text format.

[0049] In this embodiment, it is first necessary to organize the Dunhuang domain corpus for training the large model of the Dunhuang vertical field, put it into a text file to form the PhiloCorpus-ZH corpus, and use it as the unlabeled text in the Dunhuang field. The general unlabeled text and the PhiloCorpus-ZH corpus are then organized into a text file with a total of 1.23 million data, of which 1 million data are from general unlabeled text and 230,000 data are from the PhiloCorpus-ZH corpus. Then, the configuration file for continuing pre-training of the large model in the existing technology Yaowei Zheng, Richardong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma.2024.Llamafactory:Uniffed efffcient ffne-tuning of 100+language models.arXiv preprint arXiv:2403.13372 is used to continue pre-training the base large language model to obtain a large language model after continued pre-training.

[0050] Furthermore, in this embodiment, the PhiloCorpus-ZH corpus includes four categories of documents: classics, history, philosophy, and collections, provided by the School of Literature of Zhejiang University, covering public data, original museum collections, research papers, academic publications, and professional documents. These files are all in PDF format, and the PaddleOCR tool in the prior art is used to convert these files into text format. Extensive data cleaning and deduplication work was then carried out to ensure quality, including removing extra spaces, line breaks, headers, footers, illustrations, tables, formulas, annotation symbols, references, and links. Finally, all files were merged into a text file to form the PhiloCorpus-ZH corpus.

[0051] Furthermore, in this embodiment, in order to prevent the general ability of the base large language model from being significantly reduced, it is also necessary to organize general unlabeled texts. Here, the Shusheng Wanjuan corpus, Wikipedia Chinese dataset and SkyPile-150B in the existing technology are specifically used to obtain general unlabeled texts.

[0052] 2. Instruction fine-tuning

[0053] S2. Construct Dunhuang instruction fine-tuning data for the corresponding field based on the preset downstream tasks, obtain general instruction fine-tuning data, and fine-tune the large language model after continued pre-training on the Dunhuang instruction fine-tuning data and the general instruction fine-tuning data to obtain the large language model after instruction fine-tuning; among them, there are nine downstream tasks, namely text repair, splicing, age attribution, named entity recognition, topic modeling, judgment, question answering, analysis, and reasoning.

[0054] It should be noted that in step S2 of the present invention, instruction fine-tuning is intended to enable the pre-trained large language model to answer relevant questions. The data used for instruction fine-tuning is divided into two parts: one is Dunhuang instruction fine-tuning data constructed based on the preset downstream task, and the other is general instruction fine-tuning data. Therefore, it is necessary to first construct the Dunhuang instruction fine-tuning data based on the downstream task.

[0055] In this embodiment, three methods, Self-QA, Self-Instruct and manual construction, are used to fine-tune the data. Specifically, Self-QA is a self-supervised learning method that does not require labeled data and can generate questions and corresponding answers based on the provided text. The implementation process of Self-QA is as follows: Figure 3 As shown in the figure, we can call GPT4's API to generate instructions based on the Dunhuang Dictionary, then input the instructions and the corresponding documents and call GPT4 to generate the answer to the question. The process of self-instruction to generate instructions and fine-tune data is as follows: Figure 4 As shown, the instruction generation code provided by the BELLE tool in the prior art is used. First, seed data needs to be prepared. The quality of the generated data is closely related to the quality of the provided seed data, so first, a high-quality seed data set with comprehensive content must be compiled. There are three sources of high-quality seed data sets, namely, manually written data, high-quality data in Self-QA, and high-quality data screened out after preliminary Self-Instruct. Finally, the Self-Instruct operation is performed using the high-quality seed data set to generate a large amount of instruction data. It should also be noted that the implementation methods of Self-QA and Self-Instruct are all prior art and will not be described in detail in the present invention.

[0056] It should be noted that in step S2 of the present invention, the large language model after continued pre-training is fine-tuned on the Dunhuang instruction fine-tuning data and the general instruction fine-tuning data, and LoRA fine-tuning is specifically adopted. Its implementation method also belongs to the existing technology and will not be repeated here.

[0057] In this example, the two sets of data used for instruction fine-tuning are in a 1:1 ratio, totaling 48,278 data items. Common instruction fine-tuning data sources include Stanford Alpaca (zh), BELLE 2M (zh), and Fireffy 1.1M 791 (zh). Fine-tuning also uses the instruction fine-tuning file from LLaMA-Factory. Specific parameter settings are shown in Table 1. Among them, the implementation of StanfordAlpaca(zh) is derived from the existing technology Cui, Y., & Yao, X. (2024). Rethinking LLM language adaptation: A case study on Chinese Mixtral.arXiv preprint arXiv:2403.01851.https: / / arxiv.org / abs / 2403.01851; the implementation of BELLE 2M(zh) is derived from the existing technology BelleGroup. (2023). train_2M_CN dataset [Data set]. Hugging Face. https: / / huggingface.co / datasets / BelleGroup / train_2M_CN; the implementation of Fireffy1.1M 791(zh) is derived from the existing technology YeungNLP. (2023). firefly-train-1.1M dataset [Data set]. HuggingFace. https: / / huggingface.co / datasets / YeungNLP / firefly-train-1.1M.

[0058] Table 1. Instruction fine-tuning parameter settings

[0059] parameter set up Fine-tuning Type LoRA Cutoff Length 2048 Learning Rate 1e-4 Training Epoch 8 Batch Size 32 Optimizer adamw_torch LoRAγ 8 LoRA α 16 LoRA Dropout 0.1 LoRA Target All

[0060] 3. External Vector Knowledge Base

[0061] S3. Plug the pre-constructed vector knowledge base into the large language model after instruction fine-tuning to obtain the final trained Dunhuang vertical domain large model.

[0062] In the present invention, in order to improve the accuracy of generated content and reduce hallucinations, additional knowledge is provided to the Dunhuang vertical domain large model through an external vector knowledge base. The principle of the external vector knowledge base is to use an additional database as input. Knowledge can be retrieved from this knowledge base according to the user's question, and prompt words are formed based on the search content and the question. The prompt words are input into the large language model after the instruction is fine-tuned to generate the answer. However, since the knowledge base is too large and exceeds the limit of the number of words that the large language model can read at one time, the data can be divided into several pieces, and only the items related to the question are read out each time as the input of the large language model after the instruction is fine-tuned. Then it is only necessary to find the fragments in the database that are most relevant to the question. Vector retrieval is used here to vectorize all documents and keywords and store them in the vector database. Finally, the vector knowledge base is connected to the large language model after the instruction is fine-tuned through the LangChain framework. After adding the external vector knowledge base, the effect of the large language model after the instruction is fine-tuned has been significantly improved, reducing the situation of irrelevant answers and hallucinations.

[0063] It should be noted that, in step S3 of the present invention, the specific process of constructing the vector knowledge base is as follows:

[0064] S31, organizing dictionaries and materials related to Dunhuang studies to form a Dunhuang knowledge base, and dividing the text in the Dunhuang knowledge base into blocks to obtain text blocks containing Dunhuang field information;

[0065] S32. Use a word vector model to vectorize the text block to obtain a vector knowledge base;

[0066] S33. Through the LangChain framework, the vector knowledge base is connected to the large language model after instruction fine-tuning to obtain the Dunhuang vertical field large model.

[0067] It should also be noted that, as an ideographic language, one Chinese character in ancient Chinese can represent multiple meanings depending on the context and grammar. There are also some grammars in ancient Chinese that are not found in modern Chinese, such as inversion and part-of-speech conversion. In addition, religious and cultural factors lead to significant differences in naming habits between ancient and modern Chinese. Therefore, linguists usually conduct a comprehensive analysis of the above three points before conducting subsequent research. In this embodiment, in order to imitate the analysis process of linguists when studying ancient Chinese literature, a question-answering system based on a large model of Dunhuang vertical domains including the PhiloCoP framework is designed. The large model of Dunhuang vertical domains is used to solve the problem step by step, helping the large model of Dunhuang vertical domains to complete the task.

[0068] like Figure 7 As shown, a question-answering system for a large model of the Dunhuang vertical domain specifically includes:

[0069] A model acquisition module, configured to acquire a Dunhuang vertical domain large model trained by a Dunhuang vertical domain large model training method according to the above embodiment;

[0070] The first processing module is used to input user questions into the Dunhuang vertical domain large model, perform named entity recognition, extraction and classification on user questions, and obtain various named entities appearing in user questions;

[0071] The second processing module is used to input user questions and various named entities appearing in user questions into the Dunhuang vertical domain large model, infer the implicit relationships between named entities, and obtain a relationship network;

[0072] The third processing module is used to input the user questions and various named entities and relationship networks appearing in the user questions into the Dunhuang vertical domain large model, translate the classical Chinese text in the user questions, and obtain the modern Chinese translation of the classical Chinese text in the user questions;

[0073] The result acquisition module is used to input user questions, various named entities appearing in user questions, relationship networks and modern Chinese translations of classical texts in user questions into the Dunhuang vertical field big model to obtain answers to user questions.

[0074] It should be noted that, in the present invention, the above-mentioned question-answering system can be visualized to obtain a user interface of the Dunhuang vertical domain large model with the PhiloCoP framework added.

[0075] The above-mentioned user interface includes a clear history function and a retry function. The retry function is to regenerate the answers to the questions entered by the user. If the user clicks the clear history function and does not continue to ask questions, the Dunhuang vertical field large model will not retain the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and does not continue to ask questions, the Dunhuang vertical field large model will retain the user's historical questions and the answers to the historical questions; if the user clicks the clear history function and continues to ask questions, the Dunhuang vertical field large model will not retain the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions will not refer to the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and continues to ask questions, it will be a multi-round dialogue mode. The Dunhuang vertical field large model will retain the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions will refer to the user's historical questions and the answers to the historical questions.

[0076] The specific process of the above-mentioned visualization processing is as follows: the icon, name and corresponding instructions for use of the Dunhuang vertical field model are added to the original interactive interface in advance. When the user clicks the send function of the original interactive interface, the input box of the original interactive interface obtains the question entered by the user, and visualizes the question entered by the user in the dialog box of the original interactive interface and outputs the answer.

[0077] In the visualization processing of this embodiment, the original interactive interface of Qwen is used as the original interactive interface. After modification on its basis, the icon, name and corresponding instruction text of the Dunhuang vertical domain large model are added, and the visual user interface of the Dunhuang vertical domain large model with the PhiloCoP framework can be obtained, as shown in FIG. Figure 5 As shown. After the user enters the question he wants to know the answer to in the input box and clicks send, the question will appear in the dialog box above, and then the answer will be output. In addition, the user interface designed by the present invention also has a history clearing function and a retry function. If you continue to ask questions without clearing the history, it will be a multi-round dialogue mode. The Dunhuang vertical field model will retain the information of previous user questions. After clearing the history, the memory of the previous question information will be deleted, so the same question may have different answers before and after clearing the history.

[0078] In order to better demonstrate the specific implementation and technical effects of the present invention, a Dunhuang vertical domain large model training method shown in steps S1 to S3 in the above preferred implementation method is applied to a specific example.

[0079] Example

[0080] The Dunhuang vertical domain large model shows significant performance improvements on various linguistic tasks. In Table 2, the Dunhuang vertical domain large model is benchmarked against other competitive large models using a benchmark designed based on downstream tasks. "N / A" indicates that the large model cannot solve the specific linguistic task, that is, it refuses to answer or randomly generates answers. For text restoration and age attribution tasks, the smaller the score, the better the performance, while in other tasks, the larger the score, the better the performance. It can be seen that in text restoration and age attribution tasks, other open source large models are basically unable to complete the tasks, while the Dunhuang vertical domain large model of the present invention shows significant enhancements, especially in tasks centered on generation and retrieval. The PhilCoP architecture can enable most large language models to achieve performance improvements on their respective tasks. The present invention uses PhilCoP to reconstruct the original document to eliminate the lexical confusion inherent in ancient Chinese grammar and clearly extract key information through named entity recognition. This method can improve the performance of models that are not trained on domain datasets, with the exception of Baichuan2, which has insufficient ability to handle classical Chinese named entity recognition and translation.

[0081] Table 2. Test results of the Dunhuang vertical domain large model and other large models

[0082]

[0083] In this example, three base large language models are used respectively, and the GPT-4o judgment win rate is used as the evaluation indicator to evaluate the effect of the Dunhuang vertical domain large model on named entity recognition, analysis and reasoning tasks. The results are as follows: Figure 6 As shown, the higher the winning rate, the stronger the ability.

[0084] In summary, the present invention generates a large amount of high-quality data in the Dunhuang vertical field by selecting general reading materials and professional documents in the Dunhuang vertical field, and combines the general corpus to form unlabeled text for continued pre-training of the base large language model; the present invention constructs Dunhuang instruction fine-tuning data based on professional scientific research and general cultural and tourism scenarios, specifically covering Dunhuang history, stories, culture, economy, science and other fields, for supervised instruction fine-tuning of the large language model after continued pre-training; the present invention constructs a vector knowledge base with "keyword-explanation" dictionary knowledge, documents and keywords, and uses the external vector knowledge base to perform keyword retrieval on the questions input to the large language model, merge the knowledge related to the questions into prompt words as new input to the large language model, and enhance the accuracy and professionalism of the content generated by the large language model; in addition, the present invention proposes the PhiloCoP framework, which follows the thinking chain of bibliographers and gradually helps the large language model to parse the vocabulary and grammar of ancient Chinese. This method has shown significant improvements in multiple downstream tasks; finally, the present invention provides a user interface to facilitate the use of the Dunhuang vertical field large model, display the input and output of the question, and can perform sending, clearing history and retry operations.

[0085] It should also be noted that the Dunhuang vertical domain large model training method in the above embodiment can essentially be executed through a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the Dunhuang vertical domain large model training method provided in the above embodiment, which includes a memory and a processor;

[0086] The memory is used to store computer programs;

[0087] The processor is used to implement a Dunhuang vertical field large model training method in the above embodiment when executing the computer program.

[0088] Furthermore, the logic instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention.

[0089] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the Dunhuang vertical field large model training method provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, it can implement the Dunhuang vertical field large model training method in the above embodiment.

[0090] It is understood that the storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage medium may be any medium capable of storing program code, such as a USB flash drive, a mobile hard drive, a magnetic disk, or an optical disk.

[0091] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0092] It should also be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0093] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A question-answering system for a large model of Dunhuang vertical domain, characterized by: include: A model acquisition module is used to obtain a Dunhuang vertical domain large model trained according to a Dunhuang vertical domain large model training method; The first processing module is used to input user questions into the Dunhuang vertical domain large model, perform named entity recognition, extraction and classification on user questions, and obtain various named entities appearing in user questions; The second processing module is used to input user questions and various named entities appearing in user questions into the Dunhuang vertical domain large model, infer the implicit relationships between named entities, and obtain a relationship network; The third processing module is used to input the user questions and various named entities and relationship networks appearing in the user questions into the Dunhuang vertical domain large model, translate the classical Chinese text in the user questions, and obtain the modern Chinese translation of the classical Chinese text in the user questions; The result acquisition module is used to input user questions, various named entities appearing in user questions, relationship networks, and modern Chinese translations of the classical text in user questions into the Dunhuang vertical domain big model to obtain answers to user questions; A Dunhuang vertical domain large model training method includes the following steps: S1. Pre-train the base large language model on unlabeled texts in the general and Dunhuang domains to obtain a large language model after further pre-training. S2. Construct Dunhuang instruction fine-tuning data for the corresponding domain based on the preset downstream tasks, obtain general instruction fine-tuning data, and fine-tune the pre-trained large language model on the Dunhuang instruction fine-tuning data and general instruction fine-tuning data to obtain the fine-tuned large language model. There are nine downstream tasks: text restoration, splicing, chronological attribution, named entity recognition, topic modeling, judgment, question answering, analysis, and reasoning. S3. Plug the pre-constructed vector knowledge base into the large language model after instruction fine-tuning to obtain the final trained Dunhuang vertical domain large model.

2. The question-answering system for the Dunhuang vertical domain large model according to claim 1, characterized in that: In step S1, a data set containing four categories of documents, namely classics, history, philosophy, and collections, is obtained. The data set covers public data, original museum collections, research papers, academic publications, and professional documents. After converting the documents in the data set into text format, the documents in the text format are cleaned and deduplicated, and then the documents after data cleaning and deduplication are merged into a text file to form unlabeled text in the Dunhuang field; wherein, data cleaning and deduplication include removing redundant spaces, line breaks, headers, footers, illustrations, tables, formulas, annotation symbols, references, and links in the documents in the text format.

3. The question-answering system for the Dunhuang vertical domain large model according to claim 1, characterized in that: In step S2, Dunhuang instruction fine-tuning data is generated by three methods: Self-QA, Self-Instruct, and manual construction.

4. The question-answering system for the Dunhuang vertical domain model according to claim 1, characterized in that: In step S2, the instruction fine-tuning adopts LoRA fine-tuning.

5. The question-answering system for the Dunhuang vertical domain large model according to claim 1 is characterized in that: In step S3, the specific process of constructing the vector knowledge base is as follows: S31, organizing dictionaries and materials related to Dunhuang studies to form a Dunhuang knowledge base, and dividing the text in the Dunhuang knowledge base into blocks to obtain text blocks containing Dunhuang field information; S32. Use a word vector model to vectorize the text block to obtain a vector knowledge base; S33. Through the LangChain framework, the vector knowledge base is connected to the large language model after instruction fine-tuning to obtain the Dunhuang vertical field large model.

6. The question-answering system for the Dunhuang vertical domain large model according to claim 1, characterized in that: Visualize the question-answering system to obtain a user interface of the Dunhuang vertical domain model with the PhiloCoP framework added; The user interface includes a clear history function and a retry function. The retry function is to regenerate the answers to the questions input by the user. If the user clicks the clear history function and does not continue to ask questions, the Dunhuang vertical field large model will not retain the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and does not continue to ask questions, the Dunhuang vertical field large model will retain the user's historical questions and the answers to the historical questions; if the user clicks the clear history function and continues to ask questions, the Dunhuang vertical field large model will not retain the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions will not refer to the user's historical questions and the answers to the historical questions; if the user does not click the clear history function and continues to ask questions, it is a multi-round dialogue mode, the Dunhuang vertical field large model retains the user's historical questions and the answers to the historical questions, and the answers output after continuing to ask questions will refer to the user's historical questions and the answers to the historical questions.

7. The question-answering system for a large-scale vertical model of Dunhuang as claimed in claim 6, characterized in that: The specific process of the visualization processing is as follows: the icon, name and corresponding instructions for use of the Dunhuang vertical field model are added to the original interactive interface in advance. When the user clicks the send function of the original interactive interface, the input box of the original interactive interface obtains the question entered by the user, visualizes the question entered by the user in the dialog box of the original interactive interface and outputs the answer.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, it implements a question-answering system for a large model of a vertical field of Dunhuang as described in any one of claims 1 to 7.

9. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement a question-answering system for a large vertical model of Dunhuang as described in any one of claims 1 to 7 when executing the computer program.

Images