Document question and answer model training method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an enterprise-oriented document question and answer model training method and system, and the method comprises the steps: obtaining general supervision fine tuning data, general document question and answer data and enterprise internal data; mixing the general supervision fine tuning data, the general document question and answer data and the enterprise internal data according to a predetermined proportion to obtain training data; and on the basis of the training data, performing supervised fine tuning training on a basic large language model according to a unified document question and answer prompt word template to obtain a document question and answer model. According to the embodiment of the invention, the accuracy, controllability and rejection capability of the document question and answer model are remarkably improved, the defects of an existing RAG system in an enterprise application scene are overcome, and the method is more suitable for enterprise-level knowledge question and answer tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of neural networks, and particularly to a method for training a document question-answering model for enterprises, a system for training a document question-answering model for enterprises, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the Retrieval-Augmented Generation (RAG) task, the large model needs to integrate and refine answers only based on the retrieved document fragments, and should reject answering questions that are not in the document, rather than answering without basis using the knowledge memorized by the parameters.

[0003] If the existing RAG engines directly call the general basic large model, it is difficult to avoid that many answers come from the parameter memory of the model itself. For questions that require in-depth analysis based on the retrieved content or questions that need to reject recognition, the reply effect will be rather chaotic. If only prompt engineering is used to improve the credibility of the model, the effect is limited. The model cannot well distinguish between answerable / rejectable situations. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a method for training a document question-answering model for enterprises, a system for training a document question-answering model for enterprises, an electronic device, and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0005] To solve the above problems, embodiments of the present invention disclose a method for training a document question-answering model for enterprises, the method including:

[0006] Obtaining general supervised fine-tuning data, general document question-answering data, and enterprise internal data;

[0007] Mixing the general supervised fine-tuning data, the general document question-answering data, and the enterprise internal data according to a predetermined ratio to obtain training data;

[0008] Based on the training data, performing supervised fine-tuning training on the basic large language model according to a unified document question-answering prompt template to obtain a document question-answering model.

[0009] Optionally, the obtaining of the general supervised fine-tuning data includes:

[0010] Collecting conversation data, NLP task data, multi-turn conversation data, and other data from the CuteGPT dataset;

[0011] Collecting Chinese data from the Infinity Instruct dataset;

[0012] Collect English data from the Infinity Instruct dataset;

[0013] Use the dialogue data, the NLP task data, the multi-turn dialogue data, the other data, the Chinese data, and the English data as the general supervised fine-tuning data.

[0014] Optionally, the obtaining of general document Q&A data includes:

[0015] Collect Chinese single-turn RAG document Q&A data from the PAMQA dataset;

[0016] Collect English document Q&A data from the CHATQA dataset;

[0017] Collect Chinese RAG data from the TruthReader_zh dataset;

[0018] Collect English RAG data from the TruthReader_en dataset;

[0019] Convert the Chinese single-turn RAG document Q&A data, the English document Q&A data, the Chinese RAG data, and the English RAG data into the document Q&A prompt format and use them as the general document Q&A data.

[0020] Optionally, the obtaining of enterprise internal data includes:

[0021] Parse the document-type enterprise internal documents to obtain single-turn Q&A data, multi-turn Q&A data, and data recognition data;

[0022] Construct the Q&A pair enterprise internal documents into the standard format data of a RAG system;

[0023] Use the single-turn Q&A data, the multi-turn Q&A data, the data recognition data, and the standard format data as the enterprise internal data.

[0024] Optionally, the parsing of the document-type enterprise internal documents to obtain single-turn Q&A data includes:

[0025] Use a large language model to generate questions based on the document-type enterprise internal documents;

[0026] Use a large language model to generate answers based on the questions and the corresponding document-type enterprise internal documents;

[0027] Annotate the document-type enterprise internal documents, the questions, and the answers to obtain the single-turn Q&A data.

[0028] Optionally, parsing the internal documents of the document-based enterprise to obtain multi-round Q&A data, including:

[0029] Taking the internal documents of the document-based enterprise and the single-round Q&A data as the first-round input items;

[0030] Using a large language model to predict subsequent multi-round questions based on the first-round input items;

[0031] Expanding the multi-round questions into the multi-round Q&A data.

[0032] Optionally, parsing the internal documents of the document-based enterprise to obtain knowledge data, including:

[0033] Using a large language model to extract questions from the internal documents of the document-based enterprise;

[0034] Excluding the internal enterprise documents to which the questions belong during the retrieval of retrieval fragments;

[0035] Combining the retrieved other text fragments with the corresponding questions;

[0036] Taking the combination as the knowledge data.

[0037] Optionally, constructing the internal documents of the Q&A pair enterprise into standard format data of a RAG system, including:

[0038] Extracting similar questions from the internal documents of the Q&A pair enterprise as query data;

[0039] Retrieving corresponding standard Q&A pairs based on the query data;

[0040] Retrieving similar Q&A pairs and random Q&A pairs as alternative Q&A pairs;

[0041] Constructing the standard Q&A pairs and the alternative Q&A pairs into the standard format data, and taking the standard Q&A pairs as the true labels of the query data.

[0042] An embodiment of the present invention also discloses a document Q&A model training system for enterprises, and the system includes:

[0043] A data acquisition module for acquiring general supervised fine-tuning data, general document Q&A data, and internal enterprise data;

[0044] A data mixing module for mixing the general supervised fine-tuning data, the general document Q&A data, and the internal enterprise data in a predetermined ratio to obtain training data;

[0045] A model fine-tuning module for performing supervised fine-tuning training on a basic large language model based on the training data according to a unified document Q&A prompt template to obtain a document Q&A model.

[0046] Optionally, the data acquisition module includes:

[0047] A first data collection module for collecting dialogue data, NLP task data, multi-turn dialogue data, and other data from the CuteGPT dataset;

[0048] A second data collection module for collecting Chinese data from the Infinity Instruct dataset;

[0049] A third data collection module for collecting English data from the Infinity Instruct dataset;

[0050] A general supervised fine-tuning data determination module for using the dialogue data, the NLP task data, the multi-turn dialogue data, the other data, the Chinese data, and the English data as the general supervised fine-tuning data.

[0051] Optionally, the data acquisition module includes:

[0052] A fourth data collection module for collecting Chinese single-turn RAG document Q&A data from the PAMQA dataset;

[0053] A fifth data collection module for collecting English document Q&A data from the CHATQA dataset;

[0054] A sixth data collection module for collecting Chinese RAG data from the TruthReader_zh dataset;

[0055] A seventh data collection module for collecting English RAG data from the TruthReader_en dataset;

[0056] A general document Q&A data determination module for converting the Chinese single-turn RAG document Q&A data, the English document Q&A data, the Chinese RAG data, and the English RAG data into the document Q&A prompt format and using it as the general document Q&A data.

[0057] Optionally, the data acquisition module includes:

[0058] A document parsing module for parsing enterprise internal documents of the document type to obtain single-turn Q&A data, multi-turn Q&A data, and data recognition data;

[0059] A standard data construction module for constructing the enterprise internal documents of the Q&A pairs into the standard format data of a RAG system;

[0060] An enterprise internal data determination module, configured to use the single-round Q&A data, the multi-round Q&A data, the knowledge identification data, and the standard format data as the enterprise internal data.

[0061] Optionally, the document parsing module includes:

[0062] A question generation module, configured to use a large language model to generate questions based on the enterprise internal documents of the document type;

[0063] An answer generation module, configured to use a large language model to generate answers based on the questions and the enterprise internal documents of the document type corresponding to the questions;

[0064] A single-round Q&A data determination module, configured to label the enterprise internal documents of the document type, the questions, and the answers to obtain the single-round Q&A data.

[0065] Optionally, the document parsing module includes:

[0066] An input item determination module, configured to use the enterprise internal documents of the document type and the single-round Q&A data as the first-round input items;

[0067] A multi-round question prediction module, configured to use a large language model to predict subsequent multi-round questions based on the first-round input items;

[0068] A multi-round Q&A expansion module, configured to expand the multi-round questions into the multi-round Q&A data.

[0069] Optionally, the document parsing module includes:

[0070] A question extraction module, configured to use a large language model to extract questions from the enterprise internal documents of the document type;

[0071] A document exclusion module, configured to exclude the enterprise internal documents to which the questions belong during the retrieval of retrieval fragments;

[0072] A combination module, configured to combine the retrieved other text fragments with the corresponding questions;

[0073] A knowledge identification data determination module, configured to use the combination as the knowledge identification data.

[0074] Optionally, the standard data construction module includes:

[0075] A similar question extraction module, configured to extract similar questions from the enterprise internal documents of the Q&A pair type as query data;

[0076] A standard Q&A pair retrieval module, configured to retrieve corresponding standard Q&A pairs based on the query data;

[0077] An alternative Q&A pair determination module for retrieving similar Q&A pairs and random Q&A pairs as alternative Q&A pairs;

[0078] A standard format data determination module for constructing the standard Q&A pairs and the alternative Q&A pairs into the standard format data, and using the standard Q&A pairs as the true labels of the query data.

[0079] An embodiment of the present invention also discloses an electronic device, including: one or more processors; and one or more machine-readable media storing instructions thereon, which when executed by the one or more processors, cause the electronic device to execute the enterprise-oriented document Q&A model training method as described above.

[0080] An embodiment of the present invention also discloses a computer-readable storage medium, the computer program stored thereon causing a processor to execute the enterprise-oriented document Q&A model training method as described above.

[0081] The embodiments of the present invention include the following advantages:

[0082] The enterprise-oriented document Q&A model training solution provided by the embodiments of the present invention obtains general supervised fine-tuning data, general document Q&A data, and enterprise internal data; mixes the general supervised fine-tuning data, the general document Q&A data, and the enterprise internal data according to a predetermined ratio to obtain training data; and performs supervised fine-tuning training on a basic large language model based on the training data according to a unified document Q&A prompt template to obtain a document Q&A model.

[0083] Compared with the background art, the embodiments of the present invention have the following beneficial effects:

[0084] 1. Reduce the model's dependence on parameter memory and improve the accuracy of answering based on retrieved content

[0085] By obtaining and mixing general supervised fine-tuning data, general document Q&A data, and enterprise internal data, the model can take into account general knowledge and enterprise-specific domain knowledge during training, thereby reducing the dependence on parameter memory knowledge.

[0086] 2. Solve the problem of insufficient rejection ability of existing RAG models and enhance the rejection ability

[0087] By constructing a training set containing rejection data, the model can learn when to reject answering, thereby solving the problem that existing RAG systems cannot effectively distinguish between answerable and unanswerable situations.

[0088] 3. Optimize the document Q&A ability by combining enterprise internal data and improve the adaptability to enterprise scenarios

[0089] By introducing enterprise internal data, ensure that the model can adapt to enterprise document formats, terminology, and industry knowledge, thereby significantly improving the quality of question and answer in enterprise application scenarios.

[0090] 4. Improve the comprehensive performance of the model in the document question and answer task by optimizing the data mixing ratio

[0091] By adjusting the mixing ratio of general supervised fine-tuning data, general document question and answer data, and enterprise internal data, balance the general ability and domain adaptation ability of the model, so that it can not only maintain the basic language understanding ability, but also perform well in the enterprise document question and answer task.

[0092] 5. Unify the prompt template to improve the consistency and controllability of the model's question and answer

[0093] By adopting a unified document question and answer prompt template, ensure that the model follows fixed rules during the reasoning process, improve the consistency, controllability, and credibility of the answers, and avoid the problem of unstable output styles caused by different contexts in existing large models.

[0094] In summary, the embodiments of the present invention significantly improve the accuracy, controllability, and rejection recognition ability of the document question and answer model, solve the deficiencies of existing RAG systems in enterprise application scenarios, and make it more suitable for enterprise-level knowledge question and answer tasks. Brief Description of the Drawings

[0095] Figure 1 is a step flow chart of a method for training an enterprise-oriented document question and answer model according to an embodiment of the present invention;

[0096] Figure 2 is a schematic diagram of the generation process of single-document question and answer data according to an embodiment of the present invention;

[0097] Figure 3 is a schematic diagram of the generation process of multi-round document question and answer data according to an embodiment of the present invention;

[0098] Figure 4 is a schematic diagram of the generation process of FAQ high-frequency question and answer pair data according to an embodiment of the present invention;

[0099] Figure 5 is a structural block diagram of an enterprise-oriented document question and answer model training system according to an embodiment of the present invention. Detailed Embodiments

[0100] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0101] An embodiment of the present invention proposes a training scheme for an enterprise-oriented document question-answering model. First, general supervised fine-tuning data, general document question-answering data, and enterprise internal data are obtained, and these data are screened and standardized to ensure data quality and task adaptability. Subsequently, the above three types of data are mixed in a predetermined proportion, while retaining the general capabilities of the model, enhancing its adaptability to enterprise documents, and improving the model's rejection ability by constructing rejection data. Based on the mixed training data, a unified document question-answering prompt template is used to perform supervised fine-tuning training on the basic large language model, optimizing its performance in the document question-answering task, enabling it to generate answers more accurately based on the retrieved document fragments, and correctly rejecting answers when unable to infer an answer. This scheme not only reduces the model's dependence on parameter memorization, improves the accuracy of question answering, but also enhances the adaptability and rejection ability of the enterprise scenario, while optimizing the training data ratio and fine-tuning strategy to improve the comprehensive performance of the model.

[0102] Refer to Figure 1 , which shows the step flowchart of a training method for an enterprise-oriented document question-answering model according to an embodiment of the present invention. The training method for the enterprise-oriented document question-answering model may specifically include the following steps:

[0103] Step 101, obtain general supervised fine-tuning data, general document question-answering data, and enterprise internal data.

[0104] When constructing an enterprise-oriented document question-answering model, data quality and diversity are crucial for the training effect of the model. Therefore, it is first necessary to collect and organize various types of data, including general supervised fine-tuning data, general document question-answering data, and enterprise internal data. These three types of data play different roles in model training: general supervised fine-tuning data is mainly used to maintain the general capabilities of the model, general document question-answering data helps to improve the model's performance in document question-answering tasks, and enterprise internal data enhances the model's adaptability to enterprise-specific scenarios.

[0105] General supervised fine-tuning data usually consists of multiple open-source datasets, covering categories such as dialogue, natural language processing (NLP) tasks, and multi-turn dialogue. The quality of this type of data directly affects the basic capabilities of the model, so it is necessary to screen high-quality data sources. For example, it is possible to track the supervised fine-tuning (SFT) datasets used by high-quality open-source basic large models, and eliminate low-quality, irrelevant, or non-standard formatted data to ensure the reliability of the training data.

[0106] The general document Q&A data mainly comes from existing open-source RAG datasets, such as PAMQA, CHATQA, TruthReader, etc. This data usually includes user questions, relevant document fragments, and standard answers, which helps to train the model's performance in document retrieval and Q&A tasks. In data processing, it is necessary to unify the format and ensure that the Q&A data structure is clear so that the subsequent model can learn efficiently.

[0107] Enterprise internal data is a key part of improving the model's enterprise adaptability. Structured Q&A data is usually lacking within the enterprise, so it is necessary to extract relevant information from different sources. For example, enterprise technical documents, Frequently Asked Questions (FAQ), product manuals, etc. can be parsed, and large models can be used to automatically generate Q&A data. In addition, enterprise documents come in various formats, such as PDF, DOCX, PPTX, etc., and corresponding parsing tools (such as PyPDF2, Pandas, etc.) need to be used to extract the text content and standardize it. For Q&A pair data, manual screening and cleaning can be carried out to ensure the accuracy and usability of the data.

[0108] Step 102: Mix the general supervised fine-tuning data, the general document Q&A data, and the enterprise internal data in a predetermined proportion to obtain training data.

[0109] After obtaining the general supervised fine-tuning data, the general document Q&A data, and the enterprise internal data, they need to be mixed in a predetermined proportion to construct the final training dataset. A reasonable data mixing proportion is crucial for optimizing the model's performance, which not only affects the accuracy of Q&A but also determines the model's rejection ability and domain adaptability.

[0110] The proportion of the general supervised fine-tuning data and the document Q&A data needs to be carefully adjusted. If the proportion of the general supervised fine-tuning data is too high, the model may be more inclined to use general knowledge to answer questions and ignore specific document information, thus reducing the ability to answer based on retrieved content; conversely, if the proportion of the document Q&A data is too high, it may lead to a decline in the model's ability in other tasks. Therefore, it is necessary to ensure that the proportion of the two can maintain the basic ability and optimize the performance of the document Q&A task. The proportion of enterprise internal data also needs to be weighed. The introduction of enterprise internal data can enhance the model's adaptability in the enterprise scenario, but if the proportion is too high, it may limit the model's generalization ability.

[0111] Step 103: Based on the training data, perform supervised fine-tuning training on the basic large language model according to a unified document Q&A prompt template to obtain a document Q&A model.

[0112] After obtaining the training data, it is necessary to perform supervised fine-tuning training on the basic large language model based on this data to improve its performance in the document question-answering task. Supervised fine-tuning is a technique for optimizing the specific capabilities of a large model through supervised learning, which can make the model more in line with the needs of enterprises.

[0113] Before fine-tuning training, it is necessary to set a unified document question-answering prompt template to ensure that the model can learn a stable answer pattern during training. The role of the prompt template is to guide the model to answer questions only based on the retrieved document fragments, rather than relying on the knowledge memorized by parameters. The training adopts multi-machine and multi-card distributed training, such as 3D parallel training based on Deepspeed, to improve the training efficiency.

[0114] The document question-answering model training solution for enterprises provided by the embodiments of the present invention obtains general supervised fine-tuning data, general document question-answering data, and enterprise internal data; mixes the general supervised fine-tuning data, the general document question-answering data, and the enterprise internal data according to a predetermined ratio to obtain training data; and performs supervised fine-tuning training on the basic large language model based on the training data according to a unified document question-answering prompt template to obtain a document question-answering model.

[0115] Compared with the background technology, the embodiments of the present invention have the following beneficial effects:

[0116] 1. Reduce the model's dependence on parameter memory and improve the accuracy of answering based on retrieved content

[0117] By obtaining and mixing general supervised fine-tuning data, general document question-answering data, and enterprise internal data, the model can take into account general knowledge and enterprise-specific domain knowledge during training, thereby reducing the dependence on parameter memory knowledge.

[0118] 2. Solve the problem of insufficient rejection ability of existing RAG models and enhance the rejection ability

[0119] By constructing a training set containing rejection data, the model can learn when to reject answering, thereby solving the problem that existing RAG systems cannot effectively distinguish between answerable and unanswerable situations.

[0120] 3. Optimize the document question-answering ability by combining enterprise internal data and improve the adaptability to enterprise scenarios

[0121] By introducing enterprise internal data, it is ensured that the model can adapt to enterprise document formats, terms, and industry knowledge, thereby significantly improving the quality of question-answering in enterprise application scenarios.

[0122] 4. Improve the comprehensive performance of the model in the document question-answering task by optimizing the data mixing ratio

[0123] By adjusting the mixing ratio of general supervised fine-tuning data, general document Q&A data, and enterprise internal data, the general ability and domain adaptation ability of the model are balanced, enabling it to not only maintain basic language understanding ability but also perform well in enterprise document Q&A tasks.

[0124] 5. Unify the prompt template to improve the Q&A consistency and controllability of the model

[0125] By adopting a unified document Q&A prompt template, it is ensured that the model follows fixed rules during the reasoning process, improving the consistency, controllability, and credibility of the answers, and avoiding the problem of unstable output styles caused by different contexts in existing large models.

[0126] In summary, the embodiments of the present invention significantly improve the accuracy, controllability, and rejection ability of the document Q&A model, solve the deficiencies of existing RAG systems in enterprise application scenarios, and make it more suitable for enterprise-level knowledge Q&A tasks.

[0127] In an exemplary embodiment of the present invention, one implementation of obtaining general supervised fine-tuning data is as follows: collecting dialogue data, NLP task data, multi-turn dialogue data, and other data from the CuteGPT dataset; collecting Chinese data from the Infinity Instruct dataset; collecting English data from the Infinity Instruct dataset; and using the dialogue data, the NLP task data, the multi-turn dialogue data, the other data, the Chinese data, and the English data as the general supervised fine-tuning data.

[0128] This implementation collects various types of data from multiple high-quality data sources to enhance the general ability of the model and improve its adaptability to different tasks in subsequent training. In the actual application process, four major types of data are collected from the CuteGPT dataset: (1) Dialogue data, usually containing multi-turn conversations in the open domain, which can enhance the model's dialogue understanding ability and make it more adaptable to natural language interaction scenarios; (2) Natural language processing task data, including tasks such as text classification, reading comprehension, and sentiment analysis. This type of data can improve the model's ability in language understanding, reasoning, and information extraction; (3) Multi-turn dialogue data, focusing on continuous interaction dialogue flows, which helps optimize the model's performance in context connection and long dialogue processing; (4) Other data, usually containing content that does not belong to the above categories but is helpful for language model training, such as the conversion of structured data and task instruction data.

[0129] Collect Chinese data and English data from the Infinity Instruct dataset respectively to ensure that the model can handle tasks in different language environments. The collection of Chinese data can enhance the model's understanding ability when processing internal Chinese documents of enterprises, while English data enables the model to have cross-language question-answering capabilities, which is applicable to international enterprises or scenarios involving multilingual documents.

[0130] This implementation method ensures the stability of the model in different task scenarios, enabling it to not only handle open-domain conversations but also perform enterprise document question-answering tasks. Collecting multi-turn conversation data can enhance the model's context understanding ability and optimize its continuous reasoning performance in long conversations, especially having significant advantages in application scenarios such as enterprise knowledge base question-answering and technical support. In addition, combining Chinese and English data enables the model to not only handle Chinese enterprise documents but also be applicable to international enterprises, improving the applicability of cross-language document question-answering.

[0131] In an exemplary embodiment of the present invention, one implementation method for obtaining general document question-answering data is as follows: Collect Chinese single-turn RAG document question-answering data from the PAMQA dataset; collect English document question-answering data from the CHATQA dataset; collect Chinese RAG data from the TruthReader_zh dataset; collect English RAG data from the TruthReader_en dataset; convert the Chinese single-turn RAG document question-answering data, the English document question-answering data, the Chinese RAG data, and the English RAG data into the format of document question-answering prompt words and use them as the general document question-answering data.

[0132] This implementation method collects different languages and types of RAG data from multiple high-quality datasets and uniformly converts them into the format of document question-answering prompt words to enhance the adaptability and generalization ability of the model in document question-answering tasks. In the actual application process, this implementation method involves multiple data sources. The PAMQA dataset provides Chinese single-turn RAG document question-answering data, which mainly includes question-answering tasks based on document retrieval, ensuring that the model can handle single-turn question-answering questions. For example, when a user queries information about a specific document segment, the model answers based on the retrieved content. This kind of data can train the basic ability of the model in document question-answering tasks, enabling it to quickly locate and integrate key information when facing common questions. Collect English document question-answering data from the CHATQA dataset, which contains various document question-answering tasks, covering scenarios such as open-domain question-answering and technical document parsing, enabling the model to better adapt to different types of English enterprise documents and improving the document retrieval and question-answering ability in a cross-language environment.

[0133] This embodiment also collects Chinese and English RAG data from the TruthReader_zh dataset and the TruthReader_en dataset respectively. Different from the PAMQA and CHATQA data, the TruthReader dataset contains document Q&A data in the form of multi-turn conversations, enabling the model to better handle complex Q&A tasks that require multi-turn interactive reasoning. For example, a user may pose a general question in the first turn and follow up with a more in-depth question based on the answer in the second turn. The model needs to have good context understanding ability to provide coherent and accurate answers.

[0134] Since the formats of different datasets vary, to ensure the uniformity of training data, all data needs to undergo format standardization. For example, during the conversion process, it is necessary to ensure that each question is accompanied by the corresponding reference document fragment, and the model can clearly distinguish between answerable and unanswerable situations during training. In addition, the data may contain unrecognized samples, that is, when the retrieved fragment cannot answer the user's question, the data should be marked as "no relevant content found" to improve the model's unrecognized ability. This format-unifying process not only makes the training data more targeted but also improves the stability of the model in document Q&A tasks.

[0135] This embodiment enhances the model's task adaptability by introducing multiple data sources, covering single-turn and multi-turn Q&A. PAMQA data provides the accuracy of single-turn document Q&A, CHATQA data expands the model's adaptability to English documents, and the TruthReader dataset improves the model's understanding ability of multi-turn conversations, enabling it to provide a more natural interaction experience in complex enterprise application scenarios. Through multi-language data collection, the model's cross-language Q&A ability is improved, making it applicable not only to Chinese enterprise environments but also to English documents or international enterprises. In addition, the conversion of the document Q&A prompt format ensures data standardization, improving training efficiency and the model's generalization ability in different document Q&A tasks.

[0136] In an exemplary embodiment of the present invention, one embodiment of obtaining enterprise internal data is as follows: parsing the enterprise internal documents of the document type to obtain single-turn Q&A data, multi-turn Q&A data, and recognition data; constructing the enterprise internal documents of the Q&A pair into standard format data of the RAG system; and using the single-turn Q&A data, the multi-turn Q&A data, the recognition data, and the standard format data as the enterprise internal data.

[0137] This implementation method parses the internal documents of an enterprise and converts different types of data into a standard format suitable for the RAG system to enhance the adaptability of the model to specific enterprise scenarios. The internal documents of an enterprise are mainly divided into two categories: document data and question-and-answer pair data, which play different roles in the training of the question-and-answer model. In the actual application process, for the internal enterprise data of the document type, specific parsing methods are adopted to extract effective information and construct different types of question-and-answer data. Among them, single-round question-and-answer data refers to extracting question-answer pairs from enterprise documents and ensuring that each question can derive a clear answer from the document content. The construction of single-round question-and-answer data helps the model master the core knowledge of enterprise documents and provide accurate answers when users ask questions. The construction of multi-round question-and-answer data simulates the continuous question-asking scenarios of actual users. To enable the model to learn to refuse answers reasonably, this implementation method also constructs rejection recognition data, that is, for questions for which no answers can be found in the document, it ensures that the model can refuse to answer when it cannot infer the answer, rather than generating incorrect or unfounded answers. For the internal enterprise documents of the question-and-answer pair type, this implementation method converts them into the standard format of the RAG system for use in the retrieval-augmented generation process. Question-and-answer pair data usually comes from the enterprise's FAQs, customer support records, etc., and has the characteristics of high structurality. After being converted into the standard format, these data can be used as part of the enterprise knowledge base to ensure that the RAG system can match the most relevant question-and-answer pairs during the retrieval stage and provide more accurate support when generating answers.

[0138] By parsing the document data and constructing single-round question-and-answer, multi-round question-and-answer, and rejection recognition data, the model can better learn the enterprise knowledge structure, accurately answer users' questions when necessary, and also refuse to answer when necessary, improving the credibility of the enterprise-level question-and-answer system. By constructing question-and-answer pair data in the standard format, this implementation method optimizes the retrieval process of the RAG system, enabling it to more efficiently match users' questions and provide standardized answers, reducing the computational burden on the generative model.

[0139] In an exemplary embodiment of the present invention, an implementation method for parsing the internal enterprise documents of the document type to obtain single-round question-and-answer data is as follows: using a large language model to generate questions based on the internal enterprise documents of the document type; using the large language model to generate answers based on the questions and the internal enterprise documents of the document type corresponding to the questions; and annotating the internal enterprise documents of the document type, the questions, and the answers to obtain the single-round question-and-answer data.

[0140] This implementation uses a Large Language Model (LLM) to parse enterprise internal documents, automatically generate high-quality Q&A data, and ensure the accuracy and effectiveness of the data through the annotation process. In the actual application process, in the question generation stage, the LLM is used to automatically generate questions from the document-type data within the enterprise. Enterprise documents usually cover various types of information, such as product descriptions, operation manuals, legal contracts, customer support documents, etc., and the LLM can extract key information based on these documents and generate questions closely related to the content. In the answer generation stage, the LLM not only needs to understand the generated questions but also generate reasonable answers based on the content of the enterprise documents corresponding to the questions. Different from general Q&A tasks, enterprise-level document Q&A emphasizes accuracy and traceability, that is, the model's answers must be strictly based on the retrieved enterprise documents and cannot rely on its parameter memory or external knowledge. In the data annotation stage, the generated questions, answers, and corresponding enterprise internal documents are reviewed and annotated to ensure the quality of the data. Since the LLM may produce errors or redundant information when generating Q&A pairs, manual annotation or rule-based automatic screening is a necessary step.

[0141] This implementation uses the LLM to automatically generate questions and answers, significantly reducing the time cost of manually constructing Q&A data and ensuring that the data covers the core knowledge of enterprise documents. It ensures the traceability and controllability of the Q&A data, that is, all answers are strictly based on enterprise documents. Through the data annotation process, it ensures that the data finally used for training is of high quality and reduces the interference of incorrect samples on model training.

[0142] In an exemplary embodiment of the present invention, one implementation of parsing the document-type enterprise internal documents to obtain multi-round Q&A data is as follows: using the document-type enterprise internal documents and the single-round Q&A data as the first-round input items; according to the first-round input items, using the LLM to predict subsequent multi-round questions; expanding the multi-round questions into the multi-round Q&A data.

[0143] This implementation mode combines document parsing and LLM to automatically generate and expand multi-turn Q&A data to simulate user interactions in real scenarios. During the actual application process, in the first-round input stage, the in-company document data within the enterprise and the existing single-turn Q&A data are used as input items, providing the initial context for multi-turn Q&A. In-company documents within the enterprise usually contain a large amount of business knowledge, such as product descriptions, operation manuals, technical white papers, etc., and the single-turn Q&A data is specific questions and answers extracted from these documents. These data provide the problem background and basic document information for the model, and the model needs to understand and generate the subsequent conversation content based on this information. In the enterprise application scenario, the single-turn Q&A data may have answered the user's initial questions, while the multi-turn Q&A data simulates the situation where the user may further ask questions after the initial Q&A. Next, in the subsequent question generation stage, based on the input single-turn Q&A data and document content, the large language model will reason and predict according to the context to generate logical subsequent questions. These questions may involve in-depth discussions of previous answers or further inquiries about a specific detail. Finally, in the expansion stage of multi-turn Q&A data, the generated multi-turn questions and answers are combined to form complete multi-turn Q&A data. Each multi-turn question should contain context information, and each answer should be generated based on the relevant content in the document. The generated multi-turn Q&A data will be used as part of the training set to optimize the model's reasoning ability and context understanding ability in long conversations, thereby improving the performance of the Q&A system.

[0144] By using in-company document data and single-turn Q&A data as the initial input, this implementation mode can efficiently expand the conversation content, avoiding the cost of manually generating a large amount of multi-turn Q&A data. At the same time, based on the reasoning ability of the large language model, logical and business-related questions can be generated to simulate real user interactions.

[0145] In an exemplary embodiment of the present invention, an implementation mode of parsing the in-company documents of the document type to obtain knowledge data is as follows: using the large language model to extract questions from the in-company documents of the document type; excluding the in-company documents to which the questions belong during the retrieval of retrieval fragments; combining the retrieved other text fragments with the questions; and using the combination as the knowledge data.

[0146] In this implementation, the large language model is used to extract questions from the document - type data within the enterprise. Enterprise documents usually contain a large amount of professional knowledge, operation instructions, or technical details. However, the information in these documents may not directly correspond to all user queries. Through the large language model, questions related to the content of the documents can be automatically extracted. These questions may be generated based on the key information in the documents or knowledge in specific domains. Next, retrieval fragment retrieval is a key step in constructing rejection - recognition data. According to the question, the model will conduct document retrieval and try to find relevant text fragments to answer the question. In this implementation, the participation of the enterprise's internal document - type documents in the retrieval process is excluded, that is, the model is not allowed to use the content in this document to generate answers. This simulates the situation in actual applications where answers cannot be obtained from existing documents. After obtaining the retrieval results, the retrieved text fragments are combined with the question. This step is not to provide an answer but to construct training data for the model when facing situations where it cannot answer. By combining the retrieved irrelevant text fragments with the question to form rejection - recognition data, the model can learn how to identify and reject answering when it cannot find relevant answers in the document, rather than generating unfounded answers. The construction of rejection - recognition data is very important for improving the accuracy and reliability of the model, especially in a complex enterprise environment where the model must be able to effectively distinguish which questions cannot be answered.

[0147] In this implementation, by explicitly excluding the irrelevant content in the document, the model can learn to reasonably reject answering when it cannot find an answer, rather than misleading the user. The generation of rejection - recognition data not only helps reduce the generation of incorrect answers by the model but also enhances the reliability of the model, avoiding the situation of giving incorrect information due to over - confidence.

[0148] In an exemplary embodiment of the present invention, one implementation of constructing the question - and - answer pair of the enterprise's internal documents into the standard format data of the RAG system is as follows: extracting similar questions from the question - and - answer pair of the enterprise's internal documents as query data; retrieving corresponding standard question - and - answer pairs based on the query data; retrieving similar question - and - answer pairs and random question - and - answer pairs as alternative question - and - answer pairs; constructing the standard question - and - answer pairs and the alternative question - and - answer pairs into the standard format data, and using the standard question - and - answer pairs as the true labels of the query data.

[0149] This implementation extracts similar questions from the Q&A pairs in enterprise internal documents as query data. The core purpose of this step is to convert the actual questions and answers in enterprise internal documents into a standard format suitable for the system. Most enterprise documents contain a lot of frequently asked questions (FAQ) and answers, which have strong domain relevance and can provide a rich data source for subsequent model training. By extracting these questions, the system can better understand the key information in enterprise documents and common customer queries. Next, based on the query data, a search is performed to try to find the corresponding standard Q&A pairs. These standard Q&A pairs are usually screened manually or automatically, with high quality and high relevance, and can accurately answer the questions raised by users. Through the search, the most relevant answers can be quickly found from the database or knowledge base, improving the response speed and accuracy. At the same time, it also includes retrieving similar Q&A pairs and random Q&A pairs as alternative Q&A pairs, enabling it to still generate reasonable answers when encountering queries that do not exactly match. By retrieving similar Q&A pairs, more alternative answers can be provided for the questions raised by users, and by randomly selecting Q&A pairs, the range of answers of the model can be further enriched. Finally, the standard Q&A pairs and alternative Q&A pairs are constructed into data in a standard format that conforms to the RAG system. This means that each group of Q&A pairs will be uniformly formatted to ensure compatibility in the RAG system.

[0150] Through the extraction and retrieval of similar questions, this implementation can more accurately understand the user's intent, and on the basis of finding the standard answer, it can also provide alternative answers, ensuring efficient answer generation and diverse answer options. In addition, by uniformly formatting the Q&A pair data, the compatibility and training efficiency of the system are improved, enabling the RAG system to more efficiently handle complex document Q&A tasks.

[0151] Based on the above related description of a method for training a document Q&A model for enterprises, the following introduces an optimization method for the RAG model for enterprise internal use. By sorting out and analyzing enterprise data, private document Q&A training data is constructed, and at the same time, high-quality general SFT data is collected for cooperation in training. Verification in actual document scenarios shows that it can greatly improve the accuracy of the enterprise document Q&A scenario and at the same time enhance the model's rejection ability.

[0152] This optimization method involves three stages of work: data preparation, model training, and model evaluation.

[0153] (1) Data preparation

[0154] 1. General supervised fine-tuning data preparation: For the training of document question-answering tasks, if only document data is used, catastrophic forgetting may occur, resulting in the loss of most general capabilities of the model. Therefore, a considerable proportion of high-quality general SFT data needs to be collected for mixed training. Currently, tens of millions of general supervised fine-tuning data have been open-sourced in the industry, but most of the data has poor quality and has not been verified. Therefore, this solution traces the SFT datasets used by high-quality open-source base models to reverse-locate high-quality data sources as the data foundation for the basic capabilities of the model. Experiments show that retaining a considerable proportion of high-quality general SFT data in the SFT stage can significantly retain the general capabilities of the model, thereby ensuring the effectiveness in the document question-answering scenario.

[0155] 1) CuteGPT dataset - Sampled 11,000 items, including 13 datasets ('Dialogue data': {'belle_sampled': 1500, 'gpt4llm': 1500, 'alpaca_cn': 1500, 'instinwild_cn': 2000}, 'NLP task data': {'pclue': 600, 'firefly': 900}, 'Multi-turn dialogue data': {'sharegpt_cn': 2000, 'belle_multiturn': 700}, 'Other data': {'sd_multiturn': 200, 'from_cld': 40, 'from_hyc': 10, 'from_hwh': 30, 'from_zyk': 20}), using the original prompt.

[0156] 2) Infinity Instruct dataset (open-sourced by the Zhiyuan Institute) - All 7,972 Chinese data items, using the original prompt.

[0157] 3) Infinity Instruct dataset (open-sourced by the Zhiyuan Institute) - Sampled 20,000 English data items: For data with major labels 'Natural Language Processing and Understanding' or 'Information Processing and Integration' or 'Question Answering and Support', 12,000 items are sampled, and for the remaining label data, 8,000 items are sampled, using the original prompt.

[0158] 2. General document question-answering data: For the general document question-answering data that has been open-sourced in the industry, this solution constructs a unified standard processing flow and processes it into document question-answering data with consistent formats for combined training.

[0159] 1) PAMQA data [3] - Sample 40,000 pieces from the original dataset. Among them, 35,733 pieces have answers, and the remaining 4,267 pieces only predict the reference numbers. Chinese single-round RAG document Q&A data, with reference articles or retrieved content. All Q&A data is changed to a unified document Q&A prompt.

[0160] 2) CHATQA data [2] - Sample 20,000 pieces from 7 datasets in the original dataset. Among them, there are 3,937 multi-round data {drop: 1k, narrativeqa: 2k, quoref: 2k, squad2.0: 2k, synthetic_convqa: 5k, tatqa_train_others: 3k, tatqa_train_arithmetic: 5k}. English document Q&A data, with both single-round and multi-round Q&A, including 173 unrecognized data. Changed to a unified English document Q&A prompt.

[0161] 3) TruthReader_zh - All 4,716 pieces of data, Chinese RAG data, multi-round, including 117 unrecognized data. All Q&A data is changed to a unified document Q&A prompt.

[0162] 4) TruthReader_en - All 2,442 pieces of data, English RAG data, multi-round, including 111 unrecognized data. Changed to a unified English document Q&A prompt.

[0163] 3. Internal data preparation. Generally, there is no processed and mature document Q&A dataset within the enterprise. After sorting, there are two major types of data sources, namely document data and Q&A pair data. The forms of enterprise internal documents are diverse, and suitable parsing tools need to be used for different formats of documents to extract the document content. For the document data in this solution, a complete process from original content parsing to automated single-round Q&A pair generation, multi-round Q&A data expansion, unrecognized data construction, and data annotation review is constructed to form a document Q&A dataset within the enterprise. For the Q&A pair type of data, it is constructed into the standard format used by the RAG system, and then it is kept consistent with the data retrieved during the actual application process of the RAG system.

[0164] 1) After testing, the currently used parsing tools with good effects are as follows: Use PyPDF2 for PDF files, DeepDoc for DOCX files, convert DOC files to DOCX first and then parse, use Unstructured-IO for PPTX files, use XLRD for XSL files, and use PANDAS for XLSX files.

[0165] 2) Single-document Q&A data - 3,845 entries. According to the internal document, refer to the SelfQA process to use the large model to automatically generate Q&A pairs based on the document to supplement the training set, and then use manual annotation and screening. Refer to Figure 2 , first, the LLM generates questions Q in batches according to the original document Doc; then the LLM gives answers A according to each question Q and the corresponding document Doc; finally, the <document Doc, question Q, answer A> data is manually annotated and screened. Answers that cannot be inferred from the original document are classified as unrecognized data. During training, change to a unified document Q&A prompt.

[0166] 3) Multi-turn document Q&A data - 3,845 entries. Refer to Figure 3 , use the LLM to generate. Input the original document Doc and single-turn Q&A data as Round1 (the first round) of the multi-turn Q&A. Use the LLM to guess the user's next four questions, and each question is expanded into multi-turn Doc, Q, A data for Round2 - Round5. The generated multi-turn Q&A data is manually annotated. In addition to verifying the correctness of the answers, the questions also need to be quality-checked and description-optimized to make the questions more continuous, and the model can supplement the omitted conditions in the multi-turn Q&A according to the above text.

[0167] 4) Document Q&A unrecognized data - 1,000 entries. Since the large model is used to extract Q&A pairs from the document, most questions can find answers in the document, which will result in too few unrecognized data, and the model cannot learn how to distinguish answerable / rejectable situations from training. Therefore, it is necessary to construct unrecognized-related training data. When constructing unrecognized data, the questions are still the questions extracted from the document by the large model, but when retrieving the reference fragments, the original document is removed, so that the retrieved text fragments cannot answer the question, and the answer is marked as "no relevant content found".

[0168] 5) FAQ high-frequency Q&A pair generation data - 8,366 entries. Refer to Figure 4, when using the RAG engine, in many scenarios, high-frequency Q&A pairs data will be uploaded to the document library, and many Q&A pairs will be included in the retrieved chunks. Therefore, it is necessary to train the model's ability to answer user questions based on Q&A pairs. Use FAQ data in the business scenario. Similar questions are used as Queries, and the corresponding standard QA pairs are found according to the similar questions, and the retrieved top 20 similar QA pairs and random 20 QA pairs are used as generation alternatives. The standard answer to the corresponding standard question is used as the Output, and the retrieved similar QA pairs and random QA pairs are mixed in to simulate the RAG recall of document Chunks. Construct 5 Chunks (reference documents), with 4 pairs of Q&A in each Chunk (2 from the top 20 and 2 from the random), and one Chunk contains the standard QA. Each Chunk starts with a number 《n》, recording the Chunk number where the standard answer is located. During training, change it to a unified document Q&A prompt.

[0169] (II) Model Training

[0170] Based on the basic model, all data with the above sampling ratio is used for full-parameter fine-tuning, and DeepSpeed is used to achieve 3D parallel training on 4 machines with 32 cards.

[0171] The unified Chinese prompt template is as follows:

[0172] ##Role: Knowledge Base Assistant

[0173] ##Background:

[0174] - The Knowledge Base Assistant is a Q&A assistant integrated with a search engine. It can understand and process user questions and give answers based on the retrieved reference content.

[0175] # Reference Text

[0176] ${chunks}

[0177] # Historical Conversation Records

[0178] ${history}

[0179] ##Constraints:

[0180] 1. Answer the user's question in Chinese, and the answer should be comprehensively answered based on the reference content.

[0181] 2. Since the

Reference Text

[0182] 3. If the content in the

Reference Text

[0183] 4. If there are questions involving mathematical calculations or reasoning, please stay calm and answer after thinking step by step.

[0184] # Fragment Tracing Requirements

[0185] Please append the "number" in the angle brackets of the "knowledge base" fragment referred to in each paragraph of the answer at the end of the answer in the format of "[Reference Knowledge Base <n1> <n2>...]", and pay attention to adding "[]".

[0186] # User Question:

[0187] ${orgQuestion}

[0188] # Your Answer:

[0189] (3) Model Evaluation

[0190] Evaluation Metrics:

[0191] Rule-based Metrics: BLEU (based on n-gram matching), Rouge-L (based on the longest common subsequence), CHRF++ (word-level + character-level n-gram matching)

[0192] Evaluation Results:

[0193] Compared with the basic large model, due to a certain proportion of general SFT data in the fine-tuning stage, the general ability score of the fine-tuned model is basically maintained.

[0194] In the trained scenarios: BLEU-4 (14.9 -> 49.5), Rouge-L (0.43 -> 0.74), CHRF++ (39.9 -> 62.6). The model can learn effective information and has a significant improvement.

[0195] In the scenarios not seen during training, BLEU-4 (8.2 -> 26.1), Rouge-L (0.30 -> 0.40), CHRF++ (34.4 -> 44.1). The model can learn effective knowledge from the trained scenarios and also bring improvements in the unseen scenarios.

[0196] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequence, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present invention.

[0197] Reference Figure 5 As shown in Figure 5 , a structural block diagram of a document question - answering model training system for enterprises according to an embodiment of the present invention is shown. The document question - answering model training system for enterprises may specifically include the following modules.

[0198] A data acquisition module 51, configured to acquire general supervised fine - tuning data, general document question - answering data, and enterprise internal data;

[0199] A data mixing module 52, configured to mix the general supervised fine - tuning data, the general document question - answering data, and the enterprise internal data in a predetermined ratio to obtain training data;

[0200] A model fine - tuning module 53, configured to perform supervised fine - tuning training on a basic large - language model based on the training data according to a unified document question - answering prompt word template to obtain a document question - answering model.

[0201] In an exemplary embodiment of the present invention, the data acquisition module 51 includes:

[0202] A first data collection module, configured to collect conversation data, NLP task data, multi - turn conversation data, and other data from the CuteGPT dataset;

[0203] A second data collection module, configured to collect Chinese data from the Infinity Instruct dataset;

[0204] A third data collection module, configured to collect English data from the Infinity Instruct dataset;

[0205] A general supervised fine - tuning data determination module, configured to use the conversation data, the NLP task data, the multi - turn conversation data, the other data, the Chinese data, and the English data as the general supervised fine - tuning data.

[0206] In an exemplary embodiment of the present invention, the data acquisition module 51 includes:

[0207] A fourth data collection module, configured to collect Chinese single - turn RAG document question - answering data from the PAMQA dataset;

[0208] A fifth data collection module, configured to collect English document question - answering data from the CHATQA dataset;

[0209] A sixth data collection module, configured to collect Chinese RAG data from the TruthReader_zh dataset;

[0210] A seventh data collection module, configured to collect English RAG data from the TruthReader_en dataset;

[0211] A general document Q&A data determination module, configured to convert the Chinese single-round RAG document Q&A data, the English document Q&A data, the Chinese RAG data, and the English RAG data into the document Q&A prompt word format and use it as the general document Q&A data.

[0212] In an exemplary embodiment of the present invention, the data acquisition module 51 includes:

[0213] A document parsing module, configured to parse the enterprise internal documents of the document type to obtain single-round Q&A data, multi-round Q&A data, and data recognition data;

[0214] A standard data construction module, configured to construct the enterprise internal documents of the Q&A pairs into the standard format data of the RAG system;

[0215] An enterprise internal data determination module, configured to use the single-round Q&A data, the multi-round Q&A data, the data recognition data, and the standard format data as the enterprise internal data.

[0216] In an exemplary embodiment of the present invention, the document parsing module includes:

[0217] A question generation module, configured to generate questions according to the enterprise internal documents of the document type by using a large language model;

[0218] An answer generation module, configured to generate answers according to the questions and the enterprise internal documents of the document type corresponding to the questions by using a large language model;

[0219] A single-round Q&A data determination module, configured to label the enterprise internal documents of the document type, the questions, and the answers to obtain the single-round Q&A data.

[0220] In an exemplary embodiment of the present invention, the document parsing module includes:

[0221] An input item determination module, configured to use the enterprise internal documents of the document type and the single-round Q&A data as the first-round input items;

[0222] A multi-round question prediction module, configured to predict subsequent multi-round questions according to the first-round input items by using a large language model;

[0223] A multi-round Q&A expansion module, configured to expand the multi-round questions into the multi-round Q&A data.

[0224] In an exemplary embodiment of the present invention, the document parsing module includes:

[0225] A question extraction module, configured to extract questions from the enterprise internal documents of the document type by using a large language model;

[0226] A document exclusion module, which is used to exclude the enterprise internal documents to which the problem belongs when performing retrieval of retrieval fragments;

[0227] A combination module, which is used to combine the retrieved other text fragments with the problem;

[0228] An identification data determination module, which is used to use the combination as the identification data.

[0229] In an exemplary embodiment of the present invention, the standard data construction module includes:

[0230] A similar problem extraction module, which is used to extract similar problems from the Q&A pair enterprise internal documents as query data;

[0231] A standard Q&A pair retrieval module, which is used to retrieve corresponding standard Q&A pairs based on the query data;

[0232] An alternative Q&A pair determination module, which is used to retrieve similar Q&A pairs and random Q&A pairs as alternative Q&A pairs;

[0233] A standard format data determination module, which is used to construct the standard Q&A pairs and the alternative Q&A pairs into the standard format data, and use the standard Q&A pairs as the true labels of the query data.

[0234] For the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For related parts, refer to the partial description of the method embodiment.

[0235] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, refer to each other.

[0236] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention can take the form of completely hardware embodiments, completely software embodiments, or embodiments combining software and hardware aspects. Moreover, the embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0237] Embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or in one or more blocks.

[0238] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or in one or more blocks.

[0239] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, such that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or in one or more blocks.

[0240] Although the preferred embodiments of the embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concepts. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0241] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising said element.

[0242] The above has introduced in detail a method for training a document question-answering model for enterprises and a system for training a document question-answering model for enterprises provided by the present invention. Specific examples are used in this text to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for training a document question-answering model for enterprises, characterized in that, The method includes: Obtaining general supervised fine-tuning data, general document Q&A data, and enterprise internal data; Mixing the general supervised fine-tuning data, the general document Q&A data, and the enterprise internal data according to a predetermined ratio to obtain training data; Based on the training data, performing supervised fine-tuning training on the basic large language model according to a unified document Q&A prompt template to obtain a document Q&A model.

2. The method according to claim 1, characterized in that, The obtaining of the general supervised fine-tuning data includes: Collecting dialogue data, NLP task data, multi-turn dialogue data, and other data from the CuteGPT dataset; Collecting Chinese data from the Infinity Instruct dataset; Collecting English data from the Infinity Instruct dataset; Using the dialogue data, the NLP task data, the multi-turn dialogue data, the other data, the Chinese data, and the English data as the general supervised fine-tuning data.

3. The method according to claim 1, wherein The obtaining of the general document Q&A data includes: Collecting Chinese single-turn RAG document Q&A data from the PAMQA dataset; Collecting English document Q&A data from the CHATQA dataset; Collecting Chinese RAG data from the TruthReader_zh dataset; Collecting English RAG data from the TruthReader_en dataset; Converting the Chinese single-turn RAG document Q&A data, the English document Q&A data, the Chinese RAG data, and the English RAG data into the document Q&A prompt format and using them as the general document Q&A data.

4. The method according to claim 1, wherein The obtaining of the enterprise internal data includes: Parsing the enterprise internal documents of the document type to obtain single-turn Q&A data, multi-turn Q&A data, and entity recognition data; Constructing the enterprise internal documents of the Q&A pair into the standard format data of a RAG system; Using the single-turn Q&A data, the multi-turn Q&A data, the entity recognition data, and the standard format data as the enterprise internal data.

5. The method according to claim 4, characterized in that The parsing of the enterprise internal documents of the document type to obtain single-turn Q&A data includes: Using a large language model to generate questions based on the enterprise internal documents of the document type; Using a large language model to generate answers based on the questions and the corresponding enterprise internal documents of the document type; Annotating the enterprise internal documents of the document type, the questions, and the answers to obtain the single-turn Q&A data.

6. The method according to claim 4 or 5, characterized in that, The parsing of the enterprise internal documents of the document type to obtain multi-turn Q&A data includes: Using the enterprise internal documents of the document type and the single-turn Q&A data as the first-round input items; Based on the first-round input items, using a large language model to predict subsequent multi-turn questions; Expanding the multi-turn questions into the multi-turn Q&A data.

7. The method according to claim 4, wherein The parsing of the enterprise internal documents of the document type to obtain entity recognition data includes: Using a large language model to extract questions from the enterprise internal documents of the document type; Excluding the enterprise internal documents to which the questions belong during the retrieval of retrieval fragments; Combining the retrieved other text fragments with the questions; Using the combination as the entity recognition data.

8. The method according to claim 4, wherein The constructing of the enterprise internal documents of the Q&A pair into the standard format data of a RAG system includes: Extract similar questions from the Q&A pair enterprise internal documents as query data; Retrieve corresponding standard Q&A pairs based on the query data; Retrieve similar Q&A pairs and random Q&A pairs as alternative Q&A pairs; Construct the standard Q&A pairs and the alternative Q&A pairs into the standard format data, and use the standard Q&A pairs as the true labels of the query data.

9. An enterprise-oriented document question-answering model training system, characterized in that, The system includes: A data acquisition module for acquiring general supervised fine-tuning data, general document Q&A data, and enterprise internal data; A data mixing module for mixing the general supervised fine-tuning data, the general document Q&A data, and the enterprise internal data in a predetermined ratio to obtain training data; A model fine-tuning module for performing supervised fine-tuning training on the basic large language model based on the training data according to a unified document Q&A prompt word template to obtain a document Q&A model.

10. An electronic device, characterized in that, Including: One or more processors; And One or more machine-readable media storing instructions that, when executed by the one or more processors, cause the electronic device to execute the enterprise-oriented document Q&A model training method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer program stored therein causes the processor to execute the enterprise-oriented document Q&A model training method according to any one of claims 1 to 8.

Images