Enterprise intelligent question answering system dataset collection method and device

Abstract

The application discloses an enterprise intelligent question and answer system data set collection method and device, the method comprises the following steps: collecting and arranging enterprise text data, the enterprise text data comprises text materials, chat records and self-defined task data; labeling the chat records and the self-defined task data to obtain a seed data set; inputting the seed data set into an existing large language model to generate a question and answer data set, and evaluating and iteratively optimizing the generated question and answer data set until the data amount of the generated data set reaches a preset data amount. The scheme can improve the efficiency and diversity of enterprise question and answer data set collection and reduce the cost of manual labeling.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a method, apparatus, computing device, and storage medium for collecting datasets for an enterprise intelligent question-and-answer system. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have become one of the hottest research directions in the field of natural language processing. Vertical domain models based on large language models, as an extension of large language models, possess higher intelligence and accuracy.

[0003] Compared to general-purpose language models, vertical domain models possess stronger specialized domain knowledge and comprehension capabilities, enabling them to provide more accurate and professional answers to questions specific to a particular domain. Vertical domain models can also interface with an enterprise's existing knowledge base, achieving intelligent management and updates to the knowledge base.

[0004] Current technologies have relatively small-scale question-answering datasets, which cannot cover rich semantic information and thus limit the performance of the models. Furthermore, the wide variety of sources for question-answering datasets leads to significant differences in data quality and accuracy, and the infrequent appearance of certain questions and answers results in uneven distribution of the dataset samples. Summary of the Invention

[0005] To increase the data scale and sample distribution balance, this solution proposes a data acquisition method, device, computing equipment, and storage medium for an enterprise intelligent question-answering system. It utilizes existing large language models to perform data augmentation on some labeled question-answer data, thereby improving the diversity and generalization ability of the dataset. In addition, iterative generation of question-answer datasets based on semi-supervised learning can greatly reduce the errors and costs of manual annotation.

[0006] According to a first aspect of the present invention, a method for collecting datasets for an enterprise intelligent question-and-answer system is provided, comprising: collecting and organizing enterprise text data, including text documents, chat logs, and custom task data; labeling chat logs and custom task data to obtain a seed dataset; inputting the seed dataset into an existing large language model to generate a question-and-answer dataset; evaluating and iteratively optimizing the generated question-and-answer dataset until the data volume of the generated dataset reaches a preset data volume.

[0007] Optionally, in the enterprise intelligent question-answering system dataset collection method provided by the present invention, a knowledge graph with information points as the basic unit can also be constructed based on the document information in the enterprise text data.

[0008] Optionally, in the enterprise intelligent question-answering system dataset collection method provided by the present invention, the text is split according to the title content, and each title content is stored modularly; text content from different sources is integrated in the modular title content to obtain modular text data; entity recognition is performed on the modular text data, and the relationship between entities is extracted; entities and entity relationships are represented in the form of information points, and connections between information points are established in the knowledge graph.

[0009] Optionally, in the enterprise intelligent question-answering system dataset collection method provided by the present invention, the dialogue and custom task data are labeled according to the preset labeling specifications and criteria; the labeling results are quality evaluated and reviewed, and feedback is given and corrected for labeling errors or inconsistencies, and then a seed dataset is obtained.

[0010] Optionally, in the enterprise intelligent question answering system dataset collection method provided by the present invention, the large language model is fine-tuned using data in the enterprise domain of the knowledge graph to obtain a trained large language model; the seed dataset is input into the trained model for prediction to automatically generate a dialogue dataset; the questions in the dialogue dataset are used as input to generate new questions and corresponding answers through the large language model, and the generated questions and answers are combined as a new question answering dataset.

[0011] Optionally, in the enterprise intelligent question answering system dataset collection method provided by the present invention, a basic large language model suitable for the enterprise domain is selected as the initial model; the initial model is fine-tuned using enterprise domain data in the knowledge graph; the fine-tuned model is evaluated, and the fine-tuned model is further optimized and adjusted according to the evaluation results until the expected performance indicators are achieved.

[0012] Optionally, in the enterprise intelligent question-answering system dataset collection method provided by the present invention, the generated question-answering dataset is scored and corrected according to the data quality, accuracy and usability indicators; based on the feedback and scores of the annotators, the corrected data is re-input into the large language model to regenerate the dataset; when the generated dataset reaches the required amount of data, the dataset iterative generation process ends.

[0013] According to a second aspect of the present invention, a dataset acquisition device for an enterprise intelligent question-answering system is provided, comprising: a collection module, a labeling module, and an iterative generation module. The collection module is used to collect and organize enterprise text data, which includes text documents, chat logs, and custom task data; the annotation module is used to annotate chat logs and custom task data to obtain a seed dataset. The iterative generation module is used to input the seed dataset into an existing large language model to generate a question-and-answer dataset, and to evaluate and iteratively optimize the generated question-and-answer dataset until the amount of data in the generated dataset reaches the preset amount.

[0014] According to a third aspect of the present invention, a computing device is provided, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions including instructions for performing the above-described enterprise intelligent question-and-answer system dataset acquisition method.

[0015] According to a fourth aspect of the present invention, a readable storage medium storing program instructions is provided, which, when read and executed by a computing device, causes the computing device to perform the above-described enterprise intelligent question-and-answer system dataset collection method.

[0016] According to the enterprise intelligent question-answering system dataset collection method and apparatus of the present invention, data augmentation is performed on partially labeled question-answer data using existing large language models to improve the diversity and generalization ability of the dataset. In addition, the question-answer dataset generated iteratively based on semi-supervised learning can greatly reduce the error and cost of manual annotation.

[0017] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A structural block diagram of a computing device 100 according to an embodiment of the present invention is shown; Figure 2 A flowchart illustrating a data collection method 200 for an enterprise intelligent question-answering system according to an embodiment of the present invention is shown. Figure 3 A schematic diagram of the structure of an enterprise intelligent question-answering system dataset acquisition device 300 according to an embodiment of the present invention is shown. Detailed Implementation

[0019] To reduce the amount of manual annotation required in the process of generating question-and-answer datasets, this solution provides a method, device, computing equipment, and storage medium for collecting datasets for an enterprise intelligent question-and-answer system. It utilizes existing large language models to transform part of the manually annotated dataset into output questions and answers. Through interactive iterative learning, the quality and accuracy of the generated question-and-answer pairs can be gradually improved.

[0020] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0021] Figure 1 A structural diagram of a computing device 100 according to an embodiment of the present invention is shown. Figure 1 As shown, in the basic configuration 102, the computing device 100 typically includes system memory 106 and one or more processors 104. Memory bus 108 can be used for communication between processor 104 and system memory 106.

[0022] Depending on the desired configuration, processor 104 can be any type of processor, including but not limited to: microprocessors (µP), microcontrollers (µC), digital information processors (DSPs), or any combination thereof. Processor 104 may include one or more levels of cache such as L1 cache 110 and L2 cache 112, processor core 114, and registers 116. Example processor core 114 may include an arithmetic logic unit (ALU), a floating-point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. Example memory controller 118 may be used with processor 104, or in some implementations, memory controller 118 may be an internal part of processor 104.

[0023] Depending on the desired configuration, system memory 106 can be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. Physical memory in a computing device typically refers to volatile RAM, and data on a disk needs to be loaded into physical memory before it can be read by processor 104. System memory 106 may include operating system 120, one or more applications 122, and program data 124.

[0024] In some implementations, application 122 may be configured to execute instructions on an operating system using program data 124 by one or more processors 104. The operating system 120 may be, for example, Linux, Windows, etc., and includes program instructions for handling basic system services and performing hardware-dependent tasks. Application 122 includes program instructions for implementing various user-desired functions; application 122 may be, for example, a browser, instant messaging software, software development tools (e.g., integrated development environment IDE, compiler, etc.), but is not limited thereto. When application 122 is installed in computing device 100, driver modules may be added to operating system 120.

[0025] When computing device 100 starts up, processor 104 reads and executes program instructions from memory 106 of operating system 120. Application 122 runs on operating system 120, utilizing interfaces provided by operating system 120 and underlying hardware to implement various user-expected functions. When user starts application 122, application 122 is loaded into memory 106, and processor 104 reads and executes program instructions from memory 106 of application 122.

[0026] The computing device 100 also includes a storage device 132, which includes a removable storage device 136 and a non-removable storage device 138, both of which are connected to a storage interface bus 134.

[0027] The computing device 100 may also include an interface bus 140 that facilitates communication from various interface devices (e.g., output devices 142, peripheral interfaces 144, and communication devices 146) to the basic configuration 102 via a bus / interface controller 130. Example output devices 142 include a graphics processing unit 148 and an audio processing unit 150. They may be configured to facilitate communication with various external devices such as displays or speakers via one or more A / V ports 152. Example peripheral interfaces 144 may include a serial interface controller 154 and a parallel interface controller 156, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 158. Example communication devices 146 may include a network controller 160, which may be arranged to facilitate communication with one or more other computing devices 162 via a network communication link through one or more communication ports 164.

[0028] A network communication link may be an example of a communication medium. Communication media typically may be embodied as computer readable instructions, data structures, program modules in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information within the signal. By way of non-limiting example, communication media may include wired media such as a wired network or dedicated-line network, and various wireless media such as sound, radio frequency (RF), microwave, infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media. In the computing device 100 according to the present invention, the application 122 comprises instructions for executing the enterprise intelligent question-answering system dataset collection method 200 of the present invention.

[0029] Figure 2 shows a schematic flowchart of the enterprise intelligent question-answering system dataset collection method 200 according to an embodiment of the present invention. As Figure 2 shown, the method 200 starts at step S210, where enterprise text data is collected and sorted, and the enterprise text data includes text materials, chat records, and custom task data.

[0030] First, various structured and unstructured text data inside the enterprise are collected, including document materials, chat records, custom task data, frequently asked questions, etc. These text data may cover multiple aspects such as company business, product knowledge, process regulations, etc.

[0031] The document materials may be table data, text data, etc. in a database. For educational enterprises or industries, the custom task data may be divergent tasks in teaching such as summarization, exemplification, and the like.

[0032] In one embodiment of the present invention, the text may be split according to title content, and each title content is stored in a modular manner. For example, a word document has title contents with title numbers of One, Two, Three, (1), (2), (3), 1, 2, 3, and each title content may be made into a module.

[0033] The text content from different sources is integrated into the modular title content to obtain modular text data. This can reduce duplication, so as to elaborate the related subject or topic more clearly, and form a more complete and consistent representation.

[0034] Natural language processing technology and entity recognition algorithm are used to perform entity recognition on the modular text data, so as to identify entities involved in the text, such as persons, places, time, etc. Relationships between entities are extracted from the text, such as the relationship between employees and departments, the relationship between projects and personnel, etc.

[0035] The extracted entities and entity relationships are represented as information points, and knowledge graph tools are used to model and store these information points in the form of a graph. Relationships are established between entities and relationships within the knowledge graph, connecting related information points by sharing the same attributes, relationships, or contextual information.

[0036] Then, step S220 is executed to label the chat logs and custom task data to obtain the seed dataset.

[0037] Chat log and custom task data annotation is the process of converting dialogues into structured data, which helps large language models understand the intent and information within the conversation. Annotators with relevant domain knowledge or experience can be selected to annotate the dialogues and custom task data according to pre-defined annotation specifications and guidelines, including question-and-answer pairing, question-and-context pairing, etc.

[0038] For example, questions and answers in chat logs can be labeled according to criteria such as question type and answer accuracy. The labeling results are then evaluated and reviewed for quality, and feedback is provided and corrections made for any errors or inconsistencies to obtain a high-quality seed dataset.

[0039] Finally, step S230 is executed, where the seed dataset is input into the existing large language model to generate a question-and-answer dataset, and the generated question-and-answer dataset is evaluated and iteratively optimized until the amount of data in the generated dataset reaches the preset amount of data.

[0040] Large language models can generate similar dialogue texts or task-related data based on samples in the seed dataset. For smaller seed datasets, data augmentation techniques such as template replacement and synonym replacement can be used to increase the amount and diversity of data.

[0041] Then, the existing large language model is fine-tuned using data from the enterprise domain within the knowledge graph to obtain a trained, customized model.

[0042] For example, a suitable basic large language model for the enterprise domain can be selected as the initial model, such as BERT or GPT-2. The initial model is then fine-tuned using enterprise domain data from the knowledge graph; the fine-tuned model is evaluated, and based on the evaluation results, it is further optimized and adjusted until the expected performance indicators are achieved, making the model adaptable to the specific tasks and needs of the enterprise domain.

[0043] Then, the seed dataset is input into the trained model for prediction, automatically generating a dialogue dataset. The question portion of the generated dialogue dataset is extracted as a new task dataset, and then a trained large language model is used to generate new questions and corresponding answers. The generated questions and answers are combined to form a new question-and-answer dataset.

[0044] Because model predictions may contain incorrect or unreasonable answers, the predictions need to be filtered and selected to choose the most accurate one. This filtering and selection can be done using heuristic algorithms or manual review.

[0045] For example, annotators can score and correct the generated question-and-answer dataset based on metrics such as data quality, accuracy, and usability. Based on the annotators' feedback and scores, the corrected data can be re-input into the large language model to regenerate the dataset. The dataset generation process ends when the generated dataset reaches the required volume.

[0046] Figure 3 A schematic diagram of the structure of a dataset acquisition device 300 for an enterprise intelligent question-answering system according to an embodiment of the present invention is shown. Figure 3 As shown, the device 300 may include a collection module 310, a labeling module 320, and an iterative generation module 330.

[0047] The collection module 310 can collect and organize enterprise text data, including text documents, chat logs, and custom task data. Knowledge graphs can be directly constructed from the text documents.

[0048] Custom task data such as chat logs and summary examples require further processing and question-and-answer annotation. The annotation module 320 can annotate chat logs and custom task data to obtain a seed dataset.

[0049] The iterative generation module 330 can input the seed dataset into an existing large language model to generate a question-and-answer dataset, and evaluate and iteratively optimize the generated question-and-answer dataset until the amount of data in the generated dataset reaches the preset amount of data.

[0050] During the iterative generation process, low-quality samples can be eliminated based on the annotators' scores and feedback. After each iteration, it is determined whether the size and quality of the current dataset have met the preset data quantity or quality requirements. If not, seed datasets can be generated iteratively to generate more data.

[0051] According to the enterprise intelligent question-answering system dataset collection method and apparatus of the present invention, data augmentation is performed on partially labeled question-answer data using existing large language models to improve the diversity and generalization ability of the dataset. In addition, the question-answer dataset generated iteratively based on semi-supervised learning can greatly reduce the error and cost of manual annotation.

[0052] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0053] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention and form different embodiments.

[0054] Furthermore, some of the embodiments described herein are methods or combinations of method elements that can be implemented by a processor of a computer system or by other means of performing the functions. Therefore, a processor having the necessary instructions for implementing the methods or method elements forms means for implementing the methods or method elements. Furthermore, the elements described herein in the apparatus embodiments are examples of means for implementing the functions performed by elements for the purposes of carrying out the invention.

[0055] As used herein, unless otherwise specified, the use of ordinal numbers such as “first,” “second,” “third,” etc., to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, ordering, or any other manner.

[0056] Although the invention has been described with reference to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. The language used in this specification has been chosen primarily for readability and illustrative purposes, and not for the purpose of interpreting or limiting the subject matter of the invention.

Claims

1. A method for collecting datasets for an enterprise intelligent question-answering system, characterized in that, include: Collect and organize enterprise text data, including text documents, chat logs, and custom task data; The text is split according to the title content, and each title content is stored modularly; text content from different sources is integrated into the modular title content to obtain modular text data; entity recognition is performed on the modular text data, and the relationships between entities are extracted; the entities and entity relationships are represented in the form of information points, and connections between information points are established in a knowledge graph; The seed dataset is obtained by labeling the chat logs and custom task data; Choose a basic large language model suitable for the enterprise domain as the initial model; The initial model was fine-tuned using enterprise domain data from the knowledge graph; The fine-tuned model is evaluated, and based on the evaluation results, it is further optimized and adjusted until the expected performance indicators are achieved. The seed dataset is input into the trained model for prediction to generate a dialogue dataset; the questions in the dialogue dataset are used as input to generate new questions and corresponding answers through a large language model; the generated questions and answers are combined to form a new question-and-answer dataset. The generated question-and-answer dataset is scored and corrected based on data quality, accuracy, and usability metrics. Based on the annotators' feedback and scores, the corrected data is re-input into the large language model to regenerate the dataset; The iterative generation process ends when the generated dataset reaches the required amount of data. Meanwhile, during the iteration process, after each iteration, it is determined whether the quality of the current dataset meets the preset quality requirements. If not, a seed dataset is generated to iterate and generate more data.

2. The data set collection method for the enterprise intelligent question-answering system according to claim 1, characterized in that, The step of annotating the chat logs and custom task data to obtain the seed dataset includes: The dialogue and custom task data are labeled according to the preset labeling specifications and guidelines; The quality of the annotation results is assessed and reviewed. Feedback is provided on any annotation errors or inconsistencies, and these are corrected to obtain the seed dataset.

3. A dataset acquisition device for an enterprise intelligent question-and-answer system, characterized in that, include: Collection module, annotation module, iterative generation module, The collection module is used to collect and organize enterprise text data, which includes text documents, chat logs, and custom task data. The annotation module is used to annotate the chat logs and custom task data to obtain a seed dataset; The iterative generation module is used to select a suitable basic large language model for the enterprise domain as the initial model; and to fine-tune the initial model using enterprise domain data in the knowledge graph. The fine-tuned model is evaluated, and the model is further optimized and adjusted based on the evaluation results until the expected performance indicators are achieved. The seed dataset is input into the trained model for prediction to generate a dialogue dataset. The questions in the dialogue dataset are used as input to generate new questions and corresponding answers through a large language model. The generated questions and answers are combined to form a new question-and-answer dataset. The generated question-and-answer dataset is scored and corrected based on data quality, accuracy, and usability metrics; the corrected data is re-input into the large language model based on the annotators' feedback and scores to regenerate the dataset; the iterative dataset generation process ends when the generated dataset reaches the required amount of data. Meanwhile, during the iteration process, after each iteration, it is determined whether the quality of the current dataset meets the preset quality requirements. If not, the seed dataset is generated and iteratively generated to produce more data. The device is also used to split the text according to the title content, and store each title content modularly; integrate text content from different sources in the modular title content to obtain modular text data; perform entity recognition on the modular text data and extract the relationship between entities; represent the entities and entity relationships in the form of information points, and establish connections between information points in a knowledge graph.

4. A computing device, comprising: At least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions including instructions for performing the enterprise intelligent question answering system dataset acquisition method as claimed in any one of claims 1-2.

5. A readable storage medium storing program instructions, which, when read and executed by a computing device, cause the computing device to perform the enterprise intelligent question-answering system dataset acquisition method as described in any one of claims 1-2.

Images