Construction method and system for generating multi-source knowledge base based on LLM, medium and product

Abstract

The invention discloses a construction method and system for generating a multi-source knowledge base based on LLM, a medium and a product, and relates to the technical field of artificial intelligence. Preprocessing the multi-source data according to the data type to obtain preprocessed data; extracting knowledge information in the preprocessed data by using a large language model to obtain question answer pair information; converting the question and answer pair information into a standard format to obtain question and answer group information; after the question and answer group information is recognized, deduplicated and merged, standard question and answer group information is obtained; and storing the answer group information of the standard questions into a knowledge base of the intelligent customer service system to complete the construction of the multi-source knowledge base of the intelligent customer service system. According to the method, the answer group information of the standard questions can be extracted from various data sources in time by utilizing the large language model, and the answer group information is stored in the knowledge base of the intelligent customer service system, so that the service quality is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a method and system for constructing a multi-source knowledge base based on an LLM (Large Language Model), a medium and a product. BACKGROUND

[0002] With the increasing demand for enterprise user services, intelligent customer service systems have become an important tool for enterprises to improve service efficiency and user satisfaction. As the core component of an intelligent customer service system, the quality and completeness of a knowledge base directly affect the accuracy of the intelligent customer service system's response and the user experience. Traditional knowledge base construction methods mainly rely on manual sorting and input, which has the problems of low efficiency, high cost, and untimely updates.

[0003] Although there are some automatic knowledge information extraction methods in the prior art, they are usually limited to a single data source and lack a comprehensive solution for automatically extracting knowledge information from multiple types of data sources and generating standardized question and answer pairs. SUMMARY

[0004] The purpose of the present application is to provide a method and system for constructing a multi-source knowledge base based on an LLM (Large Language Model), a medium and a product, to solve the technical problem that existing knowledge base construction methods are usually limited to a single data source and lack a comprehensive solution for automatically extracting knowledge information from multiple types of data sources and generating standardized question and answer pairs. The present application can construct a high-quality intelligent customer service system knowledge base and improve the efficiency and quality of knowledge base construction.

[0005] To achieve the above-mentioned purpose, the present application provides the following solutions: In a first aspect, the present application provides a method for constructing a multi-source knowledge base based on an LLM, comprising: obtaining multi-source data; preprocessing the multi-source data according to the data type to obtain preprocessed data; extracting knowledge information from the preprocessed data using a large language model to obtain question and answer pair information; converting the question and answer pair information into a standard format to obtain question and answer group information; after identifying, deduplicating and merging the question and answer group information, obtaining standard question and answer group information; storing the standard question and answer group information into the knowledge base of an intelligent customer service system to complete the construction of the multi-source knowledge base of the intelligent customer service system.

[0006] Optionally, it further comprises a multi-source knowledge base editing step: The visual editing interface is provided to enable an administrator of the intelligent customer service system to manually check, modify and supplement questions and corresponding answers in the standard question and answer group information.

[0007] Optionally, the method further comprises a multi-source knowledge base updating step of periodically reacquiring the multi-source data, detecting whether the corresponding data content in the original knowledge base of the intelligent customer service system has changed, updating the data content in the corresponding knowledge base if there is a change, and not updating otherwise.

[0008] Optionally, the multi-source data specifically includes web page content, user chat records, work order processing records and files, and the files include Word documents, Excel documents and PDF documents. The preprocessing specifically includes cleaning and formatting the web page content, user chat records, work order processing records and files. The method specifically comprises the following steps: The method specifically comprises the following steps: The method specifically comprises the following steps: The method specifically comprises the following steps:

[0009] Optionally, the method specifically comprises the following steps: The method specifically comprises the following steps: The method specifically comprises the following steps: The method specifically comprises the following steps:

[0010] Optionally, the method specifically comprises the following steps: A similarity threshold is set in the large language model. The different types of question-answer group information are identified by using the large language model, and semantic similarities of the different types of question-answer group information are obtained. The relationship between the semantic similarity and the similarity threshold is determined. If the semantic similarity is less than or equal to the similarity threshold, the different types of question-answer group information are combined and processed. If the semantic similarity is greater than the similarity threshold, the different types of question-answer group information are de-duplicated.

[0011] In a second aspect, the present application provides a construction system for generating a multi-source knowledge base based on an LLM, comprising: A data acquisition module is configured to acquire multi-source data sources. A data preprocessing module is configured to preprocess the multi-source data according to data types to obtain preprocessed data. An LLM processing module is configured to extract knowledge information in the preprocessed data by using a large language model to obtain question-answer pair information, convert the question-answer pair information into a standard format to obtain question-answer group information, identify, de-duplicate, and combine the question-answer group information to obtain standard question-answer group information, and store the standard question-answer group information in a knowledge base of an intelligent customer service system.

[0012] Optionally, the construction system further comprises: A knowledge base editing module is configured to provide a visual editing interface, so that an administrator of the intelligent customer service system manually checks, modifies, and supplements questions and corresponding answers in the standard question-answer group information. A knowledge base updating module is configured to periodically reacquire multi-source data, detect whether corresponding data contents in an original knowledge base of the intelligent customer service system have changed, update data contents in the corresponding knowledge base if the data contents have changed, and do not update if the data contents have not changed.

[0013] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the construction method for generating a multi-source knowledge base based on an LLM according to any one of the above embodiments.

[0014] In a fourth aspect, the present application provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the construction method for generating a multi-source knowledge base based on an LLM according to any one of the above embodiments.

[0015] According to the specific embodiments provided by the present application, the following technical effects are disclosed: This application provides a method, system, medium, and product for constructing a multi-source knowledge base based on LLM (Low-Low Metrics and Modeling). The method, through a multi-source data acquisition step, fully utilizes existing data sources of various types, resulting in diverse and comprehensive data in the intelligent customer service system's knowledge base. A preprocessing step for multi-source data prepares the knowledge base for construction, significantly improving construction efficiency. A large language model extracts knowledge information from the preprocessed data, obtaining question-answer pairs, and converts these pairs into a standard format, making the knowledge information in the knowledge base more accurate and standardized. After identifying, deduplicating, and merging question-answer groups, standard question-answer groups are obtained, avoiding redundancy and improving service quality. Storing these standard question-answer groups in the intelligent customer service system's knowledge base facilitates its management and application. This application's method, by automatically extracting multi-source data using a large language model, constructs a more comprehensive and richer intelligent customer service system knowledge base, solving the technical problem of existing knowledge bases having a single data source. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a method for constructing a multi-source knowledge base based on LLM in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The purpose of this invention is to provide a method, system, medium, and product for constructing a multi-source knowledge base based on LLM. To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] like Figure 1 As shown, this invention provides a method for constructing a multi-source knowledge base based on LLM, including: Step 1: Obtain multi-source data; In this embodiment, the acquired multi-source data includes web page content, user chat logs, work order processing records, and files; files include Word documents, Excel documents, and PDF documents; web page content, such as a user's official website or introduction page, is obtained through web crawling technology; user chat logs are obtained from historical customer service chat logs in the customer service system, yielding questions and replies related to the user's own business; work order processing records contain user questions and solutions, with the solutions being the customer service replies in the work order processing records; data can also be obtained from user brochures, scripts, product descriptions, and other files. Step 1 supports extracting knowledge information from various data sources such as web page content, chat logs, and work order records, making full use of the existing data resources of various enterprises. This results in a broad coverage and diverse data in the knowledge base of the intelligent customer service system, comprehensively utilizing multiple data sources to build a more comprehensive and richer knowledge base.

[0021] Step 2: Preprocess the multi-source data according to the data type to obtain preprocessed data; In this embodiment, preprocessing involves cleaning and formatting multi-source data according to its data type. For example, web page content needs to have its formatting removed, retaining only the text content; user chat records need to have user information, customer service information, time information, etc. removed, retaining only the chat content; work order processing records need to have user information, customer service information, time information, etc. removed, retaining only the questions and answers in the work order processing records; files only need to have their content parsed. The preprocessed data obtained in step 2 can reduce manual processing costs, ensure that knowledge information is standardized and usable, thereby accelerating the construction efficiency of the knowledge base of the intelligent customer service system.

[0022] Step 3: Use a large language model to extract knowledge information from the preprocessed data to obtain question-answer pairs; Large Language Models (LLMs) are deep learning models trained on massive amounts of text data. These models can generate natural language text or understand the meaning of language text, and can also provide in-depth knowledge and language production on various topics through training on large datasets. The core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, simulating human language cognition and generation processes to a certain extent. LLMs perform well in various application scenarios, not only performing simple language tasks such as spell checking and grammar correction, but also handling complex tasks such as text summarization, machine translation, sentiment analysis, dialogue generation, and content recommendation. Pre-training on large datasets gives LLMs powerful general modeling and generalization capabilities. Therefore, in step 3 of this application, the large language model automatically analyzes various knowledge information in the preprocessed data to automatically generate question-answer pairs, greatly improving the efficiency of knowledge base construction in the intelligent customer service system.

[0023] In this embodiment, a large language model is used to extract web page content from the preprocessed data, and the content that has been displayed in the form of questions and answers and the content that has not been directly displayed in the form of questions and answers but can be converted into the form of questions and answers are used as question-answer pair information; for example, there is no obvious question on the product introduction page, but it is necessary to extract the product model, price, use, etc. When a user asks about a product model, the corresponding question can be answered according to the content of the knowledge base.

[0024] The system utilizes a large language model to extract user chat logs from preprocessed data, extracts effective question-and-answer interactions based on keywords, and uses these interactions as question-and-answer pairs. Keywords are the category labels input to the large language model; effective question-and-answer interactions are the top 50 terms or terms appearing 100 or more times in the replies to questions related to user business and showing clear termination or topic shifts. For example, in after-sales scenarios, users frequently ask "How long is the expected delivery time for the purchased product?" This question appears more than 100 times, thus forming a fixed response script. After analyzing user chat logs / work order processing records, corresponding question-and-answer pairs can be generated.

[0025] Step 4: Convert the question-answer pair information into a standard format to obtain the question-answer group information; The large language model is used to rewrite and optimize the question-answer pairs. Specifically, question rewriting involves using the large language model to rewrite colloquial and incomplete questions in the question-answer pairs into standard question formats. Answer optimization involves using the large language model to organize lengthy and scattered answers in the question-answer pairs into answer formats. In step 4, the powerful semantic understanding capabilities of the large language model are utilized to extract and standardize question-answer pairs from various knowledge information more accurately, further improving the quality of knowledge information in the knowledge base of the intelligent customer service system.

[0026] In this embodiment, the question-answer pair information is converted into a standard format to obtain question-answer group information. For example, questions and answers that appear multiple times in the same work order processing record are as follows: User: I bought this dress, how long will it take to arrive? Customer service: Please wait a moment while I check the address you entered.

[0027] Customer service: I see you're from Beijing, it will arrive in about 3 days.

[0028] User: What kind of courier service is this? It's so slow. Customer service: ZTO Express.

[0029] User: Let's switch to SF Express.

[0030] Customer service: Sorry, we default to using ZTO Express.

[0031] After rewriting the question and optimizing the answer, it becomes: Question: Which courier service do you use? How long does it take to arrive? Answer: In Beijing, we use ZTO Express by default, which takes about 3 days to arrive. We do not support changing the courier company.

[0032] Step 5: After identifying, deduplicating, and merging the question-answer group information, the standard question-answer group information is obtained, specifically: Set a similarity threshold in the large language model; Large language models are used to identify different types of question-answer group information and obtain the semantic similarity of different types of question-answer group information. Determine the relationship between semantic similarity and similarity threshold: If the semantic similarity is less than or equal to the similarity threshold, then the information of question answer groups of different types will be merged and processed. If the semantic similarity is greater than the similarity threshold, then the answer group information of different types of questions will be deduplicated. For example, for the same logistics issue, different work order records show users using different logistics companies and with different location information. If multiple questions appear from the Beijing area, deduplication will occur because the semantic similarity of the questions exceeds the similarity threshold of 80%. If questions from different areas appear, they may be merged because they are all logistics information questions.

[0033] After identification, deduplication, and merging, the result is as follows: Question: What logistics company will be used, and how long will it take to arrive? Answer: For areas like Jiangsu, Zhejiang, and Shanghai, use SF Express for next-day delivery; for areas like Xinjiang and Tibet, use EMS for approximately 7-14 days; for other areas, use ZTO Express for about 3 days.

[0034] Therefore, by identifying, deduplicating, and merging the question answer group information in step 5, redundancy in the knowledge base of the intelligent customer service system is avoided.

[0035] Step 6: Store the standard question answer group information into the knowledge base of the intelligent customer service system to complete the construction of the multi-source knowledge base of the intelligent customer service system.

[0036] By implementing the above-described method for constructing a multi-source knowledge base based on LLM, the knowledge base of the intelligent customer service system obtained, compared with the data sources of existing knowledge bases, can uniformly standardize and process question-answer pairs from different sources to facilitate knowledge management and application. It can also extract new standard question-answer pairs from various data sources in a timely manner to automatically update the knowledge base, thereby providing more accurate and comprehensive standard question-answer pairs to reduce the maintenance cost of the knowledge base. As a result, the knowledge base of the intelligent customer service system in this invention has a higher degree of standardization, lower maintenance costs, and better service performance.

[0037] This application also provides an application scenario in which the above-described method for constructing a multi-source knowledge base based on LLM is applied. Specifically: The knowledge base has already formed a set of standard questions and answers regarding the above logistics information. At this point, a new user asks: "How long will it take to arrive?" After the large language model collects the questions, it analyzes the keywords as "how long will it take to arrive" and identifies the user's intent as "logistics information". When it searches the knowledge base, it finds that there is already a set of standard questions and answers related to logistics. It then directly replies with the corresponding answers: "SF Express is used in areas such as Jiangsu, Zhejiang and Shanghai, with next-day delivery; EMS is used in areas such as Xinjiang and Tibet, with delivery in about 7-14 days; ZTO Express is used in other areas, with delivery in about 3 days."

[0038] In this embodiment, the standard question-answer group information is stored in the knowledge base of the intelligent customer service system. After the construction of the multi-source knowledge base of the intelligent customer service system is completed, the multi-source knowledge base editing step is also included: a visual editing interface is provided so that the administrator of the intelligent customer service system can manually verify, modify and supplement the questions and their corresponding answers in the standard question-answer group information.

[0039] After editing the multi-source knowledge base, the process also includes updating the multi-source knowledge base: periodically re-acquiring multi-source data and checking whether the corresponding data content in the original intelligent customer service system's knowledge base has changed; if so, updating the corresponding data content in the knowledge base; otherwise, not updating.

[0040] The multi-source knowledge base editing and updating steps can extract new standard question answer group information from various data sources in a timely manner and update it automatically on a regular basis, reducing the maintenance cost of the knowledge base.

[0041] In one exemplary embodiment, a system for building a multi-source knowledge base based on LLM is provided, comprising: The data acquisition module is used to acquire data from multiple sources; The data preprocessing module is used to preprocess multi-source data according to data type to obtain preprocessed data; The LLM processing module is used to extract knowledge information from preprocessed data using a large language model to obtain question-answer pairs, convert the question-answer pairs into a standard format to obtain question-answer groups, identify, deduplicate, and merge the question-answer groups to obtain standard question-answer groups, and then store the standard question-answer groups in the knowledge base of the intelligent customer service system.

[0042] As an optional implementation, the construction system for generating a multi-source knowledge base based on LLM also includes the following in the LLM processing module: a knowledge base editing module, which provides a visual editing interface, allowing the administrator of the intelligent customer service system to manually verify, modify, and supplement the questions and their corresponding answers in the standard question-answer group information; and a knowledge base updating module, which periodically re-acquires multi-source data and checks whether the corresponding data content in the original intelligent customer service system's knowledge base has changed: if so, the corresponding data content in the knowledge base is updated; otherwise, it is not updated.

[0043] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0044] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0045] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0046] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0047] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0048] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0049] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for constructing a multi-source knowledge base based on LLM, characterized in that, include: Acquire multi-source data; Preprocess multi-source data according to data type to obtain preprocessed data; Use large language models to extract knowledge information from preprocessed data to obtain question-answer pairs; Convert the question-answer pairs into a standard format to obtain the question-answer group information; After identifying, deduplicating, and merging the question-answer group information, the standard question-answer group information is obtained; The standard question answer set information is stored in the knowledge base of the intelligent customer service system, thus completing the construction of the multi-source knowledge base of the intelligent customer service system.

2. The method for constructing a multi-source knowledge base based on LLM according to claim 1, characterized in that, It also includes multi-source knowledge base editing steps: It provides a visual editing interface, allowing administrators of the intelligent customer service system to manually verify, modify, and supplement the questions and their corresponding answers in the standard question and answer group information.

3. The method for constructing a multi-source knowledge base based on LLM according to claim 1, characterized in that, It also includes the steps for updating the multi-source knowledge base: Periodically reacquire multi-source data and check whether the corresponding data content in the knowledge base of the original intelligent customer service system has changed: if so, update the corresponding data content in the knowledge base; otherwise, do not update.

4. The method for constructing a multi-source knowledge base based on LLM according to claim 1, characterized in that, The multi-source data specifically includes: web page content, user chat logs, work order processing records, and files; the files include Word documents, Excel documents, and PDF documents. The preprocessing specifically includes cleaning and formatting the webpage content, user chat history, work order processing records, and files respectively; The step of extracting knowledge information from preprocessed data using a large language model to obtain question-answer pairs specifically includes: The large language model is used to extract web page content from the preprocessed data, and the content that has been displayed in the form of questions and answers and the content that has not been directly displayed in the form of questions and answers but can be converted into the form of questions and answers are used as question-answer pairs; The system utilizes a large language model to extract user chat logs from preprocessed data, extracts effective question-and-answer interactions based on keywords, and uses these effective interactions as question-answer pairs. The keywords are classification tags input to the large language model. The effective question-and-answer interactions are the top 50 words or words that appear more than or equal to 100 times in the questions related to user business and which have ended or changed topics. By using a large language model to extract work order processing records from preprocessed data, the top 50 terms or terms that appear 100 or more times in the user's description of the problem and the customer service's reply within a certain period are used as the question-answer pair information.

5. The method for constructing a multi-source knowledge base based on LLM according to claim 4, characterized in that, The process of converting question-answer pairs into a standard format specifically includes: Using a large language model, question-answer pairs are rewritten and answers are optimized. Specifically, the question rewriting process involves using a large language model to rewrite colloquial and incomplete questions in question-answer pairs into standard question formats. The answer optimization involves using a large language model to organize the lengthy and scattered answers in the question-answer pair information into an answer format.

6. The method for constructing a multi-source knowledge base based on LLM according to claim 5, characterized in that, The identification, deduplication, and merging of question answer group information specifically includes: Set a similarity threshold in the large language model; Large language models are used to identify different types of question-answer group information and obtain the semantic similarity of different types of question-answer group information. Determine the relationship between semantic similarity and similarity threshold: If the semantic similarity is less than or equal to the similarity threshold, then the information of question answer groups of different types will be merged and processed. If the semantic similarity is greater than the similarity threshold, then the answer group information of different types of questions will be deduplicated.

7. A system for constructing a multi-source knowledge base based on LLM, characterized in that, include: The data acquisition module is used to acquire data from multiple sources; The data preprocessing module is used to preprocess multi-source data according to data type to obtain preprocessed data; The LLM processing module is used to extract knowledge information from preprocessed data using a large language model to obtain question-answer pairs, convert the question-answer pairs into a standard format to obtain question-answer groups, identify, deduplicate, and merge the question-answer groups to obtain standard question-answer groups, and then store the standard question-answer groups in the knowledge base of the intelligent customer service system.

8. The construction system for generating multi-source knowledge bases based on LLM according to claim 7, characterized in that, Also includes: The knowledge base editing module provides a visual editing interface, allowing administrators of the intelligent customer service system to manually verify, modify, and supplement the questions and their corresponding answers in the standard question and answer group information. The knowledge base update module is used to periodically reacquire multi-source data and check whether the corresponding data content in the knowledge base of the original intelligent customer service system has changed. If so, the corresponding data content in the knowledge base is updated; otherwise, it is not updated.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the construction method for generating a multi-source knowledge base based on LLM as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the construction method for generating a multi-source knowledge base based on LLM as described in any one of claims 1-6.

Images