Question and answer knowledge base updating method and device, equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of dialogue questions and answers, is applied to a questions and answers knowledge processing scene, and relates to a questions and answers knowledge base updating method, device and equipment and a storage medium thereof.The method comprises the steps that questions and answers basic data sent by a multi-data source end are obtained and input into a data comprehensive processing component to be processed, and questions and answers promotion data are obtained; on the basis of question and answer promotion data, a question and answer knowledge generation component based on PingAnGPT is adopted to generate question and answer pairs, then the question and answer pairs are cached to a target question and answer knowledge base, updating of the question and answer knowledge base is completed, it is avoided that the question and answer knowledge generation component is directly adopted to sort question and answer basic data of a document class, an image class and a structural form class, and the question and answer knowledge generation efficiency is improved. The question and answer knowledge sorting and using efficiency is improved, different knowledge types such as forms, pictures and documents are processed in combination with the data comprehensive processing assembly, compatible processing of multi-carrier type data is achieved, and the multi-carrier type data source end can be effectively utilized for large-scale question and answer knowledge updating.

Description

Technical Field

[0001] The present application relates to the technical field of dialogue question and answer, and is applied to question and answer knowledge processing scenarios, and in particular to a question and answer knowledge base updating method, device, equipment and storage medium thereof. Background Art

[0002] In the traditional operation process, underwriters need to frequently communicate product information with sales staff at the stages of business development, quotation inquiry, insurance transfer, and policy issuance. The types of questions and answers include but are not limited to product highlights, product consultation, quotation inquiry materials, product comparison, etc. The current status of business Q&A is that the average number of online consultations per underwriter is 21.3 per day, and the first response to the question takes 10 hours. The workload is extremely heavy and the efficiency is very low, resulting in insufficient front-line support and fatigue in responding to questions from the back line.

[0003] At present, the question-answering robots used in the industry are often used in two ways: one is through FAQ (question-answer pair matching), and the other is through RAG large models. However, both methods rely on a large number of highly professional and high-quality QA question-answer pairs to provide accurate question-answer responses. At present, the method used by group (group insurance) and property (property insurance) is to find experts in the business line to manually generate QA knowledge and feed it back to the robot knowledge base for knowledge update. However, this method often relies on manual experience in knowledge update, and human knowledge is limited. It is impossible to be proficient in various question-answering fields, resulting in limitations in knowledge update, and it is impossible to effectively use the existing data source to update large-scale question-answering knowledge. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a question-and-answer knowledge base updating method, apparatus, device and storage medium thereof, so as to solve the problem that the existing technology for question-and-answer knowledge updating causes limitations in knowledge updating and cannot effectively utilize the existing data source for large-scale question-and-answer knowledge updating.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for updating a question-and-answer knowledge base, which adopts the following technical solution:

[0006] A method for updating a question-answer knowledge base comprises the following steps:

[0007] Obtain the basic question and answer data sent by multiple data sources;

[0008] Inputting the question-answer basic data into a preset data comprehensive processing component;

[0009] Processing the question and answer basic data through the data comprehensive processing component to obtain question and answer promotion data;

[0010] Inputting the question and answer promotion data into a preset question and answer knowledge generation component;

[0011] The question-answer knowledge generation component generates question-answer pairs for the question-answer promotion data, caches the generated question-answer pairs in a target question-answer knowledge base, and completes the update of the target question-answer knowledge base.

[0012] Furthermore, the preset data comprehensive processing component includes a basic data type identification subcomponent. After executing the step of inputting the question and answer basic data into the preset data comprehensive processing component, the method further includes:

[0013] Performing carrier type identification on all basic data in the question and answer basic data through the basic data type identification subcomponent, wherein the carrier refers to a sending carrier of the question and answer basic data, and the carrier type includes a document, an image, and a structured form;

[0014] According to the carrier type identification result, all basic data in the question and answer basic data are classified and sorted to obtain the classified and sorted document data set, image data set and structured data set.

[0015] Furthermore, the data comprehensive processing component includes a document processing subcomponent, an image processing subcomponent and a structured form processing subcomponent. The step of processing the question and answer basic data by the data comprehensive processing component to obtain the question and answer promotion data specifically includes:

[0016] Allocating the document-type data set to the document processing subcomponent to obtain the question-answering promotion data processed by the document processing subcomponent;

[0017] Allocating the image data set to the image processing subcomponent to obtain the question-answering promotion data processed by the image processing subcomponent;

[0018] The structured data set is distributed to the structured form processing subcomponent to obtain the question and answer promotion data processed by the structured form processing subcomponent.

[0019] Furthermore, the document processing subcomponent includes a natural language processing component based on spaCy, and the step of allocating the document class data set to the document processing subcomponent to obtain the question and answer promotion data processed by the document processing subcomponent specifically includes:

[0020] Performing paragraph layout recognition on all documents in the document dataset by using the spaCy-based natural language processing component to identify the paragraph layout of all documents;

[0021] Performing document segmentation processing on all documents in the document dataset according to the paragraph layout of all documents to obtain document slices after segmentation processing;

[0022] The grouped question slices and answer slices in the document slices are distinguished and marked, and all answer slices corresponding to the same question are clustered and sorted, and the clustering results are obtained as the question and answer promotion data after being processed by the document processing subcomponent.

[0023] Furthermore, the image processing subcomponent includes a multimodal data processing component based on GPT-4, and the step of allocating the image data set to the image processing subcomponent to obtain the question-answering promotion data processed by the image processing subcomponent specifically includes:

[0024] Performing question-answering knowledge extraction on the image data and text data in the image data set through the GPT-4-based multimodal data processing component to obtain a question-answering knowledge extraction result;

[0025] The question-answer knowledge extraction results are distinguished and marked according to the difference in images, and all the answer data corresponding to the same question are clustered and sorted, and the clustering results are obtained as the question-answer promotion data processed by the image processing subcomponent.

[0026] Furthermore, the structured form processing subcomponent includes a hierarchical data processing component based on LLMs, and the step of allocating the structured data set to the structured form processing subcomponent to obtain the question and answer promotion data processed by the structured form processing subcomponent specifically includes:

[0027] Identifying the hierarchical structure of all structure tables in the structured data set by the LLMs-based hierarchical data processing component;

[0028] According to the hierarchical structure of all the structural forms, all the structural forms in the structured data set are parsed layer by layer to obtain hierarchical parsed data corresponding to all the structural forms;

[0029] The hierarchical parsing data corresponding to the same question and answer fields in different structure forms are clustered and merged, and the clustering and merging results are obtained as the question and answer promotion data processed by the structure form processing subcomponent.

[0030] Further, the question-answer knowledge generation component includes a question-answer knowledge generation component based on PingAnGPT, and the step of generating question-answer pairs for the question-answer promotion data by the question-answer knowledge generation component and caching the generated question-answer pairs into the target question-answer knowledge base specifically includes:

[0031] Performing semantic analysis on all question fields contained in the question and answer promotion data by using the PingAnGPT-based question and answer knowledge generation component;

[0032] Using a preset clustering algorithm to cluster question fields with the same semantic analysis results into the same question field, and obtaining a question field clustering result, wherein the preset clustering algorithm includes a K-means clustering algorithm or a K nearest neighbor clustering algorithm;

[0033] According to the question field clustering result, the answer data corresponding to all question fields in the same clustering cluster are clustered together to generate answer data sets corresponding to different clustering clusters;

[0034] By extracting keywords from the answer data sets corresponding to different clusters, the answer keywords corresponding to different answer data sets are obtained;

[0035] The same question field and answer keyword corresponding to the same cluster are sorted to generate question-answer pairs, and the generated question-answer pairs are cached in the target question-answer knowledge base, wherein the same question field and the answer keyword in the question-answer pair are in a one-to-one relationship or a one-to-many relationship.

[0036] In order to solve the above technical problems, the embodiment of the present application also provides a question-answer knowledge base updating device, which adopts the following technical solution:

[0037] A question-and-answer knowledge base updating device, comprising:

[0038] A question-and-answer basic data acquisition module is used to acquire question-and-answer basic data sent by multiple data sources;

[0039] A question-and-answer basic data input module, used to input the question-and-answer basic data into a preset data comprehensive processing component;

[0040] A question and answer basic data processing module, used to process the question and answer basic data through the data comprehensive processing component to obtain question and answer promotion data;

[0041] A question and answer promotion data input module, used to input the question and answer promotion data into a preset question and answer knowledge generation component;

[0042] The question and answer knowledge base updating module is used to generate question and answer pairs for the question and answer promotion data through the question and answer knowledge generation component, cache the generated question and answer pairs to the target question and answer knowledge base, and complete the update of the target question and answer knowledge base.

[0043] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0044] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned question-and-answer knowledge base updating method when executing the computer-readable instructions.

[0045] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0046] A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the question-and-answer knowledge base updating method as described above.

[0047] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0048] The question-and-answer knowledge base updating method described in the embodiment of the present application obtains question-and-answer basic data sent by multiple data sources and inputs it into the data comprehensive processing component for processing to obtain question-and-answer promotion data; the question-and-answer promotion data is input into the question-and-answer knowledge generation component to generate question-and-answer pairs, and the generated question-and-answer pairs are cached into the target question-and-answer knowledge base to obtain the question-and-answer knowledge base. Based on the question-and-answer promotion data, the present application uses a question-and-answer knowledge generation component based on PingAnGPT to generate question-and-answer pairs, and then caches the question-and-answer pairs into the target question-and-answer knowledge base to complete the update of the question-and-answer knowledge base, avoiding the direct use of the question-and-answer knowledge generation component to organize the basic question-and-answer data of document classes, image classes, and structured form classes, thereby improving the efficiency of question-and-answer knowledge organization and use. In combination with the data comprehensive processing component, different knowledge types such as forms, images, and documents are processed separately, achieving compatible organization of multi-carrier type data, and being able to effectively utilize multi-carrier type data sources for large-scale question-and-answer knowledge updates. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0051] Figure 2 is a flow chart of an embodiment of a method for updating a question-answer knowledge base according to the present application;

[0052] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;

[0053] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 301 is shown;

[0054] Figure 5 yes Figure 3 A flowchart of a specific embodiment of step 302 is shown;

[0055] Figure 6 yes Figure 3 A flowchart of a specific embodiment of step 303 is shown;

[0056] Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown;

[0057] Figure 8 is a structural schematic diagram of an embodiment of a question-and-answer knowledge base updating device according to the present application;

[0058] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0060] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0061] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0062] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0063] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0064] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0065] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0066] It should be noted that the question and answer knowledge base updating method provided in the embodiment of the present application is generally executed by a server, and accordingly, the question and answer knowledge base updating device is generally set in the server.

[0067] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0068] Continue to refer Figure 2 , shows a flow chart of an embodiment of a question-answer knowledge base updating method according to the present application. The question-answer knowledge base updating method comprises the following steps:

[0069] Step 201, obtaining basic question and answer data sent by multiple data sources.

[0070] Specifically, the multi-data source end includes a document data source end, an image data source end, and a structured form data source end, wherein the document data source end contains pre-organized question and answer basic data in a document format, the image data source end contains pre-organized question and answer basic data in an image format, and the structured form data source end contains pre-organized question and answer basic data in a structured form format. Of course, the multi-data source end can also be a plurality of different data sending platforms, and the same point is that the data sent by the multi-data source end are all question and answer basic data, and the question and answer basic data can refer to question and answer data that has not been processed too much, for example: directly obtained question and answer dialogue data between customer service and customers. Especially in the field of customer service consulting applications, due to the diversity of business, a large amount of question and answer dialogue data between customer service and customers is often generated. By obtaining the question and answer basic data sent by the multi-data source end, it is convenient to perform secondary processing on the question and answer basic data, so as to organize the speech text to assist customer service and customers in business communication.

[0071] Step 202, input the question and answer basic data into a preset data comprehensive processing component.

[0072] In this embodiment, the preset data comprehensive processing component includes a basic data type identification subcomponent, a document processing subcomponent, an image processing subcomponent and a structured form processing subcomponent. By inputting the question and answer basic data into the preset data comprehensive processing component, it is possible to process question and answer basic data of different carrier types, thus avoiding the limitation of only being able to process question and answer basic data of a single carrier type.

[0073] In this embodiment, after executing the step of inputting the question and answer basic data into a preset data comprehensive processing component, the method further includes: performing carrier type identification on all basic data in the question and answer basic data through the basic data type identification subcomponent, wherein the carrier refers to the sending carrier of the question and answer basic data, and the carrier types include documents, images and structured forms; according to the carrier type identification result, all basic data in the question and answer basic data are classified and organized to obtain the classified and organized document data set, image data set and structured data set.

[0074] Specifically, the step of identifying the carrier type of all the basic data in the question and answer basic data through the basic data type identification subcomponent specifically includes: identifying the carrier format of all the basic data in the question and answer basic data by means of suffix string identification, and determining the carrier type of all the basic data in the question and answer basic data. For example: when multiple data sources send question and answer basic data, the corresponding carrier types include documents, images and structured forms. The corresponding document suffix strings are ".doc", ".docx", etc., while the image suffix strings are ".jpg", ".png", ".tif", ".gif", etc., and the structured form suffix strings are ".csv", ".xls", ".xlsx", etc. By identifying these suffix strings, the carrier format of all the basic data in the question and answer basic data is identified, and the carrier type of all the basic data in the question and answer basic data is determined, so as to facilitate the classification and sorting of question and answer basic data of different carrier types, and facilitate the subsequent use of different processing subcomponents to process the question and answer basic data.

[0075] Step 203, processing the question and answer basic data through the data comprehensive processing component to obtain question and answer promotion data.

[0076] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes:

[0077] Step 301, assigning the document data set to the document processing subcomponent, and obtaining the question and answer promotion data processed by the document processing subcomponent;

[0078] Specifically, the document processing subcomponent includes a natural language processing component based on spaCy, wherein spaCy is a natural language processing library that can obtain the entire text content from the target document through the .text attribute, and then the text content can be segmented into paragraphs through the split function in the spaCy library. In this embodiment, the corresponding question and answer basic data text content is obtained from the document class carrier of the question and answer basic data through the .text attribute, and then the corresponding question and answer basic data text content is segmented into paragraphs through the split function in the spaCy library.

[0079] Continue to refer Figure 4 , Figure 4 yes Figure 3 The flowchart of a specific embodiment of step 301 shown includes:

[0080] Step 401, performing paragraph layout recognition on all documents in the document dataset respectively through the spaCy-based natural language processing component to recognize the paragraph layout of all documents;

[0081] Step 402, performing document segmentation processing on all documents in the document dataset according to the paragraph layout of all documents, and obtaining document slices after segmentation processing;

[0082] Specifically, all the documents in the document data set are segmented according to the paragraph layout of all the documents to obtain the segmented document slices. In fact, all the documents in the document data set can be segmented in sections.

[0083] Step 403, distinguish and mark the grouped question slices and answer slices in the document slices, and cluster and sort all the answer slices corresponding to the same question, and obtain the clustering result as the question and answer promotion data processed by the document processing subcomponent.

[0084] In this embodiment, the question slices and answer slices grouped in the document slices are distinguished and marked, for example: the question slices and answer slices can be identified based on identification characters, the identification characters of the question slices include "question" or "Q" at the beginning of a paragraph, and "?" at the end of a paragraph, and the identification characters of the answer slices include "answer" or "A" at the beginning of a paragraph; all answer slices corresponding to the same question are clustered and sorted, and semantic recognition can be performed on all question slices using semantic analysis, and question slices with the same semantic recognition results are summarized as the same question.

[0085] By slicing the basic question and answer data in the document format into paragraphs, categorizing the questions, and clustering the answer slices of the same question, it is convenient to directly use the clustered question and answer promotion data in the subsequent question and answer pair generation and question and answer knowledge base generation.

[0086] Step 302, assigning the image data set to the image processing subcomponent to obtain the question-answering promotion data processed by the image processing subcomponent;

[0087] Specifically, the image processing subcomponent includes a multimodal data processing component based on GPT-4. GPT-4 is a multimodal perception technology provided by OpenAI, which has powerful image and text processing capabilities and can extract image information and text information from images. In this embodiment, the multimodal data processing component based on GPT-4 is applied to the extraction of basic question and answer data for image carrier types, which can obtain both the basic question and answer data expressed by the image and the basic question and answer data expressed by the text content in the image, thereby realizing the extraction of basic question and answer data in the image in a multimodal extraction manner.

[0088] Continue to refer Figure 5 , Figure 5 yes Figure 3 The flowchart of a specific embodiment of step 302 shown includes:

[0089] Step 501, extracting question-answering knowledge from the image data and text data in the image data set by the GPT-4-based multimodal data processing component to obtain a question-answering knowledge extraction result;

[0090] Step 502, distinguishing and marking the question and answer knowledge extraction results according to the difference of the images, and clustering and arranging all the answer data corresponding to the same question, and obtaining the clustering and arranging results as the question and answer promotion data processed by the image processing subcomponent.

[0091] By extracting the basic multimodal question and answer data in the image carrier, classifying and organizing the subsequent questions, and clustering and organizing all the answer data corresponding to the same question, it is convenient to directly use the clustered question and answer promotion data in the subsequent question and answer pair generation and question and answer knowledge base generation.

[0092] Step 303, assigning the structured data set to the structured form processing subcomponent to obtain the question and answer promotion data processed by the structured form processing subcomponent.

[0093] Specifically, the structured form processing subcomponent includes a hierarchical data processing component based on LLMs. LLMs (Large Language Models, LLMs) are large language models developed by Amazon, which can deeply understand natural language and perform hierarchical summarization of data content according to the arrangement structure of the data. For example: for structured form data, manual summary of the hierarchical structure is relatively complex. Therefore, the hierarchical data processing component based on LLMs is used to perform hierarchical summary, which saves manpower consumption and makes the summary more accurate.

[0094] Continue to refer Figure 6 , Figure 6 yes Figure 3The flowchart of a specific embodiment of step 303 shown includes:

[0095] Step 601, identifying the hierarchical structure of all structure tables in the structured data set through the LLMs-based hierarchical data processing component;

[0096] Step 602, all the structural forms in the structured data set are parsed layer by layer according to the hierarchical structures of all the structural forms, to obtain hierarchical parsed data corresponding to all the structural forms;

[0097] Step 603, clustering and merging the hierarchical parsing data corresponding to the same question and answer fields in different structure forms, and obtaining the clustering and merging results as the question and answer promotion data processed by the structure form processing subcomponent.

[0098] By identifying the hierarchical structure of the structural forms, the basic question and answer data is extracted according to the hierarchical structure of all structural forms, and the hierarchical parsing data corresponding to the same question and answer fields in different structural forms are clustered and merged. This makes it easier to directly use the clustered and merged question and answer promotion data in the subsequent question and answer pair generation and question and answer knowledge base generation.

[0099] Step 204, input the question and answer promotion data into a preset question and answer knowledge generation component.

[0100] Step 205, generating question-answer pairs for the question-answer promotion data through the question-answer knowledge generation component, caching the generated question-answer pairs into the target question-answer knowledge base, and completing the update of the target question-answer knowledge base.

[0101] In this embodiment, the question and answer knowledge generation component includes a question and answer knowledge generation component based on PingAnGPT. PingAnGPT is a Ping An big model platform that can analyze and integrate large quantities of input data according to business requirements to output expected target data. For example, in this embodiment, question and answer pair generation is used as a business goal, and the input question and answer promotion data is used as the data to be analyzed and integrated. After analysis and integration, the corresponding question and answer pairs are output, and then the constructed question and answer knowledge base is obtained through storage caching.

[0102] Continue to refer Figure 7 , Figure 7 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes:

[0103] Step 701, performing semantic analysis on all question fields contained in the question and answer promotion data by using the PingAnGPT-based question and answer knowledge generation component;

[0104] Step 702: clustering the question fields with the same semantic analysis results into the same question field using a preset clustering algorithm to obtain a question field clustering result, wherein the preset clustering algorithm includes a K-means clustering algorithm or a K nearest neighbor clustering algorithm;

[0105] Step 703, according to the question field clustering result, cluster the answer data corresponding to all question fields in the same cluster together to generate answer data sets corresponding to different clusters;

[0106] Step 704, extracting keywords from the answer data sets corresponding to different clusters to obtain answer keywords corresponding to different answer data sets;

[0107] Step 705, sorting the same question field and answer keyword corresponding to the same cluster to generate question-answer pairs, and caching the generated question-answer pairs to the target question-answer knowledge base, wherein the same question field and the answer keyword in the question-answer pair are in a one-to-one relationship or a one-to-many relationship.

[0108] In this embodiment, on the basis of question and answer promotion data, a question and answer knowledge generation component based on PingAnGPT is used to generate question and answer pairs, and then the question and answer pairs are cached in the target question and answer knowledge base to complete the update of the question and answer knowledge base, thereby avoiding the direct use of the question and answer knowledge generation component to organize the basic question and answer data of document type, image type and structure form type, thereby improving the efficiency of question and answer knowledge organization and use, and combining the data comprehensive processing component to process different knowledge types such as forms, pictures, documents, etc., thereby realizing compatible organization of multi-carrier type data.

[0109] This application obtains question and answer basic data sent by multiple data sources and inputs it into the data comprehensive processing component for processing to obtain question and answer promotion data; the question and answer promotion data is input into the question and answer knowledge generation component to generate question and answer pairs, and the generated question and answer pairs are cached into the target question and answer knowledge base to obtain the question and answer knowledge base. Based on the question and answer promotion data, this application uses a question and answer knowledge generation component based on PingAnGPT to generate question and answer pairs, and then caches the question and answer pairs into the target question and answer knowledge base to complete the update of the question and answer knowledge base, avoiding the direct use of the question and answer knowledge generation component to organize the basic question and answer data of document classes, image classes, and structured form classes, thereby improving the efficiency of question and answer knowledge organization and use. In combination with the data comprehensive processing component, different knowledge types such as forms, pictures, and documents are processed separately, achieving compatible organization of multi-carrier type data, and being able to effectively use multi-carrier type data sources for large-scale question and answer knowledge updates.

[0110] The embodiments of the present application can acquire and process relevant data based on image processing technology. Among them, image processing (Artificial Intelligence, AI) is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0111] Image processing basic technologies generally include sensors, dedicated image processing chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Image processing software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0112] In an embodiment of the present application, the question and answer promotion data is obtained by acquiring the basic question and answer data sent by multiple data sources and inputting it into the data comprehensive processing component for processing; the question and answer promotion data is input into the question and answer knowledge generation component to generate question and answer pairs, and the generated question and answer pairs are cached into the target question and answer knowledge base to obtain the question and answer knowledge base. Based on the question and answer promotion data, the present application uses a question and answer knowledge generation component based on PingAnGPT to generate question and answer pairs, and then caches the question and answer pairs into the target question and answer knowledge base to complete the update of the question and answer knowledge base, avoiding the direct use of the question and answer knowledge generation component to organize the basic question and answer data of document classes, image classes, and structured form classes, thereby improving the efficiency of question and answer knowledge organization and use. In combination with the data comprehensive processing component, different knowledge types such as forms, pictures, and documents are processed separately, achieving compatible organization of multi-carrier type data, and being able to effectively utilize multi-carrier type data sources for large-scale question and answer knowledge updates.

[0113] Further references Figure 8 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a question-answer knowledge base updating device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0114] like Figure 8 As shown, the question and answer knowledge base updating device 800 described in this embodiment includes: a question and answer basic data acquisition module 801, a question and answer basic data input module 802, a question and answer basic data processing module 803, a question and answer promotion data input module 804 and a question and answer knowledge base updating module 805. Among them:

[0115] The question and answer basic data acquisition module 801 is used to acquire the question and answer basic data sent by multiple data sources;

[0116] A question and answer basic data input module 802 is used to input the question and answer basic data into a preset data comprehensive processing component;

[0117] The question and answer basic data processing module 803 is used to process the question and answer basic data through the data comprehensive processing component to obtain question and answer promotion data;

[0118] A question and answer promotion data input module 804, used to input the question and answer promotion data into a preset question and answer knowledge generation component;

[0119] The question and answer knowledge base updating module 805 is used to generate question and answer pairs for the question and answer promotion data through the question and answer knowledge generation component, cache the generated question and answer pairs to the target question and answer knowledge base, and complete the update of the target question and answer knowledge base.

[0120] In this embodiment, the question-answer basic data processing module 803 includes a first processing submodule, a second processing submodule and a third processing submodule. Among them:

[0121] The first processing submodule is used to assign the document-type data set to the document processing subcomponent to obtain the question and answer promotion data processed by the document processing subcomponent. Specifically, the paragraph layout of all documents in the document-type data set is respectively identified by the spaCy-based natural language processing component to identify the paragraph layout of all documents; document segmentation is performed on all documents in the document-type data set according to the paragraph layout of all documents to obtain document slices after segmentation; grouped question slices and answer slices in the document slices are distinguished and marked, and all answer slices corresponding to the same question are clustered and sorted to obtain the clustering results as the question and answer promotion data processed by the document processing subcomponent.

[0122] The second processing submodule is used to assign the image data set to the image processing subcomponent to obtain the question and answer promotion data processed by the image processing subcomponent. Specifically, the image data and text data in the image data set are subjected to question and answer knowledge extraction by the multimodal data processing component based on GPT-4 to obtain the question and answer knowledge extraction result; the question and answer knowledge extraction result is distinguished and marked according to the difference of the images, and all the answer data corresponding to the same question are clustered and sorted to obtain the clustering and sorting result as the question and answer promotion data processed by the image processing subcomponent.

[0123] The third processing sub-module is used to assign the structured data set to the structured form processing sub-component to obtain the question and answer promotion data processed by the structured form processing sub-component. Specifically, the hierarchical structure of all the structured forms in the structured data set is identified through the LLMs-based hierarchical data processing component; all the structured forms in the structured data set are parsed layer by layer according to the hierarchical structure of all the structured forms to obtain the hierarchical parsed data corresponding to all the structured forms; the hierarchical parsed data corresponding to the same question and answer fields in different structured forms are clustered and merged to obtain the clustered and merged results as the question and answer promotion data processed by the structured form processing sub-component.

[0124] This application obtains question and answer basic data sent by multiple data sources and inputs it into the data comprehensive processing component for processing to obtain question and answer promotion data; the question and answer promotion data is input into the question and answer knowledge generation component to generate question and answer pairs, and the generated question and answer pairs are cached into the target question and answer knowledge base to obtain the question and answer knowledge base. Based on the question and answer promotion data, this application uses a question and answer knowledge generation component based on PingAnGPT to generate question and answer pairs, and then caches the question and answer pairs into the target question and answer knowledge base to complete the update of the question and answer knowledge base, avoiding the direct use of the question and answer knowledge generation component to organize the basic question and answer data of document classes, image classes, and structured form classes, thereby improving the efficiency of question and answer knowledge organization and use. In combination with the data comprehensive processing component, different knowledge types such as forms, pictures, and documents are processed separately, achieving compatible organization of multi-carrier type data, and being able to effectively use multi-carrier type data sources for large-scale question and answer knowledge updates.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0126] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0127] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0128] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 9 with components 9a-9c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0129] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0130] The memory 9a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 9a can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 9a can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 9a can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 9a is generally used to store the operating system and various application software installed on the computer device 9, such as a computer-readable instruction of a question-answer knowledge base update method, etc. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or are to be output.

[0131] The processor 9b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 9b is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to run the computer-readable instructions stored in the memory 9a or process data, such as running the computer-readable instructions of the question-answer knowledge base update method.

[0132] The network interface 9c may include a wireless network interface or a wired network interface. The network interface 9c is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0133] The computer device proposed in this embodiment belongs to the technical field of dialogue question and answer, and is applied to question and answer knowledge processing scenarios. This application obtains question and answer basic data sent by multiple data sources, and inputs it into the data comprehensive processing component for processing to obtain question and answer promotion data; the question and answer promotion data is input into the question and answer knowledge generation component to generate question and answer pairs, and the generated question and answer pairs are cached to the target question and answer knowledge base to obtain the question and answer knowledge base. Based on the question and answer promotion data, this application uses a question and answer knowledge generation component based on PingAnGPT to generate question and answer pairs, and then caches the question and answer pairs to the target question and answer knowledge base to complete the update of the question and answer knowledge base, avoiding the direct use of the question and answer knowledge generation component to organize the basic question and answer data of document classes, image classes and structured form classes, improving the efficiency of question and answer knowledge organization and use, and combining the data comprehensive processing component to process different knowledge types such as forms, pictures, documents, etc., to achieve compatible organization of multi-carrier type data, and can effectively use multi-carrier type data sources to perform large-scale question and answer knowledge updates.

[0134] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the question-and-answer knowledge base updating method as described above.

[0135] The computer-readable storage medium proposed in this embodiment belongs to the technical field of dialogue question and answer, and is applied to question and answer knowledge processing scenarios. This application obtains question and answer basic data sent by multiple data sources and inputs it into the data comprehensive processing component for processing to obtain question and answer promotion data; the question and answer promotion data is input into the question and answer knowledge generation component to generate question and answer pairs, and the generated question and answer pairs are cached to the target question and answer knowledge base to obtain the question and answer knowledge base. Based on the question and answer promotion data, this application uses a question and answer knowledge generation component based on PingAnGPT to generate question and answer pairs, and then caches the question and answer pairs to the target question and answer knowledge base to complete the update of the question and answer knowledge base, avoiding the direct use of the question and answer knowledge generation component to organize the basic question and answer data of document classes, image classes and structured form classes, improving the efficiency of question and answer knowledge organization and use, and combining the data comprehensive processing component to process different knowledge types such as forms, pictures, documents, etc., to achieve compatible organization of multi-carrier type data, and can effectively use multi-carrier type data sources to perform large-scale question and answer knowledge updates.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0137] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A question-answer knowledge base updating method, characterized in that: The steps include: Obtain the basic question and answer data sent by multiple data sources; Inputting the question-answer basic data into a preset data comprehensive processing component; Processing the question and answer basic data through the data comprehensive processing component to obtain question and answer promotion data; Inputting the question and answer promotion data into a preset question and answer knowledge generation component; The question-answer knowledge generation component generates question-answer pairs for the question-answer promotion data, caches the generated question-answer pairs in a target question-answer knowledge base, and completes the update of the target question-answer knowledge base.

2. The question-answer knowledge base updating method according to claim 1, characterized in that: The preset data comprehensive processing component includes a basic data type identification subcomponent. After executing the step of inputting the question and answer basic data into the preset data comprehensive processing component, the method further includes: Performing carrier type identification on all basic data in the question and answer basic data through the basic data type identification subcomponent, wherein the carrier refers to a sending carrier of the question and answer basic data, and the carrier type includes a document, an image, and a structured form; According to the carrier type identification result, all basic data in the question and answer basic data are classified and sorted to obtain the classified and sorted document data set, image data set and structured data set.

3. The question-answer knowledge base updating method according to claim 2, characterized in that: The data comprehensive processing component includes a document processing subcomponent, an image processing subcomponent and a structure form processing subcomponent. The step of processing the question and answer basic data by the data comprehensive processing component to obtain the question and answer promotion data specifically includes: Allocating the document-type data set to the document processing subcomponent to obtain the question-answering promotion data processed by the document processing subcomponent; Allocating the image data set to the image processing subcomponent to obtain the question-answering promotion data processed by the image processing subcomponent; The structured data set is distributed to the structured form processing subcomponent to obtain the question and answer promotion data processed by the structured form processing subcomponent.

4. The question-answer knowledge base updating method according to claim 3, characterized in that: The document processing subcomponent includes a natural language processing component based on spaCy. The step of allocating the document class data set to the document processing subcomponent and obtaining the question-answering promotion data processed by the document processing subcomponent specifically includes: Performing paragraph layout recognition on all documents in the document dataset by using the spaCy-based natural language processing component to identify the paragraph layout of all documents; Performing document segmentation processing on all documents in the document dataset according to the paragraph layout of all documents to obtain document slices after segmentation processing; The grouped question slices and answer slices in the document slices are distinguished and marked, and all answer slices corresponding to the same question are clustered and sorted, and the clustering results are obtained as the question and answer promotion data after being processed by the document processing subcomponent.

5. The question-answer knowledge base updating method according to claim 3, characterized in that: The image processing subcomponent includes a multimodal data processing component based on GPT-4. The step of allocating the image data set to the image processing subcomponent to obtain the question-answering promotion data processed by the image processing subcomponent specifically includes: Performing question-answering knowledge extraction on the image data and text data in the image data set through the GPT-4-based multimodal data processing component to obtain a question-answering knowledge extraction result; The question-answer knowledge extraction results are distinguished and marked according to the difference in images, and all the answer data corresponding to the same question are clustered and sorted, and the clustering results are obtained as the question-answer promotion data processed by the image processing subcomponent.

6. The question-answer knowledge base updating method according to claim 3, characterized in that: The structured form processing subcomponent includes a hierarchical data processing component based on LLMs. The step of allocating the structured data set to the structured form processing subcomponent and obtaining the question and answer promotion data processed by the structured form processing subcomponent specifically includes: Identifying the hierarchical structure of all structure tables in the structured data set by the LLMs-based hierarchical data processing component; According to the hierarchical structure of all the structural forms, all the structural forms in the structured data set are parsed layer by layer to obtain hierarchical parsed data corresponding to all the structural forms; The hierarchical parsing data corresponding to the same question and answer fields in different structure forms are clustered and merged, and the clustering and merging results are obtained as the question and answer promotion data processed by the structure form processing subcomponent.

7. The question-answer knowledge base updating method according to any one of claims 1 to 6, characterized in that: The question-answer knowledge generation component includes a question-answer knowledge generation component based on PingAnGPT, and the step of generating question-answer pairs for the question-answer promotion data by using the question-answer knowledge generation component and caching the generated question-answer pairs into a target question-answer knowledge base specifically includes: Performing semantic analysis on all question fields contained in the question and answer promotion data by using the PingAnGPT-based question and answer knowledge generation component; Using a preset clustering algorithm to cluster question fields with the same semantic analysis results into the same question field, and obtaining a question field clustering result, wherein the preset clustering algorithm includes a K-means clustering algorithm or a K nearest neighbor clustering algorithm; According to the question field clustering result, the answer data corresponding to all question fields in the same clustering cluster are clustered together to generate answer data sets corresponding to different clustering clusters; By extracting keywords from the answer data sets corresponding to different clusters, the answer keywords corresponding to different answer data sets are obtained; The same question field and answer keyword corresponding to the same cluster are sorted to generate question-answer pairs, and the generated question-answer pairs are cached in the target question-answer knowledge base, wherein the same question field and the answer keyword in the question-answer pair are in a one-to-one relationship or a one-to-many relationship.

8. A question-answer knowledge base updating device, characterized in that: include: A question-and-answer basic data acquisition module is used to acquire question-and-answer basic data sent by multiple data sources; A question-and-answer basic data input module, used to input the question-and-answer basic data into a preset data comprehensive processing component; A question and answer basic data processing module, used to process the question and answer basic data through the data comprehensive processing component to obtain question and answer promotion data; A question and answer promotion data input module, used to input the question and answer promotion data into a preset question and answer knowledge generation component; The question and answer knowledge base updating module is used to generate question and answer pairs for the question and answer promotion data through the question and answer knowledge generation component, cache the generated question and answer pairs to the target question and answer knowledge base, and complete the update of the target question and answer knowledge base.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the question-and-answer knowledge base updating method as claimed in any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the question-and-answer knowledge base updating method according to any one of claims 1 to 7 are implemented.

Images