Question and answer method, electronic device and storage medium

Abstract

The present application discloses a question-and-answer method, an electronic device and a storage medium. The electronic device responds to a question request, performs question recall processing on the question request based on data in a preset database, and obtains first question information; when second question information exists in the first question information, determines whether the second question information is question information in a first data source; wherein the second question information represents question information in the first question information, in which a first correlation parameter is greater than a first threshold; the first correlation parameter represents a parameter of the correlation between the first question information and the question request; the first data source represents question-and-answer pair data in a preset database; when the second question information is question information in the first data source, determines answer information corresponding to the second question information as answer data for the question request, thereby effectively improving the accuracy and reliability of question-and-answer.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a question-answering method, an electronic device, and a storage medium. Background Art

[0002] For current question-answering methods, it is usually possible to retrieve text data, or determine the answer to a question through knowledge graphs or direct text generation. However, these methods still have certain limitations. For example, they may be limited by the content of the retrieval and the coverage of the knowledge graph, resulting in low accuracy or poor reliability of the answer. This shows that how to improve the accuracy and reliability of question-answering is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a question-and-answer method, an electronic device, and a storage medium, which can effectively improve the accuracy and reliability of question-and-answer.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a question-answering method, the method comprising:

[0006] In response to the question request, the question request is recalled based on the data in the preset database to obtain first question information; wherein the preset database is used to provide the data for the recall processing; the preset database is constructed by acquiring the document data to be processed, performing data cleaning processing on the document data to be processed, obtaining the processed document data, and based on the processed document data; the data cleaning processing is used to remove the erroneous data in the document data to be processed;

[0007] In the case where the second question information exists in the first question information, determining whether the second question information is question information in the first data source; wherein the second question information represents question information in the first question information, in which the first correlation parameter is greater than the first threshold; the first correlation parameter represents a parameter of correlation between the first question information and the question request; the first data source represents question-answer pair data in a preset database;

[0008] In the case where the second question information is question information in the first data source, answer information corresponding to the second question information is determined as answer data of the question request.

[0009] In this embodiment, when the electronic device receives a question request, it can first use the data in the preset database to perform question recall processing on the question request to obtain first question information related to the question request, and then select second question information with a first correlation parameter greater than a first threshold from the first question information. If there is second question information that meets the conditions in the first question information, it can continue to determine whether these second question information come from the question and answer pair data in the preset database. If so, the answer information corresponding to the second question information can be directly output as the answer data. Therefore, through the constructed preset database, in conjunction with the question recall processing, the accuracy and reliability of the question and answer can be greatly improved. In addition, by acquiring the to-be-processed document data in different fields and performing relevant data processing, a preset database can be constructed, which can improve the knowledge covered by the preset database, thereby enhancing the application effect of the question and answer method in different fields.

[0010] In some embodiments of the present application, the method further comprises:

[0011] In the case that the second question information does not exist in the first question information, a relevance evaluation process is performed based on the question request and the document data in the preset database to determine the third question information; wherein the third question information represents the question information generated based on the document data and the relevance with the question request meets the first relevance condition;

[0012] When the fourth question information exists in the third question information, the fourth question information is reordered to obtain reordered question information, and answer data is determined based on the reordered question information; wherein the fourth question information represents question information in the third question information whose similarity with the question request is greater than a second threshold.

[0013] In this embodiment, when the second question information does not exist in the first question information, relevance evaluation processing can be performed based on the question request and the document data in the preset database to generate third question information based on the document data and the question request information, and then determine whether there is fourth question information in the third question information whose similarity with the question request is greater than a second threshold. If so, the fourth question information can be reordered, and the answer data can be determined based on the reordered question information, which can improve the accuracy of questions and answers.

[0014] In some embodiments of the present application, determining answer data based on the reordered question information includes:

[0015] Determine whether there is fifth question information in the reordered question information; wherein the fifth question information represents the reordered question information, the question information whose second correlation parameter is greater than the third threshold, and the second correlation parameter represents a parameter of correlation between the reordered question information and the question request;

[0016] When the fifth question information exists in the reordered question information, a document recall process is performed based on the fifth question information to determine the answer data.

[0017] In this embodiment, it can be determined whether there is a fifth question information in the reordered question information whose second correlation parameter is greater than the third threshold. If so, document recall processing can be performed based on the fifth question information to obtain answer data, which can greatly improve the accuracy of questions and answers.

[0018] In some embodiments of the present application, the method further comprises:

[0019] When the fifth question information does not exist in the reordered question information, answer data is determined based on the first language model, the question request, and the historical interaction data.

[0020] In this embodiment, the first language model can be used as a fallback strategy. When the fifth question information does not exist in the reordered question information, answer data can be generated based on the first language model, question requests, and historical interaction data to improve the comprehensiveness and accuracy of the answer.

[0021] In some embodiments of the present application, the method further comprises:

[0022] When the fourth question information does not exist in the third question information, answer data is determined based on the first language model, the question request, and the historical interaction data.

[0023] In this embodiment, when the fourth question information does not exist in the third question information, answer data can be generated based on the first language model, the question request and the historical interaction data, thereby improving the comprehensiveness and accuracy of the answer.

[0024] In some embodiments of the present application, the preset database includes a first database and a second database; the first question information includes sixth question information and seventh question information; and the question request is recalled based on the data in the preset database to obtain the first question information, including:

[0025] Calculate similarity between the question data in the question-answer pair data and the document and query data stored in the first database and the question request to obtain a first similarity result, and determine sixth question information from the question data based on the first similarity result;

[0026] Vectorize the question request to obtain a vectorized question request;

[0027] A similarity calculation is performed on the vectorized question data and the vectorized question request to obtain a second similarity result, and the seventh question information is determined from the vectorized question data based on the second similarity result; wherein the vectorized question data is obtained based on the vectorized question-answer pair data and the vectorized document and query data stored in the second database.

[0028] In this embodiment, the preset database may include a first database and a second database. The first database can be used to store question-answer pair data and document and query data, and the second database can be used to store vectorized question-answer pair data and vectorized document and query data. When performing question recall processing to obtain the first question information, the electronic device can perform similarity calculations based on the first database and the question request to determine the sixth question information. After vectorizing the question request, a similarity calculation is performed based on the second database and the vectorized question request to determine the seventh question information. By recalling questions in the first database and the second database at the same time, the reliability of question recall can be effectively improved, thereby improving the reliability of question and answer.

[0029] In some embodiments of the present application, determining the sixth question information from the question data based on the first similarity result includes:

[0030] selecting a first number of eighth question information from the question data based on the first similarity result;

[0031] Performing vectorization processing on the eighth question information to obtain vectorized eighth question information;

[0032] A similarity calculation is performed on the vectorized eighth question information and the vectorized question request to obtain a third similarity result, and question information with similarity information greater than a fourth threshold in the third similarity result is determined as sixth question information.

[0033] In this embodiment, when determining the sixth question information, the first number of eighth question information can be selected from the question data based on the first similarity result, and then the eighth question information can be vectorized to calculate the similarity between the vectorized eighth question information and the vectorized question request, so as to select the sixth question information based on the third similarity result obtained, thereby making the selected sixth question information have a higher similarity with the question request, thereby improving the accuracy of the answer.

[0034] In some embodiments of the present application, the method further comprises:

[0035] In the case where the second question information is not the question information in the first data source, a document recall process is performed based on the second question information to determine the answer data.

[0036] In this embodiment, if the second question information does not come from the question-answer pair data in the preset database, the document recall processing will continue to be performed based on the second question information to determine the answer data. In this way, the accuracy and reliability of the question and answer can be improved by combining the recall processing of the document with the recall processing of the question.

[0037] In some embodiments of the present application, the document recall process is performed based on the second question information to determine the answer data, including:

[0038] Performing context expansion processing on the document unit corresponding to the second question information to obtain an expanded document unit;

[0039] The expanded document unit and question request are analyzed and processed based on the second language model to obtain answer data.

[0040] In this embodiment, when recalling documents based on the second question information, context expansion processing can be first performed based on the document unit corresponding to the second question information to expand the relevant document units that are subsequently analyzed and processed based on the second language model, and then the second language model is used to analyze and process the expanded document units and question requests to obtain answer data, which can effectively improve the reliability and accuracy of the answers.

[0041] In some embodiments of the present application, the preset database includes a third database; constructing the preset database based on the processed document data includes:

[0042] Performing document segmentation processing on the processed document data based on the format of the processed document data to obtain document units corresponding to the processed document data;

[0043] storing the document unit and first information of the document unit in a first database; wherein the first information includes at least one of source information, author information and date information of the document unit;

[0044] The processed document data is stored in the third database to complete the construction of the preset database.

[0045] In this embodiment, the processed document data can be segmented according to the format of the processed document data to obtain corresponding document units, and then the document units and the first information of the document units are stored in the first database, and the processed document data is stored in the third database to complete the construction of the preset database.

[0046] In some embodiments of the present application, the preset database further includes a fourth database; and the method further includes:

[0047] generating a first password corresponding to the processed document data;

[0048] The document unit corresponding to the processed document data and the context information of the document unit in the processed document data are stored in the fourth database according to the first password.

[0049] In this embodiment, the document unit corresponding to each document data can be stored in the fourth database according to the unified first password, so as to improve the traceability of the document unit in the fourth database and improve the post-processing capability of the preset database.

[0050] In some embodiments of the present application, the method further comprises:

[0051] Get question-answer pair data;

[0052] Performing expansion processing on the question-answer pair data based on a preset expansion model to obtain expanded question-answer pair data; wherein the expansion processing includes at least one of semantic expansion processing, synonym replacement processing, and intent recognition processing;

[0053] The expanded question-answer pair data is stored in the first database.

[0054] In this embodiment, question-and-answer pair data from different fields can be obtained, and then the question-and-answer pair data can be subjected to at least one of semantic expansion processing, synonym replacement processing, and intent recognition processing, so as to improve the semantic richness of the question-and-answer pair data. The preset database thus constructed can improve the accuracy of subsequent questions and answers.

[0055] In some embodiments of the present application, the method further comprises:

[0056] Performing query extraction processing on the document unit using the first language model to obtain query data corresponding to the document unit;

[0057] Documents and query data are constructed according to the document units and the query data, and the documents and query data are stored in the first database.

[0058] In this embodiment, when acquiring documents and query data, the first language model can be used to perform query extraction processing on the document units obtained after the segmentation processing, so as to obtain the potential query data of each document unit, and store the documents and query data based on the document units and the corresponding query data, which can enhance the understanding and matching capabilities of the document units and improve the flexibility and accuracy when processing complex documents.

[0059] In a second aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a first recall unit and a determination unit;

[0060] A first recall unit is used to respond to a question request and perform a question recall process on the question request based on data in a preset database to obtain first question information; the preset database is constructed by acquiring document data to be processed, performing data cleaning process on the document data to be processed, obtaining processed document data, and based on the processed document data; the data cleaning process is used to remove erroneous data in the document data to be processed;

[0061] A determination unit is used to determine whether the second question information is question information in the first data source when the second question information exists in the first question information; wherein the second question information represents question information in the first question information whose first correlation parameter is greater than a first threshold; the first correlation parameter represents a parameter of the correlation between the first question information and the question request; the first data source represents question-answer pair data in a preset database; and when the second question information is question information in the first data source, determining answer information corresponding to the second question information as answer data for the question request.

[0062] In this embodiment, when the electronic device receives a question request, it can first use the data in the preset database to perform question recall processing on the question request to obtain first question information related to the question request, and then select second question information whose first correlation parameter is greater than the first threshold from the first question information. If there is second question information that meets the conditions in the first question information, it can continue to determine whether the second question information comes from the question and answer pair data in the preset database. If so, the answer information corresponding to the second question information can be directly output as the answer data. Therefore, through the constructed preset database, the accuracy and reliability of the question and answer can be greatly improved in combination with the question recall processing.

[0063] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory storing instructions executable by the processor; when the instructions are executed by the processor, the above-mentioned question and answer method is implemented.

[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned question-and-answer method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below.The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application.

[0066] Figure 1 Schematic diagram of the implementation process of the question-answering method proposed in the embodiment of the present application Figure 1 ;

[0067] Figure 2 Schematic diagram of the implementation process of the question-answering method proposed in the embodiment of the present application Figure 2 ;

[0068] Figure 3 Schematic diagram of the implementation process of the question-answering method proposed in the embodiment of the present application Figure 3 ;

[0069] Figure 4 Schematic diagram of the implementation process of the question-answering method proposed in the embodiment of the present application Figure 4 ;

[0070] Figure 5 A schematic diagram of building a preset database proposed in an embodiment of the present application;

[0071] Figure 6 Schematic diagram of the implementation process of the question-answering method proposed in the embodiment of the present application Figure 5 ;

[0072] Figure 7 Schematic diagram of the composition structure of the electronic device proposed in the embodiment of the present application Figure 1 ;

[0073] Figure 8 Schematic diagram of the composition structure of the electronic device proposed in the embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for ease of description, only the parts related to the related applications are shown in the drawings.

[0075] With the significant improvement of the performance of Large Language Model (LLM), its application in various application scenarios has shown a rapid growth trend. However, the current popular knowledge question-answering systems, including retrieval-based, generation-based and knowledge graph-based methods, all have certain limitations. For example, the retrieval-based method is limited by the indexed content. If the indexed data does not contain the answer to the question, the system cannot provide the correct answer. The ambiguity and complexity of natural language make accurate retrieval a challenge. The knowledge graph-based method is limited by the coverage of the knowledge graph, resulting in limited answers. Building a comprehensive and updated knowledge graph requires a lot of resources, and the knowledge graph usually needs manual maintenance and verification, which is a time-consuming and complicated process. The generation-based method may lack accuracy, especially for some answers that require detailed factual support, and the model does not directly rely on specific data sources when generating answers. The historical data that the model training relies on may contain bias or outdated information. The internal working mechanism of the generation model is complex and difficult to explain and verify. It can be seen that the current question-answering methods generally have the problem of low accuracy or poor reliability.

[0076] In order to solve the problems existing in the statistics and query of current indicator data, the recall mechanism proposed in the present application can, upon receiving a question request, first use the data in the preset database to recall the question request to obtain the first question information related to the question request, and then select the second question information whose first correlation parameter is greater than the first threshold from the first question information. If there is second question information that meets the conditions in the first question information, it can be further determined whether the second question information comes from the question and answer pair data in the preset database. If so, the answer information corresponding to the second question information can be directly output as the answer data. Through the constructed preset database, the recall processing of the question can greatly improve the accuracy and reliability of the question and answer.

[0077] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0078] An embodiment of the present application provides a question-answering method, such as Figure 1 As shown, the question-answering method of the electronic device may include the following steps:

[0079] Step 101: In response to a question request, recall processing is performed on the question request based on data in a preset database to obtain first question information.

[0080] In an embodiment of the present application, the electronic device may respond to the question request and perform question recall processing on the question request based on data in a preset database to obtain first question information.

[0081] In the embodiment of the present application, the question request is a question request input by a user and received by the electronic device; for example, the question request may be “which article does XX content come from”.

[0082] In an embodiment of the present application, question recall processing may be used to recall questions related to question requests.

[0083] In an embodiment of the present application, a preset database can be used to provide data for recall processing; the preset database is constructed based on the processed document data by acquiring document data to be processed, performing data cleaning processing on the document data to be processed, obtaining processed document data; and data cleaning processing is used to remove erroneous data in the document data to be processed.

[0084] In an embodiment of the present application, the preset database may include a first database and a second database; wherein the first database may be used to store question-answer pair data and document and query data; the second database may be used to store vectorized question-answer pair data and vectorized document and query data; the present application does not specifically limit the types of the first database and the second database, for example, the first database may be an elastic search engine (Elasticsearch); the second database may be an open source vector database Milvus.

[0085] Among them, question-answer data is a type of data that includes questions (Question) and corresponding answers (Answer), and document (Document) and query (Question) data is a type of data that includes documents and queries extracted based on documents, wherein queries can also be understood as questions, and the number of queries extracted based on the same document can be multiple, that is, multiple queries can be extracted from one document to obtain query inputs from multiple angles, thereby enhancing the understanding and matching capabilities of document content.

[0086] Exemplarily, question-answer data may include the question "What does XX poem mean" and the corresponding answer "The poem means XXX"; document and query data may include document A and multiple potential queries based on document A and extracted, for example, multiple queries may include "When is the publication date of document A", "Which chapter of document A does XX content come from", and "Who is the author of document A", etc.

[0087] In an embodiment of the present application, recall processing of questions can be performed based on the first database and the second database respectively; the first question information may include sixth question information and seventh question information, wherein the sixth question information is a question obtained by recall processing of questions based on the first database, and the seventh question information is a question obtained by recall processing of questions based on the second database.

[0088] In some embodiments of the present application, when the electronic device performs question recall processing on a question request based on data in a preset database to obtain first question information, it can perform similarity calculation on the question data in the question and answer pair data and the document and query data stored in the first database and the question request to obtain a first similarity result, and determine sixth question information from the question data based on the first similarity result; and perform vectorization processing on the question request to obtain a vectorized question request; and then perform similarity calculation on the vectorized question data and the vectorized question request to obtain a second similarity result, and determine seventh question information from the vectorized question data based on the second similarity result; wherein, the vectorized question data is obtained based on the vectorized question and answer pair data and the vectorized document and query data stored in the second database.

[0089] In an embodiment of the present application, when performing similarity calculation, an inverted index can be constructed based on the data in the first database, and the question request can be segmented to obtain a term list; then the best match (BM) 25 algorithm can be used to calculate the similarity of the question in the inverted index and the term list corresponding to the question request to obtain a first similarity result.

[0090] In some embodiments of the present application, when using the BM25 algorithm to perform similarity calculation on the term lists corresponding to the questions in the inverted index and the question requests to obtain a first similarity result, the inverse document frequency of each word in the term list can be calculated first, and then the BM25 algorithm and the inverse document frequency of each word can be used to calculate the correlation score between each word in the term list and the questions in the inverted index, and then the overall correlation score between the question request and each question in the inverted index is determined based on the correlation score and the weight of each word, so that the overall correlation score is determined as the first similarity result.

[0091] In some embodiments of the present application, when the electronic device determines the sixth question information from the question data based on the first similarity result, it may select a first number of eighth question information from the question data based on the first similarity result; perform vectorization processing on the eighth question information to obtain vectorized eighth question information; perform similarity calculation on the vectorized eighth question information and the vectorized question request to obtain a third similarity result, and determine the question information in the third similarity result whose similarity information is greater than a fourth threshold as the sixth question information.

[0092] In some embodiments of the present application, when calculating the similarity between the vectorized eighth question information and the vectorized question request, cosine similarity may be used to calculate to obtain a third similarity result; the calculation method of cosine similarity may be expressed as the following formula:

[0093] (1)

[0094] in, Represents a vectorized question request, Indicates its modulus length; represents the vectorized problem data, Indicates its modulus length.

[0095] In the embodiments of the present application, the value of the fourth threshold is not limited in the present application.

[0096] In the embodiments of the present application, the specific value of the first number is not limited in the present application, for example, the first number may be 10.

[0097] In some embodiments of the present application, when selecting a first number of eighth question information from question data based on the first similarity result, the similarities in the first similarity result can be arranged in order from large to small, and the questions corresponding to the first number of similarities in the sorted first similarity result are taken as the eighth question information.

[0098] Exemplarily, the questions corresponding to the top 10 similarities in the first similarity result are used as the eighth question information.

[0099] In some embodiments of the present application, the question request can be vectorized based on the vectorization model to obtain a vectorized question request; the eighth question information can also be vectorized based on the vectorization model to obtain vectorized eighth question information; wherein, the vectorization model is not limited in this application, for example, the vectorization model can be a semantic vector model (BAAI General Embedding, BGE) embedding model; for example, when the question request is vectorized, the question request can be input into the vectorization model, and the vectorization model can be used to convert the question request into a vector to obtain a vectorized question request.

[0100] Exemplarily, the eighth question information includes 10 questions, and among the 10 questions, there are 5 questions whose similarity information is greater than the fourth threshold, and then these 5 questions can be determined as the sixth question information.

[0101] It can be understood that in an embodiment of the present application, the question-answer pair data includes questions, and the document and query data include queries, then the vectorized question-answer pair data may include vectorized question data, and the vectorized document and query data may include vectorized query data; the vectorized question data is composed of vectorized question data and vectorized query data.

[0102] In some embodiments of the present application, when determining the seventh question information from the vectorized question data based on the second similarity result, the similarities in the second similarity result can also be arranged in order from large to small, and the questions corresponding to the first number of similarities in the sorted second similarity results are taken as the ninth question information, and then the question information in the ninth question information whose similarity information is greater than the fifth threshold is determined as the seventh question information; wherein, the numerical value of the fifth threshold is not limited in this application.

[0103] In some embodiments of the present application, when calculating the similarity between the vectorized question data and the vectorized question request, the calculation can also be performed based on the aforementioned cosine similarity method to obtain a second similarity result.

[0104] Step 102: When the second question information exists in the first question information, determine whether the second question information is the question information in the first data source; wherein the second question information represents the question information in the first question information, in which the first correlation parameter is greater than the first threshold; the first correlation parameter represents the parameter of the correlation between the first question information and the question request; and the first data source represents the question-answer pair data in the preset database.

[0105] In an embodiment of the present application, the electronic device can, in response to a question request, perform question recall processing on the question request based on data in a preset database, and after obtaining the first question information, determine whether the second question information is the question information in the first data source if the second question information exists in the first question information; wherein the second question information represents the question information in the first question information in which the first correlation parameter is greater than the first threshold; the first correlation parameter represents the parameter of the correlation between the first question information and the question request; and the first data source represents the question and answer pair data in the preset database.

[0106] In an embodiment of the present application, the first question information can be reordered based on the first reordering model to obtain the reordered first question information, and then the first correlation parameter of the reordered first question information and the first threshold are compared to determine whether second question information exists.

[0107] In an embodiment of the present application, the first re-ranking model may be used to re-rank the question data obtained through the question recall process, so as to re-evaluate the relevance of similar questions obtained through the question recall process.

[0108] In an embodiment of the present application, the question-answer pair data in the preset database includes the question-answer pair data in the first database and the vectorized question-answer pair data in the second database.

[0109] Exemplarily, the second question information exists in the first question information. If the second question information is a question in a question-answer pair in the first database, it can be determined that the second question information is the question information in the first data source.

[0110] Step 103: When the second question information is question information in the first data source, the answer information corresponding to the second question information is determined as answer data of the question request.

[0111] In an embodiment of the present application, the electronic device may, when the second question information exists in the first question information, determine whether the second question information is the question information in the first data source, and, when the second question information is the question information in the first data source, determine the answer information corresponding to the second question information as the answer data of the question request.

[0112] It can be understood that in an embodiment of the present application, if the second question information is question information from the first data source, since the preset database already has answer information in the question and answer data corresponding to the question information, the corresponding answer information can be directly output as answer data to complete the answer to the question request input by the user.

[0113] Exemplarily, the second question information is question C in a question-answer pair data in the first database, and the answer to question C in the question-answer pair data is B, then B can be directly output as the answer data to the question request.

[0114] In the embodiments of the present application, Figure 2 As shown, in the case where the second question information exists in the first question information, after determining whether the second question information is the question information in the first data source, that is, after step 102, the following steps may also be included:

[0115] Step 104: When the second question information is not the question information in the first data source, a document recall process is performed based on the second question information to determine answer data.

[0116] In an embodiment of the present application, the electronic device may, if the second question information exists in the first question information, determine whether the second question information is the question information in the first data source; and if the second question information is not the question information in the first data source, perform document recall processing based on the second question information to determine answer data.

[0117] In some embodiments of the present application, when the electronic device performs a document recall process based on the second question information to determine the answer data, it can perform context expansion processing based on the document unit corresponding to the second question information to obtain the expanded document unit; and then analyze and process the expanded document unit and the question request based on the second language model to obtain the answer data.

[0118] It can be understood that in the embodiments of the present application, document recall processing is performed based on the second question information, and document data related to the document unit corresponding to the second question information can be recalled, so that answer data can be determined based on the recalled document data related to the document unit corresponding to the second question information.

[0119] In some embodiments of the present application, when performing context expansion processing based on the document unit corresponding to the second question information, the context expansion processing may be performed based on the contextual relationship of the original document in which the document unit is located.

[0120] In some embodiments of the present application, the contextual relationship corresponding to the document unit may be acquired from a fourth database in the preset database.

[0121] In the embodiments of the present application, the second language model is not limited in the present application. For example, the second language model may be a large language model. The second language model may use its ability to understand language and capture contextual information to perform a comprehensive analysis on input questions and recalled document units, thereby obtaining more accurate answer data.

[0122] In the embodiments of the present application, traditional text retrieval based on inverted index and semantic retrieval based on vector can obtain relevant information from multiple angles to improve the accuracy and comprehensiveness of recall, thereby improving the accuracy of question and answer.

[0123] In the embodiments of the present application, Figure 3 As shown, after the electronic device performs question recall processing on the question request based on the data in the preset database in response to the question request and obtains the first question information, that is, after step 101, the following steps may also be included:

[0124] Step 105: When the second question information does not exist in the first question information, a relevance evaluation process is performed based on the question request and the document data in the preset database to determine the third question information.

[0125] In an embodiment of the present application, the electronic device, in response to a question request, performs question recall processing on the question request based on data in a preset database, and after obtaining first question information, may, if the second question information does not exist in the first question information, perform relevance evaluation processing based on the question request and document data in the preset database to determine third question information.

[0126] In some embodiments of the present application, the BM25 algorithm may be used to perform correlation evaluation processing, and its evaluation method may be expressed as the following formula:

[0127] (2)

[0128] in, Represents document data, Indicates a question request, which can include n words ; Expressive words In the documentation The frequency of occurrence in Representation Document The length of , i.e. the number of words; Indicates the average length of all documents in the document set of the preset database; and Indicates adjustable parameters, common default values ​​are 1.2 and 0.75; Expressive words The inverse document frequency can be calculated by the following formula:

[0129] (3)

[0130] Where N represents the total number of documents. Indicates that it contains words The number of documents.

[0131] In an embodiment of the present application, the third question information represents question information generated based on document data and whose relevance to the question request meets the first relevance condition; wherein the first relevance condition may be to rank the second number of relevance results in the relevance evaluation results, and the numerical value of the second number is not limited in this application, for example, the second number may be 10.

[0132] Step 106: When the fourth question information exists in the third question information, the fourth question information is reordered to obtain reordered question information, and answer data is determined based on the reordered question information.

[0133] In an embodiment of the present application, when the second question information does not exist in the first question information, the electronic device may perform a relevance evaluation based on the question request and the document data in a preset database to determine the third question information. Then, when the fourth question information exists in the third question information, the electronic device may reorder the fourth question information to obtain reordered question information, and determine the answer data based on the reordered question information.

[0134] In an embodiment of the present application, the fourth question information represents question information in the third question information whose similarity with the question request is greater than a second threshold, wherein the value of the second threshold is not limited in this application.

[0135] Exemplarily, the determined third question information includes 10 questions, and the similarities corresponding to 6 of the 10 questions are greater than the second threshold. Then it can be determined that the third question information contains fourth question information, and the fourth question information includes these 6 questions whose similarities are greater than the second threshold.

[0136] In some embodiments of the present application, the fourth question information may be reordered based on the second reordering model.

[0137] In the embodiments of the present application, the specific value of the third threshold is not limited in the present application.

[0138] In an embodiment of the present application, when determining answer data based on reordered question information, it can be determined whether fifth question information exists in the reordered question information; and then, in the event that fifth question information exists in the reordered question information, document recall processing is performed based on the fifth question information to determine the answer data.

[0139] In an embodiment of the present application, the fifth question information represents question information in the reordered question information whose second correlation parameter is greater than the third threshold, and the second correlation parameter represents a parameter of the correlation between the reordered question information and the question request.

[0140] It can be understood that, in the embodiment of the present application, the recall processing of the document based on the fifth question information can recall the document data related to the document unit corresponding to the fifth question information.

[0141] In some embodiments of the present application, when the electronic device performs a document recall process based on the fifth question information to determine the answer data, it can perform context expansion processing based on the document unit corresponding to the fifth question information to obtain the expanded document unit; and analyze and process the expanded document unit and the question request based on the second language model to obtain the answer data.

[0142] In an embodiment of the present application, after determining whether the fifth question information exists in the reordered question information, the electronic device may also determine the answer data based on the first language model, the question request and the historical interaction data if the fifth question information does not exist in the reordered question information.

[0143] In an embodiment of the present application, the first language model can be used as a fallback strategy, that is, when it is impossible to rely on the recall processing of the aforementioned questions and the recall processing of the documents to provide answers, the answer data can be determined based on the first language model; wherein, the first language model can also adopt a large language model, but the first language model can be more suitable for processing larger-scale and more complex historical interaction data than the second language model.

[0144] In an embodiment of the present application, the historical interaction data represents the historical interaction data corresponding to the user who inputs the question request.

[0145] In some embodiments of the present application, question requests and historical interaction data may be input into a first language model, and answer data may be output after being processed by the first language model.

[0146] In the embodiments of the present application, Figure 4 As shown, when the electronic device does not contain the second question information in the first question information, after performing relevance evaluation based on the question request and the document data in the preset database to determine the third question information, that is, after step 105, the following steps may also be included:

[0147] Step 107: When the fourth question information does not exist in the third question information, answer data is determined based on the first language model, the question request, and the historical interaction data.

[0148] In an embodiment of the present application, before the electronic device performs question recall processing on the question request based on data in a preset database in response to the question request and obtains the first question information, that is, before step 101, the electronic device may further include the following steps:

[0149] Step 108: Obtain document data to be processed.

[0150] In an embodiment of the present application, the electronic device may obtain the document data to be processed before responding to the question request and performing question recall processing on the question request based on data in a preset database and obtaining the first question information.

[0151] In an embodiment of the present application, the document data to be processed may include documents from different fields or professions, for example, may include academic papers, professional manuals, industry reports and other documents from different fields.

[0152] Step 109: Perform data cleaning on the document data to be processed to obtain processed document data.

[0153] In an embodiment of the present application, after acquiring the document data to be processed, the electronic device may perform data cleaning processing on the document data to be processed to obtain processed document data.

[0154] In some embodiments of the present application, data cleaning processing can be used to remove erroneous data in the document data to be processed. For example, data cleaning processing can include operations of eliminating irrelevant information, correcting language errors, and unifying formats, so that there is no erroneous data in the processed document data and the formats of each document are unified.

[0155] In some embodiments of the present application, semantic annotation processing can also be performed on the document data to be processed, and then when the document data to be processed is subsequently stored in a database, the document data to be processed and the corresponding annotation semantics can be stored, so as to facilitate post-processing of the document data in the database. For example, when querying and updating the database, the annotated semantics can be used to perform targeted queries and updates.

[0156] Step 110: construct a preset database based on the processed document data.

[0157] In an embodiment of the present application, after the electronic device performs data cleaning processing on the document data to be processed to obtain the processed document data, it can build a preset database based on the processed document data.

[0158] In some embodiments of the present application, the preset database may also include a third database; the third database can be used to store complete document data, and can maintain the originality, integrity and traceability of the data; the type of the third database is not limited in this application, for example, the third database can be Amazon Simple Storage Service (Amazon S3) or Minio.

[0159] In some embodiments of the present application, when the electronic device constructs a preset database based on the processed document data, it can perform document segmentation processing on the processed document data based on the format of the processed document data to obtain document units corresponding to the processed document data; and then store the document units and the first information of the document units in the first database; and store the processed document data in the third database to complete the construction of the preset database.

[0160] In an embodiment of the present application, by performing document segmentation processing on the processed document data, the document can be decomposed into smaller units based on the fine granularity of the document, and different segmenters can be used to perform document segmentation processing on the processed document data according to the formats of different documents; for example, the recursive character text splitter (Recursive Character Text Splitter) in the open source framework Langchain can be used to perform document segmentation processing. When performing document segmentation processing, document segmentation can be performed based on multi-level separation characters, such as "\n\n", "\n", etc. When segmenting, segmentation is performed preferentially according to the paragraph level. If the unit size (chunk size) is exceeded, segmentation can be performed at a smaller granularity, wherein the unit size represents the maximum length of the segmentation of the input text sequence; for documents in the lightweight markup language (Markdown) format, the lightweight markup language title text splitter (Markdown Header) in Langchain can be used. TextSplitter) can identify the titles marked by "#" in the document and split them, which can ensure the consistency of content and title; for documents in PDF format, PDF documents can be converted to HTML and then split. This method can deduce the hierarchical relationship of the content while retaining the font size information; in addition, for documents in PDF format, the PyMuPDF module can be used to obtain block information, restore the article structure by font splicing, and then apply the recursive character text splitter for segmentation.

[0161] Exemplarily, the first database may be an elastic search engine, which may support building an efficient index structure, thereby achieving fast retrieval.

[0162] In an embodiment of the present application, the first information may include at least one of source information, author information, and date information of the document unit.

[0163] In some embodiments of the present application, when constructing a preset database, the electronic device may also vectorize the document unit based on a preset embedding model to obtain a vectorized document unit; thereby storing the vectorized document unit in a second database; for example, the second database may be Milvus, which can support semantic retrieval.

[0164] In some embodiments of the present application, the preset embedding model may be a model that has been fine-tuned in a vertical field, and its performance may be better than the model before fine-tuning. When selecting a preset embedding model, a model with a lower dimensionality may be given priority, which can reduce the retrieval load of the database and thus improve the efficiency of question and answering.

[0165] In some embodiments of the present application, the preset database may also include a fourth database, which can be used to store document units and context information corresponding to the document units; the type of the fourth database is not limited in this application, for example, the fourth database can adopt the relational database management system MySQL, thereby supporting complex queries and data association analysis.

[0166] In some embodiments of the present application, the electronic device may generate a first password corresponding to the processed document data; and then store the document unit corresponding to the processed document data and the context information of the document unit in the processed document data in a fourth database according to the first password.

[0167] In some embodiments of the present application, the first password can be generated using a message digest algorithm (MD5 Message-Digest Algorithm, MD5); the electronic device can generate an identifier for each document through MD5, so that each document unit corresponding to each document is stored according to the identifier of the document.

[0168] In some embodiments of the present application, the electronic device may also obtain question-answer pair data; perform expansion processing on the question-answer pair data based on a preset expansion model to obtain expanded question-answer pair data; and then store the expanded question-answer pair data in the first database.

[0169] In an embodiment of the present application, the expansion process may include at least one of a semantic expansion process, a synonym replacement process, and an intent recognition process.

[0170] In an embodiment of the present application, the preset extension model can be any one of Claude3.5, GPT-4o and Qwen2; through semantic expansion processing, a single semantic can be expanded into other semantics, and through synonym replacement processing, the expressions in the question and answer pair data can be replaced by synonyms to obtain a variety of different expressions. Intent recognition processing can be used to determine the intention or purpose of the questions in the question and answer pair data, thereby improving the ability to understand the questions input by users and improving the relevance and accuracy of recall processing.

[0171] In some embodiments of the present application, the electronic device may also use the first language model to perform query extraction processing on the document unit to obtain query data corresponding to the document unit; then construct documents and query data based on the document unit and the query data, and store the documents and query data in the first database.

[0172] In an embodiment of the present application, multiple potential queries can be extracted from each document unit based on the first language model, thereby forming multiple documents and query data, enhancing the understanding and matching capabilities of the document content, enabling the electronic device to obtain multi-angle query input from the same document, and improving flexibility and accuracy in processing complex documents.

[0173] In summary, the present application proposes a multi-way recall mechanism, which can, upon receiving a question request, first use the data in a preset knowledge base to perform recall processing on the question request for related questions to obtain first question information related to the question request, and then select second question information with a first correlation parameter greater than a first threshold in the first question information. If there is second question information that meets the conditions in the first question information, it can continue to determine whether these second question information come from the question-answer pair data in the preset knowledge base. If so, the answer information corresponding to the second question information can be directly output as the answer data, and if not, the recall processing of related documents is continued based on the second question information to determine the answer data; and when the recall of the question and the recall of the document cannot obtain an effective answer, the first language model is used as a fallback strategy to generate the answer data; thereby, through the constructed preset knowledge base, in conjunction with the recall processing of related questions and the recall of related documents, it is ensured that relevant information is obtained from multiple angles, the accuracy and comprehensiveness of the recall are improved, and the accuracy and reliability of the question and answer can be greatly improved.

[0174] An embodiment of the present application provides a question-and-answer method, in which an electronic device can respond to a question request and perform question recall processing on the question request based on data in a preset database to obtain first question information; in a case where second question information exists in the first question information, determine whether the second question information is question information in a first data source; wherein the second question information represents question information in the first question information in which a first correlation parameter is greater than a first threshold; the first correlation parameter represents a parameter of the correlation between the first question information and the question request; the first data source represents question-and-answer pair data in a preset database; in a case where the second question information is question information in the first data source, determine answer information corresponding to the second question information as answer data for the question request; in a case where the second question information is not question information in the first data source, perform document recall processing based on the second question information to determine answer data. It can be seen that when the electronic device receives a question request, it can first use the data in the preset database to perform question recall processing on the question request to obtain first question information related to the question request, and then select second question information with a first correlation parameter greater than the first threshold in the first question information. If there is second question information that meets the conditions in the first question information, it can continue to determine whether these second question information come from the question and answer pair data in the preset database. If so, the answer information corresponding to the second question information can be directly output as the answer data. If not, the document recall processing is continued based on the second question information to determine the answer data. Therefore, through the constructed preset database, in conjunction with the recall processing of questions and the recall of documents, the accuracy and reliability of questions and answers can be greatly improved.

[0175] Based on the above embodiment, in another embodiment of the present application, illustratively, as Figure 5 As shown, when constructing a preset database, the document 22 uploaded by the user 21 can be stored in the third database 23; at the same time, the document 22 is subjected to data cleaning processing to obtain processed document data 24, and then the document unit 25 is obtained based on the processed document data 24; the document unit 25 is stored in the first database 26, and the document unit is vectorized using the preset embedding model 27, and the vectorized document unit is stored in the second database 28, and the context information corresponding to the document unit can also be stored in the fourth database 29; in addition, question and answer pair data can be obtained, the question and answer pair data can be expanded to obtain expanded question and answer pair data 210, and then the question and answer pair data 210 can be stored in the first database.

[0176] In the embodiment of the present application, recall processing of question requests can be understood as symmetrical recall, while recall processing of documents can be understood as asymmetrical recall.

[0177] In an embodiment of the present application, when storing different data in each database, due to the large scale of the data, for example, the amount of data may range from hundreds to millions, and considering the huge scale of the entire offline calculation, it is necessary to adopt a strategy of parallel document segmentation and parallel vectorized reasoning to inject each data in batches in parallel into the first database, the second database, the third database, and the fourth database.

[0178] In an embodiment of the present application, each database in the preset database can be regularly checked and updated. For example, document data can be searched according to annotation semantics, and then corresponding checks and updates can be performed to ensure the accuracy and timeliness of data and information. At the same time, the data in the database can be monitored autonomously. If there are anomalies or errors in the data, the latest correct data can be downloaded autonomously to ensure the accuracy and reliability of the data and information.

[0179] In an embodiment of the present application, an inverted index may be constructed based on the data stored in the first database. The inverted index may index documents based on the frequency and position of occurrence of keywords, and is suitable for processing exact matches and keyword queries.

[0180] In some embodiments of the present application, when using the BM25 algorithm to calculate the similarity between questions and question requests in the inverted index, the question request can be first decomposed using a word segmenter, for example, using the ik_max_word word segmenter of the word segmentation toolkit IK Analyzer, which is suitable for processing Chinese text and can extract richer keywords; the text is decomposed into as many words as possible using the word segmenter to obtain a list of terms corresponding to the question request, and then calculation and scoring are performed based on the BM25 algorithm, and the scoring results are sorted and output.

[0181] In some embodiments of the present application, an index can also be built using the vectorized question-answer pair data and the vectorized document and query data in the second database to support semantically related retrieval; and then, when similarity is calculated based on the vectorized data in the second database and the vectorized question request, the cosine similarity calculation method can be used to measure the similarity between the two.

[0182] In an embodiment of the present application, the overall recall process of the question-answering method can be roughly divided into two parts, wherein the first part is the recall processing of questions and the recall processing of documents, and the second part is determining the answer data based on the first language model; that is, when the recall processing based on questions and the recall processing based on documents cannot obtain the answer data, the first language model can be used as a fallback answer strategy to obtain the answer data, which can ensure that relevant answers can be provided even when conventional recall fails. The entire recall process achieves efficient and accurate question recall and document processing by combining document- and vector-based retrieval technology with first language model processing.

[0183] In some embodiments of the present application, the hardware of the electronic device can also be selected and deployed to improve the performance of the electronic device when processing the question-answering method; for example, the electronic device can deploy a high-performance computing (HPC) cluster application, including setting up multiple high-performance processors to improve the parallel processing capability of the electronic device; a high transmission rate interface can also be deployed to achieve a high transmission rate for data, reduce communication delays, and optimize parallel computing performance; a distributed file system can also be deployed to support high throughput and low latency data access requirements; in addition, the electronic device can also be deployed with a graphics processor accelerator and a tensor processing unit accelerator. The graphics processor accelerator can provide powerful floating-point computing capabilities, thereby accelerating the reasoning process of the deep learning model, and the tensor processing unit accelerator can improve the processing efficiency of the first language model; the electronic device can also be deployed with a distributed memory cache system to optimize the read and write performance of the inverted index and vector index through its high-speed key-value access capability, while achieving larger memory capacity and better data persistence.

[0184] For example, Figure 6 As shown, the question-answering method may include the following steps:

[0185] Step 301: respond to a question input by a user.

[0186] Step 302: Calculate the correlation between the input question and the question data in the first database.

[0187] Step 303: Get 10 related questions.

[0188] Step 304: vectorize the 10 related questions and the input question.

[0189] Step 305: Calculate the similarity between the 10 vectorized related questions and the input question.

[0190] In this embodiment, similarity calculation may be performed based on a cosine similarity calculation method.

[0191] Step 306: Determine whether there is a problem where the similarity is greater than a fourth threshold.

[0192] Step 307: vectorize the input question.

[0193] Step 308: perform similarity calculation on the vectorized questions and the vectorized question data in the second database to determine 10 related questions.

[0194] In this embodiment, similarity calculation may be performed based on a cosine similarity calculation method.

[0195] Step 309: Determine whether there is a problem where the similarity is greater than a fifth threshold.

[0196] Step 310: Integrate the two-way recall problem.

[0197] Step 311: reorder the recalled questions.

[0198] Step 312: Whether there is any problem with a correlation parameter greater than a first threshold among the reordered problems.

[0199] Step 313: Whether it is a problem in the first data source.

[0200] Step 314: Use the answer information corresponding to the question as answer data

[0201] Step 315: Perform context expansion processing based on the document unit corresponding to the question.

[0202] Step 316: Analyze the expanded document unit and the input question based on the second language model to determine answer data.

[0203] Step 317: Perform relevance evaluation on the input questions and the document data in the preset database to determine 10 questions.

[0204] In this embodiment, the BM25 algorithm may be used to perform correlation evaluation processing.

[0205] Step 318: Whether there is any question among the 10 questions whose similarity is greater than a second threshold.

[0206] Step 319: reorder the questions whose values ​​are greater than the second threshold.

[0207] Step 320: Whether there is a problem with a correlation parameter greater than a third threshold among the reordered problems.

[0208] Step 321: Determine answer data based on the first language model, the input question, and the historical interaction data.

[0209] Step 322: Output answer data.

[0210] In some embodiments of the present application, a first rearrangement model and a second rearrangement model can be trained based on a deep learning algorithm, so that the first rearrangement model can be used to reorder the recalled questions, and the second rearrangement model can be used to reorder the questions greater than a second threshold, thereby ensuring that the most relevant and accurate content is prioritized and improving the quality of the answers.

[0211] In summary, the question-answering method provided by this application can effectively integrate and utilize knowledge resources in different fields, and provide users with a structured and easy-to-retrieve high-quality knowledge base, which not only enhances the coverage and depth of the knowledge base, but also improves the accessibility and availability of data through diversified data storage strategies. The multi-channel knowledge recall mechanism can maximize the coverage of professional knowledge fields, achieve a deep understanding and precise matching of user queries, and significantly improve the accuracy, comprehensiveness and adaptability of information retrieval; and by combining the first language model with detailed contextual data, it can achieve more accurate answer generation. For some professional fields, even when the multi-channel recall method cannot obtain an answer, it can provide the most relevant and high-quality answers.

[0212] An embodiment of the present application provides a question-and-answer method, in which an electronic device can respond to a question request and perform question recall processing on the question request based on data in a preset database to obtain first question information; in a case where second question information exists in the first question information, determine whether the second question information is question information in a first data source; wherein the second question information represents question information in the first question information in which a first correlation parameter is greater than a first threshold; the first correlation parameter represents a parameter of the correlation between the first question information and the question request; the first data source represents question-and-answer pair data in a preset database; in a case where the second question information is question information in the first data source, determine answer information corresponding to the second question information as answer data for the question request; in a case where the second question information is not question information in the first data source, perform document recall processing based on the second question information to determine answer data. It can be seen that when the electronic device receives a question request, it can first use the data in the preset database to perform question recall processing on the question request to obtain first question information related to the question request, and then select second question information with a first correlation parameter greater than the first threshold in the first question information. If there is second question information that meets the conditions in the first question information, it can continue to determine whether these second question information come from the question and answer pair data in the preset database. If so, the answer information corresponding to the second question information can be directly output as the answer data. If not, the document recall processing is continued based on the second question information to determine the answer data. Therefore, through the constructed preset database, in conjunction with the recall processing of questions and the recall of documents, the accuracy and reliability of questions and answers can be greatly improved.

[0213] Based on the above embodiment, in another embodiment of the present application, an electronic device is provided, such as Figure 7 As shown, the electronic device 1 may include a first recall unit 11 , a determination unit 12 , a second recall unit 13 and a construction unit 14 .

[0214] The first recall unit 11 can be used to respond to a question request and perform a question recall process on the question request based on data in a preset database to obtain first question information; wherein the preset database is used to provide data for recall processing; the preset database is constructed by acquiring document data to be processed, performing data cleaning processing on the document data to be processed, obtaining processed document data, and based on the processed document data; the data cleaning processing is used to remove erroneous data in the document data to be processed.

[0215] The determination unit 12 can be used to determine whether the second question information is the question information in the first data source when the second question information exists in the first question information; wherein the second question information represents the question information in the first question information, in which the first correlation parameter is greater than the first threshold; the first correlation parameter represents the parameter of the correlation between the first question information and the question request; the first data source represents the question and answer pair data in the preset database; and when the second question information is the question information in the first data source, the answer information corresponding to the second question information is determined as the answer data of the question request.

[0216] The second recall unit 13 may be used to perform a document recall process based on the second question information to determine answer data when the second question information is not the question information in the first data source.

[0217] In some embodiments of the present application, the determination unit 12 may also be used for the first recall unit 11 to respond to a question request, perform question recall processing on the question request based on data in a preset database, and after obtaining the first question information, if the second question information does not exist in the first question information, perform relevance evaluation processing based on the question request and document data in the preset database to determine third question information; wherein the third question information represents question information generated based on the document data and whose relevance to the question request meets the first relevance condition; if the fourth question information exists in the third question information, reorder the fourth question information to obtain reordered question information, and determine answer data based on the reordered question information; wherein the fourth question information represents question information in the third question information whose similarity with the question request is greater than a second threshold.

[0218] In some embodiments of the present application, the determination unit 12 can also be used to determine whether there is fifth question information in the reordered question information; wherein the fifth question information represents question information in the reordered question information, in which the second correlation parameter is greater than the third threshold, and the second correlation parameter represents a parameter of the correlation between the reordered question information and the question request.

[0219] In some embodiments of the present application, the second recall unit 13 may also be used to perform a document recall process based on the fifth question information to determine answer data when the fifth question information exists in the reordered question information.

[0220] In some embodiments of the present application, the determination unit 12 may also be configured to determine answer data based on the first language model, the question request, and the historical interaction data when the fifth question information does not exist in the reordered question information.

[0221] In some embodiments of the present application, the determination unit 12 may also be configured to determine answer data based on the first language model, the question request, and the historical interaction data when the fourth question information does not exist in the third question information.

[0222] In some embodiments of the present application, the preset database includes a first database and a second database; the first question information includes sixth question information and seventh question information; the determination unit 12 can also be used to perform similarity calculation on the question data in the question and answer pair data and the document and query data stored in the first database and the question request to obtain a first similarity result, and determine the sixth question information from the question data based on the first similarity result; and to vectorize the question request to obtain a vectorized question request; and to perform similarity calculation on the vectorized question data and the vectorized question request to obtain a second similarity result, and determine the seventh question information from the vectorized question data based on the second similarity result; wherein the vectorized question data is obtained based on the vectorized question and answer pair data and the vectorized document and query data stored in the second database.

[0223] In some embodiments of the present application, the determination unit 12 can also be used to select a first number of eighth question information from the question data based on the first similarity result; and perform vectorization processing on the eighth question information to obtain vectorized eighth question information; and perform similarity calculation on the vectorized eighth question information and the vectorized question request to obtain a third similarity result, and determine the question information in the third similarity result whose similarity information is greater than a fourth threshold as the sixth question information.

[0224] In some embodiments of the present application, the second recall unit 13 can also be used to perform context expansion processing based on the document unit corresponding to the second question information to obtain an expanded document unit; and analyze and process the expanded document unit and the question request based on the second language model to obtain answer data.

[0225] The construction unit 14 can be used to respond to the question request in the first recall unit 11, perform question recall processing on the question request based on the data in the preset database, and obtain the document data to be processed before obtaining the first question information; and perform data cleaning processing on the document data to be processed to obtain the processed document data; and construct the preset database based on the processed document data; wherein the data cleaning processing can be used to remove erroneous data in the document data to be processed.

[0226] In some embodiments of the present application, the preset database includes a third database; the construction unit 14 can also be used to perform document segmentation processing on the processed document data based on the format of the processed document data to obtain document units corresponding to the processed document data; and store the document units and the first information of the document units in the first database; wherein the first information includes at least one of the source information, author information and date information of the document unit; and store the processed document data in the third database to complete the construction of the preset database.

[0227] In some embodiments of the present application, the construction unit 14 may also be used to perform vectorization processing on the document unit based on a preset embedding model to obtain a vectorized document unit; and store the vectorized document unit in the second database.

[0228] In some embodiments of the present application, the preset database also includes a fourth database; the construction unit 14 can also be used to generate a first password corresponding to the processed document data; and store the document unit corresponding to the processed document data and the context information of the document unit in the processed document data in the fourth database according to the first password.

[0229] In some embodiments of the present application, the construction unit 14 can also be used to obtain question-answer pair data; and perform expansion processing on the question-answer pair data based on a preset expansion model to obtain expanded question-answer pair data; wherein the expansion processing includes at least one of semantic expansion processing, synonym replacement processing, and intent recognition processing; and store the expanded question-answer pair data in the first database.

[0230] In some embodiments of the present application, the construction unit 14 can also be used to use the first language model to perform query extraction processing on the document unit to obtain query data corresponding to the document unit; and to construct documents and query data based on the document unit and the query data, and store the documents and query data in the first database.

[0231] In the embodiments of the present application, further, Figure 8 Schematic diagram of the composition structure of the electronic device provided in the embodiment of the present application Figure 2 ,like Figure 8As shown, the electronic device 1 provided in the embodiment of the present application may further include a processor 15 and a memory 16 storing executable instructions of the processor 15; further, the electronic device 1 may further include a communication interface 17, and a bus 18 for connecting the processor 15, the memory 16 and the communication interface 17.

[0232] In the embodiment of the present application, the processor 15 may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the above-mentioned processor function may also be other, and the embodiment of the present application is not specifically limited. The electronic device 10 may also include a memory 16, which may be connected to the processor 15, wherein the memory 16 is used to store executable program code, the program code includes computer operation instructions, and the memory 16 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0233] In the embodiment of the present application, the bus 18 is used to connect the communication interface 17, the processor 15 and the memory 16, and the mutual communication between these devices.

[0234] In the embodiment of the present application, the memory 16 is used to store instructions and data.

[0235] Further, in an embodiment of the present application, the above-mentioned processor 15 is used to respond to a question request, perform question recall processing on the question request based on data in a preset database, and obtain first question information; the preset database is constructed based on the processed document data by acquiring document data to be processed, performing data cleaning processing on the document data to be processed, obtaining processed document data; the data cleaning processing is used to remove erroneous data in the document data to be processed; in the case where there is second question information in the first question information, determine whether the second question information is question information in the first data source; wherein the second question information represents question information in the first question information, in which the first correlation parameter is greater than the first threshold; the first correlation parameter represents a parameter of the correlation between the first question information and the question request; the first data source represents the question and answer pair data in the preset database; in the case where the second question information is question information in the first data source, determine the answer information corresponding to the second question information as the answer data of the question request.

[0236] In practical applications, the memory 16 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 15.

[0237] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0238] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0239] An embodiment of the present application provides an electronic device, which includes a first recall unit and a determination unit; the first recall unit is used to respond to a question request and perform question recall processing on the question request based on data in a preset database to obtain first question information; the determination unit is used to determine whether the second question information is question information in a first data source when the second question information exists in the first question information; wherein the second question information represents question information in the first question information whose first correlation parameter is greater than a first threshold; the first correlation parameter represents a parameter of the correlation between the first question information and the question request; the first data source represents question-answer pair data in a preset database; and when the second question information is question information in the first data source, the answer information corresponding to the second question information is determined as the answer data of the question request. It can be seen that when the electronic device receives a question request, it can first use the data in the preset database to perform question recall processing on the question request to obtain first question information related to the question request, and then select second question information whose first correlation parameter is greater than the first threshold in the first question information. If there is second question information that meets the conditions in the first question information, it can continue to determine whether these second question information come from the question and answer pair data in the preset database. If so, the answer information corresponding to the second question information can be directly output as the answer data. Therefore, through the constructed preset database, the accuracy and reliability of the question and answer can be greatly improved in combination with the question recall processing.

[0240] Specifically, the program instructions corresponding to a question-and-answer method in this embodiment may be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to a question-and-answer method in the storage medium are read or executed by an electronic device, the following steps are included:

[0241] In response to the question request, the question request is recalled based on the data in the preset database to obtain the first question information; the preset database is constructed by acquiring the document data to be processed, performing data cleaning processing on the document data to be processed, obtaining the processed document data; the data cleaning processing is used to remove the erroneous data in the document data to be processed;

[0242] In the case where the second question information exists in the first question information, determining whether the second question information is question information in the first data source; wherein the second question information represents question information in the first question information, in which the first correlation parameter is greater than the first threshold; the first correlation parameter represents a parameter of correlation between the first question information and the question request; the first data source represents question-answer pair data in a preset database;

[0243] In the case where the second question information is question information in the first data source, answer information corresponding to the second question information is determined as answer data of the question request.

[0244] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0245] The present application is described with reference to the implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0246] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0247] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0248] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. A question-answering method, characterized in that: The method comprises: In response to the question request, recall processing is performed on the question request based on data in a preset database to obtain first question information; wherein the preset database is used to provide data for recall processing; the preset database is constructed by acquiring document data to be processed, performing data cleaning processing on the document data to be processed, obtaining processed document data, and based on the processed document data; the data cleaning processing is used to remove erroneous data in the document data to be processed; In the case where the second question information exists in the first question information, determining whether the second question information is question information in the first data source; wherein the second question information represents question information in the first question information whose first correlation parameter is greater than a first threshold; the first correlation parameter represents a parameter of correlation between the first question information and the question request; the first data source represents question-answer pair data in the preset database; In the case where the second question information is question information in the first data source, determining answer information corresponding to the second question information as answer data of the question request; In the case that the second question information does not exist in the first question information, performing relevance evaluation processing on the question request and the document data in the preset database, and determining the third question information according to the relevance results ranked first by the second number in the obtained relevance evaluation results; In the case where the fourth question information exists in the third question information, the fourth question information is reordered to obtain reordered question information; the fourth question information represents question information in the third question information whose similarity with the question request is greater than a second threshold; In a case where the fifth question information does not exist in the reordered question information, the answer data is determined based on the first language model, the question request, and historical interaction data corresponding to the user who inputs the question request; the fifth question information represents question information in the reordered question information whose second correlation parameter is greater than a third threshold, and the second correlation parameter represents a parameter of the correlation between the reordered question information and the question request.

2. The question-answering method according to claim 1, characterized in that: The method further comprises: In a case where the fifth question information exists in the reordered question information, a document recall process is performed based on the fifth question information to determine the answer data.

3. The question-answering method according to claim 2, characterized in that: The method further comprises: In a case where the fourth question information does not exist in the third question information, the answer data is determined based on the first language model, the question request, and the historical interaction data.

4. The question-answering method according to any one of claims 1 to 3, characterized in that: The preset database includes a first database and a second database; the first question information includes sixth question information and seventh question information; The step of performing question recall processing on the question request based on the data in the preset database to obtain first question information includes: performing similarity calculation on the question data in the question-answer pair data and the document and query data stored in the first database and the question request to obtain a first similarity result, and determining the sixth question information from the question data based on the first similarity result; Performing vectorization processing on the question request to obtain a vectorized question request; Perform similarity calculation on the vectorized question data and the vectorized question request to obtain a second similarity result, and determine the seventh question information from the vectorized question data based on the second similarity result; wherein the vectorized question data is obtained based on the vectorized question-answer pair data and vectorized document and query data stored in the second database.

5. The question-answering method according to claim 4, characterized in that: The determining the sixth question information from the question data based on the first similarity result includes: selecting a first number of eighth question information from the question data based on the first similarity result; Performing vectorization processing on the eighth question information to obtain vectorized eighth question information; A similarity calculation is performed on the vectorized eighth question information and the vectorized question request to obtain a third similarity result, and question information with similarity information greater than a fourth threshold in the third similarity result is determined as the sixth question information.

6. The question-answering method according to claim 1, characterized in that: The method further comprises: In the case where the second question information is not the question information in the first data source, a document recall process is performed based on the second question information to determine the answer data.

7. The question-answering method according to claim 6, characterized in that: The recalling of the document based on the second question information to determine the answer data includes: Performing context expansion processing on the document unit corresponding to the second question information to obtain an expanded document unit; The expanded document unit and the question request are analyzed and processed based on the second language model to obtain the answer data.

8. The question-answering method according to claim 1, characterized in that: The preset database includes a third database; the method further includes: Performing document segmentation processing on the processed document data based on the format of the processed document data to obtain document units corresponding to the processed document data; storing the document unit and first information of the document unit in a first database; wherein the first information includes at least one of source information, author information and date information of the document unit; The processed document data is stored in the third database to complete the construction of the preset database.

9. The question-answering method according to claim 8, characterized in that: The method further comprises: Performing vectorization processing on the document unit based on a preset embedding model to obtain a vectorized document unit; The vectorized document unit is stored in a second database.

10. The question-answering method according to claim 9, characterized in that: The preset database also includes a fourth database; the method also includes: generating a first password corresponding to the processed document data; The document unit corresponding to the processed document data and the context information of the document unit in the processed document data are stored in a fourth database according to the first password.

11. The question-answering method according to any one of claims 8 to 10, characterized in that: The method further comprises: Get question-answer pair data; Performing expansion processing on the question-answer pair data based on a preset expansion model to obtain expanded question-answer pair data; wherein the expansion processing includes at least one of semantic expansion processing, synonym replacement processing, and intention recognition processing; The expanded question-answer pair data is stored in the first database.

12. The question-answering method according to claim 11, characterized in that: The method further comprises: Performing query extraction processing on the document unit using the first language model to obtain query data corresponding to the document unit; Documents and query data are constructed according to the document units and the query data, and the documents and query data are stored in the first database.

13. An electronic device, characterized in that: The electronic device comprises a first recall unit and a determination unit; The first recall unit is used to respond to the question request and perform question recall processing on the question request based on data in a preset database to obtain first question information; The preset database is constructed by acquiring document data to be processed, performing data cleaning on the document data to be processed, obtaining processed document data, and constructing the database based on the processed document data; the data cleaning is used to remove erroneous data in the document data to be processed; The determination unit is used to determine whether the second question information is question information in the first data source when the second question information exists in the first question information; wherein the second question information represents question information in the first question information whose first correlation parameter is greater than a first threshold; the first correlation parameter represents a parameter of the correlation between the first question information and the question request; the first data source represents question-answer pair data in the preset database; and when the second question information is question information in the first data source, determining the answer information corresponding to the second question information as the answer data of the question request; and when the second question information does not exist in the first question information, performing correlation evaluation processing on the question request and the document data in the preset database, and obtaining the result according to the correlation parameter. The third question information is determined by the top second number of correlation results in the correlation evaluation results; and in the case where the fourth question information exists in the third question information, the fourth question information is reordered to obtain the reordered question information; the fourth question information represents the question information in the third question information whose similarity with the question request is greater than a second threshold; and in the case where the fifth question information does not exist in the reordered question information, the answer data is determined based on the first language model, the question request and the historical interaction data corresponding to the user who inputs the question request; the fifth question information represents the question information in the reordered question information whose second correlation parameter is greater than the third threshold, and the second correlation parameter represents the parameter of the correlation between the reordered question information and the question request.

14. An electronic device, characterized in that: The electronic device includes a processor and a memory storing instructions executable by the processor; when the instructions are executed by the processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images