Private domain document question and answer method, device, electronic device and storage medium

Abstract

The present disclosure relates to a private domain document question-and-answer method, device, electronic device and storage medium, which determine a private domain knowledge base including a keyword database and a vector database, wherein the keyword database and the vector database respectively include keywords and vectors with corresponding segmented documents. User question information is extracted to obtain a keyword list and a question vector. According to the keyword list and the question vector, matching is performed in the keyword database and the vector database respectively to obtain keyword matching results and vector matching results to determine the relevant segmented documents that match the user question information. Finally, a search is performed based on the relevant segmented documents through a pre-trained large language model to obtain answer information corresponding to the user question information. The present disclosure improves the matching degree between the search results and the questions by jointly searching for keywords and vectors, thereby ensuring that more reliable answer information is obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a private domain document question-and-answer method, device, electronic device, and storage medium. Background Art

[0002] In the current information age, large language models have significantly improved work efficiency and life convenience in common scenarios. These models, such as GPT-3, have the ability to automatically understand and generate natural language texts through deep learning technology. They are widely used in fields such as automatic text generation, automatic translation, intelligent customer service, and virtual assistants, and have significantly improved the interactivity between people and computer systems. Since the advent of ChatGPT, large language models have demonstrated breakthrough achievements in the field of natural language understanding and generation, with powerful language understanding and creation capabilities, as well as rich common sense, leading the wave of generative artificial intelligence.

[0003] However, current large language models are trained on large-scale Internet corpora and have language understanding and generation capabilities in general domains. In private domains, such as confidential information and commercial competition, the direct application of these models is limited, and even in these private domains, the limitations of large language models may be further magnified. On the one hand, the "hallucination" phenomenon of large language models is essentially caused by the domain difference between training data and user questions. Private domain data is not involved in model training, which leads to the aggravation of the "hallucination" phenomenon when large language models are applied to private domains. On the other hand, since private domain data has high real-time characteristics, and there is a gap between the knowledge reserve of general large language models and real-time information, general large language models often find it difficult to provide accurate answers in private domains. Summary of the invention

[0004] In view of this, the present disclosure proposes a private domain document question and answer method, device, electronic device and storage medium, aiming to improve the accuracy of answers obtained when conducting private domain document question and answer through a large language model and the consistency of questions and answers.

[0005] According to a first aspect of the present disclosure, a private domain document question-answering method is provided, the method comprising:

[0006] Determine a private domain knowledge base including a keyword database and a vector database, wherein the keyword database includes a plurality of keywords having corresponding shard documents, and the vector database includes a plurality of vectors having corresponding shard documents;

[0007] In response to receiving user question information, extracting a keyword list consisting of a corresponding plurality of question keywords and a corresponding question vector;

[0008] According to the keyword list and the question vector, matching is performed in a keyword database and a vector database respectively to obtain a keyword matching result and a vector matching result;

[0009] Determine relevant segment documents matching the user question information according to the keyword matching result and the vector matching result;

[0010] The pre-trained large language model is used to perform retrieval based on the relevant segmented documents to obtain answer information corresponding to the user's question information.

[0011] In a possible implementation, determining a private domain knowledge base including a keyword database and a vector database includes:

[0012] determining a private domain corpus comprising a plurality of private domain documents;

[0013] Slice each of the private domain documents to obtain a plurality of slice documents;

[0014] Perform keyword extraction and vectorization processing on each of the shard documents according to the pre-trained language model to obtain a vector and at least one keyword corresponding to each of the shard documents;

[0015] Determine a keyword database according to the keywords corresponding to the plurality of shard documents, and determine a vector database according to the vectors corresponding to the plurality of shard documents;

[0016] A private domain knowledge base is determined according to the keyword database and the vector database.

[0017] In one possible implementation, the keyword matching result includes the keyword similarity between the keyword list and at least one keyword corresponding to each of the shard documents in the keyword database, and the vector matching result includes the vector similarity between the question vector and the vector corresponding to each of the shard documents in the vector database.

[0018] In a possible implementation, determining the relevant segment documents matching the user question information according to the keyword matching result and the vector matching result includes:

[0019] Calculate a weighted sum of at least one keyword similarity of each of the shard documents in the keyword matching result and a vector similarity in the vector matching result to obtain a corresponding shard document matching degree;

[0020] Determine relevant segment documents matching the user question information from the plurality of segment documents according to the corresponding segment document matching degrees.

[0021] In a possible implementation, the retrieving based on the relevant segmented documents using the pre-trained large language model to obtain answer information corresponding to the user's question information includes:

[0022] Perform the following steps multiple times in an iterative manner:

[0023] Determine at least one current relevant shard document;

[0024] Input all relevant segment documents into a pre-trained large language model for retrieval to obtain at least one question-answer response, each of which has a corresponding relevant segment document;

[0025] Performing consistency evaluation on each of the question-answer responses to obtain a corresponding evaluation score;

[0026] In response to satisfying a preset stop condition, the iteration process is terminated and the question-and-answer reply with the highest evaluation score is determined as the answer information corresponding to the user's question information;

[0027] In response to not meeting the preset stop condition, the relevant shard documents corresponding to the question and answer replies whose evaluation scores are less than the preset threshold are deleted.

[0028] In a possible implementation, performing consistency evaluation on each of the question-answer responses to obtain a corresponding evaluation score includes:

[0029] The question-and-answer reply, at least one corresponding relevant segment document, and the user question information are input into the trained natural language model, and a corresponding evaluation score is output.

[0030] In a possible implementation, the stopping condition is that a preset number of iterations is reached, or there is no evaluation score less than the preset threshold.

[0031] According to a second aspect of the present disclosure, a private domain document question-answering device is provided, the device comprising:

[0032] A knowledge base determination module, used to determine a private domain knowledge base including a keyword database and a vector database, wherein the keyword database includes a plurality of keywords having corresponding shard documents, and the vector database includes a plurality of vectors having corresponding shard documents;

[0033] An information extraction module, configured to extract a keyword list consisting of a plurality of corresponding question keywords and a corresponding question vector in response to receiving user question information;

[0034] A data search module, used to perform matching in a keyword database and a vector database according to the keyword list and the question vector, respectively, to obtain a keyword matching result and a vector matching result;

[0035] A document determination module, configured to determine relevant segment documents matching the user question information according to the keyword matching result and the vector matching result;

[0036] The answer retrieval module is used to retrieve the relevant segmented documents based on the pre-trained large language model to obtain the answer information corresponding to the user's question information.

[0037] In a possible implementation, the knowledge base determination module is further used to:

[0038] determining a private domain corpus comprising a plurality of private domain documents;

[0039] Slice each of the private domain documents to obtain a plurality of slice documents;

[0040] Perform keyword extraction and vectorization processing on each of the shard documents according to the pre-trained language model to obtain a vector and at least one keyword corresponding to each of the shard documents;

[0041] Determine a keyword database according to the keywords corresponding to the plurality of shard documents, and determine a vector database according to the vectors corresponding to the plurality of shard documents;

[0042] A private domain knowledge base is determined according to the keyword database and the vector database.

[0043] In one possible implementation, the keyword matching result includes the keyword similarity between the keyword list and at least one keyword corresponding to each of the shard documents in the keyword database, and the vector matching result includes the vector similarity between the question vector and the vector corresponding to each of the shard documents in the vector database.

[0044] In a possible implementation, the document determination module is further configured to:

[0045] Calculate a weighted sum of at least one keyword similarity of each of the shard documents in the keyword matching result and a vector similarity in the vector matching result to obtain a corresponding shard document matching degree;

[0046] Determine relevant segment documents matching the user question information from the plurality of segment documents according to the corresponding segment document matching degrees.

[0047] In a possible implementation, the answer retrieval module is further used to:

[0048] Perform the following steps multiple times in an iterative manner:

[0049] Determine at least one current relevant shard document;

[0050] Input all relevant segment documents into a pre-trained large language model for retrieval to obtain at least one question-answer response, each of which has a corresponding relevant segment document;

[0051] Performing consistency evaluation on each of the question-answer responses to obtain a corresponding evaluation score;

[0052] In response to satisfying a preset stop condition, the iteration process is terminated and the question-and-answer reply with the highest evaluation score is determined as the answer information corresponding to the user's question information;

[0053] In response to not meeting the preset stop condition, the relevant shard documents corresponding to the question and answer replies whose evaluation scores are less than the preset threshold are deleted.

[0054] In a possible implementation, the answer retrieval module is further used to:

[0055] The question-and-answer reply, at least one corresponding relevant segment document, and the user question information are input into the trained natural language model, and a corresponding evaluation score is output.

[0056] In a possible implementation, the stopping condition is that a preset number of iterations is reached, or there is no evaluation score less than the preset threshold.

[0057] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0058] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0059] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0060] In an embodiment of the present disclosure, a private domain knowledge base including a keyword database and a vector database is determined, and the keyword database and the vector database respectively include keywords and vectors with corresponding shard documents. User question information is extracted to obtain a keyword list and a question vector. Matching is performed in the keyword database and the vector database according to the keyword list and the question vector, respectively, to obtain keyword matching results and vector matching results, so as to determine the relevant shard documents that match the user question information. Finally, a search is performed based on the relevant shard documents through a pre-trained large language model to obtain answer information corresponding to the user question information. The present disclosure improves the matching degree between the retrieval results and the questions by jointly retrieving keywords and vectors, thereby ensuring that more reliable answer information is obtained, and further improves the consistency between the answer information and the user question information by performing multiple rounds of consistency evaluation in the process of retrieving the answer information.

[0061] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0063] Figure 1 A flowchart showing a private domain document question-answering method according to an embodiment of the present disclosure is shown;

[0064] Figure 2 A schematic diagram showing a consistency evaluation process according to an embodiment of the present disclosure;

[0065] Figure 3 A schematic diagram showing a method of determining an evaluation score according to an embodiment of the present disclosure is shown;

[0066] Figure 4 A schematic diagram showing a private domain document question-answering process according to an embodiment of the present disclosure;

[0067] Figure 5 A schematic diagram showing a private domain document question-and-answer device according to an embodiment of the present disclosure;

[0068] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0069] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0070] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0071] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0072] The private domain document question and answer method of the disclosed embodiment can be any fixed or mobile terminal such as a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the private domain document question and answer method of the disclosed embodiment by calling the computer-readable instructions stored in the memory by the processor.

[0073] Figure 1 FIG. 1 is a flowchart of a private domain document question-answering method according to an embodiment of the present disclosure. Figure 1 As shown, the private domain document question and answer method of the embodiment of the present disclosure may include the following steps S10-S50.

[0074] Step S10: Determine a private domain knowledge base including a keyword database and a vector database.

[0075] The private domain usually includes areas that are not open to the outside world, such as internal documents of an enterprise, private documents of users, etc. In one possible implementation, a private domain knowledge base is determined by an electronic device, which includes a keyword database and a vector database. The keyword database includes multiple keywords with corresponding shard documents, and the vector database includes multiple vectors with corresponding shard documents. Optionally, multiple keywords in the keyword database may correspond to the same shard document, and each vector in the vector database corresponds to a different shard document. The private domain knowledge base can be pre-constructed or constructed by an electronic device during the execution of the private domain document question-and-answer method.

[0076] Optionally, the construction process of the private domain knowledge base can be to determine a private domain corpus including multiple private domain documents, and to perform segmentation processing on each private domain document to obtain multiple segmented documents. Furthermore, keyword extraction and vectorization processing are performed on each segmented document according to the language model obtained by pre-training to obtain a high-dimensional vector and at least one keyword corresponding to each segmented document that records its semantic information. A keyword database is determined based on the keywords corresponding to the multiple segmented documents, and a vector database is determined based on the vectors corresponding to the multiple segmented documents, and then the private domain knowledge base is determined based on the keyword database and the vector database. Among them, the keyword extraction process and the vector extraction process can be to input the segmented documents into the trained language model, and extract keywords and semantic feature vectors by the language model. Alternatively, it can also be implemented by any existing extraction method, which will not be repeated here.

[0077] Step S20: In response to receiving the user's question information, extract a keyword list consisting of a corresponding plurality of question keywords and a corresponding question vector.

[0078] In a possible implementation, when receiving user question information, the electronic device performs keyword extraction and vector extraction on the user question information to obtain a keyword list consisting of multiple question keywords and a question vector. Optionally, the keyword list and question vector are determined in the same way as the keyword and vector of the segmented document are extracted in step S10. That is, the electronic device can perform keyword extraction on the user question information to obtain a keyword list consisting of multiple keywords. At the same time, the user question information is vectorized to obtain a high-dimensional vector that records the semantic information therein as a question vector.

[0079] Step S30: Matching is performed in a keyword database and a vector database according to the keyword list and the question vector, respectively, to obtain a keyword matching result and a vector matching result.

[0080] In a possible implementation, after the electronic device extracts the user's question information to obtain a keyword list and a question vector, it can match the keyword list in the keyword database to obtain a keyword matching result. And match the question vector in the vector database to obtain a vector matching result. Optionally, the keyword matching result includes the keyword similarity between the keyword list and at least one keyword corresponding to each shard document in the keyword database, and the vector matching result includes the vector similarity between the question vector and the vector corresponding to each shard document in the vector database. In other words, the keyword matching process can be to obtain a keyword combination by obtaining at least one keyword corresponding to each shard document in the keyword database, and calculate the similarity between each keyword combination and the keyword list to obtain the corresponding keyword similarity. The vector matching process can be to calculate the vector similarity between the question vector and each vector in the vector database.

[0081] Optionally, the keyword matching process can be combined with keyword character matching and semantic matching to perform similarity calculation, for example, the formula Calculate the similarity between the keyword list and the keyword combination consisting of multiple keywords corresponding to each shard document. Among them, d is used to represent the shard document for which the similarity is currently calculated, and Q is used to represent the user's question information. kw (d, Q) is the keyword similarity, KW Q is a keyword list, KW d is a keyword combination consisting of multiple keywords corresponding to a shard document, α and β are character matching and semantic matching weight parameters, which can be adjusted according to actual conditions. V(.) is a vector function, which is used to convert a keyword list or keyword combination into a corresponding vector in the above formula, and cosine_Similarity() represents a function used to calculate the cosine similarity of two vectors.

[0082] Furthermore, the vector matching process can be to calculate the similarity between the question vector and the corresponding vector of each shard document, for example, by formula S vector (d, Q) = Cosine_Similarity(V(d), V(Q)) calculates the similarity S between the vectors V(Q) and V(d) corresponding to the user question information Q and the fragment document d respectively vector (d, Q), cosine_Similarity() represents a function used to calculate the cosine similarity between two vectors.

[0083] After keyword matching and vector matching, the electronic device obtains a keyword matching result including a keyword list and keyword similarity of keywords corresponding to each segment document, and a vector matching result including a vector similarity of a question vector and a vector corresponding to each segment document.

[0084] Step S40: Determine relevant segment documents matching the user question information according to the keyword matching result and the vector matching result.

[0085] In a possible implementation, after determining the keyword matching results and the vector matching results, the electronic device searches for at least one relevant segmented document with the highest matching degree with the user's question information from multiple segmented documents. Optionally, the electronic device may first calculate the weighted sum of at least one keyword similarity of each segmented document in the keyword matching results and the vector similarity in the vector matching results to obtain the corresponding segmented document matching degree. Then, the relevant segmented document matching the user's question information is determined from the multiple segmented documents based on the corresponding segmented document matching degree. That is, for each segmented document d, the keyword similarity corresponding to its keyword list with the user's question information Q is Skw (d, Q), and the vector similarity with the question vector is S vector (d, Q), by formula S combined (d, Q) = w vector ·S vector (d,Q)+w kw ·S kw (d, Q) Calculate the segment document matching degree S of the segment file and the user's question information combined (d, Q), where w vector and w kw The weights may be preset to balance the contributions of vector similarity and keyword similarity.

[0086] Furthermore, after calculating the segment document matching degrees between the plurality of segment documents and the user's question information, a preset number of segment documents with the largest segment document matching degrees may be obtained as relevant segment documents matching the user's question information.

[0087] Step S50: retrieve the relevant segmented documents using a pre-trained large language model to obtain answer information corresponding to the user's question information.

[0088] In a possible implementation, after the electronic device searches for multiple relevant segmented documents that are most relevant to the user's question information, it retrieves the results based on the multiple relevant segmented documents through the pre-trained large language model to obtain the answer information corresponding to the user's question information. In order to ensure that the answer information output by the final large language model is consistent with the user's question information input by the user, in the process of determining the answer information, the embodiment of the present disclosure can use natural language reasoning technology to perform consistency evaluation on the user questions, retrieval results and model replies in the retrieval enhancement process. In order to improve the granularity of the evaluation, the embodiment of the present disclosure can perform iterative multiple retrieval evaluations based on an open source natural language reasoning model (such as TRUE), so as to perform consistency evaluation on each retrieval result and corresponding reply involved in the reply during each iteration, and further eliminate the detected irrelevant content. Finally, through multiple rounds of iterations and multiple calls to the large language model to generate replies, the reply with the highest consistency is selected as the final answer.

[0089] That is to say, the process of retrieving the answer information corresponding to the user's question information through the large language model in the embodiment of the present disclosure can be to perform the following steps multiple times in an iterative manner: determine at least one current relevant segmented document, input all relevant segmented documents into the pre-trained large language model for retrieval, and obtain at least one question and answer reply, each question and answer reply has a corresponding relevant segmented document. Perform consistency evaluation on each question and answer reply to obtain a corresponding evaluation score. When the preset stop condition is met, the iterative process is terminated and the question and answer reply with the highest evaluation score is determined as the answer information corresponding to the user's question information. When the preset stop condition is not met, the relevant segmented documents corresponding to the question and answer reply with an evaluation score less than a preset threshold are deleted. Among them, the method of performing consistency evaluation on the question and answer reply can be to input the question and answer reply and the corresponding at least one relevant segmented document and the user's question information into the trained natural language model, and output the corresponding evaluation score. The preset stop condition can be to stop the iterative process when the preset number of iterations is reached, or to stop the iterative process when there is no evaluation score less than the preset threshold.

[0090] Figure 2 A schematic diagram of a consistency evaluation process according to an embodiment of the present disclosure is shown. Figure 2 As shown, multiple searches are performed iteratively based on the relevant segmented documents (references). During each iteration, at least one question and answer response (paragraph) retrieved by the large language model has a corresponding relevant segmented document (reference). The natural language inference model is used to perform consistency evaluation based on the question and answer response, the corresponding relevant segmented documents and the user's question information to obtain the evaluation score (support level) corresponding to each question and answer response. Then, based on the result of the consistency evaluation and the current number of iterations, it is decided whether to stop the iteration process. If it does not stop, the relevant segmented documents corresponding to the question and answer responses with lower evaluation scores are deleted and the next iteration is entered, that is, the execution is returned to "determine the current at least one relevant segmented document, and input all relevant segmented documents into the pre-trained large language model for retrieval". At this time, "at least one relevant segmented document" is the relevant segmented document remaining after deleting the relevant segmented documents corresponding to the question and answer responses with lower evaluation scores. If it stops, the question and answer response with the highest evaluation score is selected as the answer information for the user's question information.

[0091] Figure 3 FIG. 2 is a schematic diagram showing a method for determining an evaluation score according to an embodiment of the present disclosure. Figure 3 As shown, all the responses Ri generated by the i-th iteration process can be divided into n question-answer responses (response paragraphs): Answers to questions about the existence of relevant sharding documents (references) K related shard documents and question-answer responses can be constructed The reference relationship set between The disclosed embodiment can use the natural language inference model to use the F1 score to evaluate the consistency of each question-answer response in a reply, and determine whether the reference in the reference set supports the corresponding question-answer response during the evaluation process to obtain the precision and recall of each response. The F1 score can be calculated by the formula Implementation, where citation precision is the precision of the relevant sharded documents, and citation recall is the recall of the relevant sharded documents. Optionally, the TRUE natural language inference model can be used in the disclosed embodiment to assist in completing the question-answer response consistency evaluation. TRUE is an inference model fine-tuned on the T5 model using a natural language inference dataset, which is used to automatically evaluate the factual consistency of natural language.

[0092] Figure 4 FIG. 1 is a schematic diagram showing a private domain document question-answering process according to an embodiment of the present disclosure. Figure 4 As shown, when answering user question information of private domain documents through a large language model, the electronic device first determines a private domain knowledge base including a keyword database and a vector database based on the private domain document, and extracts a keyword list consisting of multiple question keywords and a corresponding question vector when receiving the user's question information. Furthermore, the keyword list and the question vector are respectively matched in the keyword database and the vector database for combined retrieval to obtain at least one relevant segmented document matching the user's question information. Finally, multiple retrievals are iterated based on the relevant segmented documents through the pre-trained large language model, and the consistency evaluation of the question and answer response is performed after each retrieval. The next iteration is performed according to the evaluation results until the answer information corresponding to the user's question information is obtained after the iteration.

[0093] Based on the above technical features, the disclosed embodiment can use vectors and keywords for combined retrieval in the process of private domain question-answering, integrate the results of vector retrieval and keyword retrieval, select the most relevant document fragments, and ensure the accuracy of the final output answer. At the same time, multiple rounds of retrieval and the use of natural language inference technology after each round of retrieval to evaluate the consistency of user questions, retrieval results and model responses ensure the consistency of the final output results with the questions. Combining the above features, the disclosed embodiment achieves efficient and accurate answers to private domain questions based on a large language model, significantly improving the efficiency and quality of information acquisition.

[0094] Figure 5 A schematic diagram of a private domain document question-answering device according to an embodiment of the present disclosure is shown. Figure 5 As shown, the private domain document question-and-answer device in the embodiment of the present disclosure may include:

[0095] A knowledge base determination module 50, configured to determine a private domain knowledge base including a keyword database and a vector database, wherein the keyword database includes a plurality of keywords having corresponding shard documents, and the vector database includes a plurality of vectors having corresponding shard documents;

[0096] An information extraction module 51 is used to extract a keyword list consisting of a plurality of corresponding question keywords and a corresponding question vector in response to receiving user question information;

[0097] A data search module 52, configured to perform matching in a keyword database and a vector database according to the keyword list and the question vector, respectively, to obtain a keyword matching result and a vector matching result;

[0098] A document determination module 53, configured to determine relevant segment documents matching the user question information according to the keyword matching result and the vector matching result;

[0099] The answer retrieval module 54 is used to retrieve the relevant segmented documents based on the pre-trained large language model to obtain the answer information corresponding to the user's question information.

[0100] In a possible implementation, the knowledge base determination module 50 is further configured to:

[0101] determining a private domain corpus comprising a plurality of private domain documents;

[0102] Slice each of the private domain documents to obtain a plurality of slice documents;

[0103] Perform keyword extraction and vectorization processing on each of the shard documents according to the pre-trained language model to obtain a vector and at least one keyword corresponding to each of the shard documents;

[0104] Determine a keyword database according to the keywords corresponding to the plurality of shard documents, and determine a vector database according to the vectors corresponding to the plurality of shard documents;

[0105] A private domain knowledge base is determined according to the keyword database and the vector database.

[0106] In one possible implementation, the keyword matching result includes the keyword similarity between the keyword list and at least one keyword corresponding to each of the shard documents in the keyword database, and the vector matching result includes the vector similarity between the question vector and the vector corresponding to each of the shard documents in the vector database.

[0107] In a possible implementation, the document determination module 53 is further configured to:

[0108] Calculate a weighted sum of at least one keyword similarity of each of the shard documents in the keyword matching result and a vector similarity in the vector matching result to obtain a corresponding shard document matching degree;

[0109] Determine relevant segment documents matching the user question information from the plurality of segment documents according to the corresponding segment document matching degrees.

[0110] In a possible implementation, the answer retrieval module 54 is further configured to:

[0111] Perform the following steps multiple times in an iterative manner:

[0112] Determine at least one current relevant shard document;

[0113] Input all relevant segment documents into a pre-trained large language model for retrieval to obtain at least one question-answer response, each of which has a corresponding relevant segment document;

[0114] Performing consistency evaluation on each of the question-answer responses to obtain a corresponding evaluation score;

[0115] In response to satisfying a preset stop condition, the iteration process is terminated and the question-and-answer reply with the highest evaluation score is determined as the answer information corresponding to the user's question information;

[0116] In response to not meeting the preset stop condition, the relevant shard documents corresponding to the question and answer replies whose evaluation scores are less than the preset threshold are deleted.

[0117] In a possible implementation, the answer retrieval module 54 is further configured to:

[0118] The question-and-answer reply, at least one corresponding relevant segment document, and the user question information are input into the trained natural language model, and a corresponding evaluation score is output.

[0119] In a possible implementation, the stopping condition is that a preset number of iterations is reached, or there is no evaluation score less than the preset threshold.

[0120] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0121] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0122] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0123] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0124] Figure 6 1 is a schematic diagram showing an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 6 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0125] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0126] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0127] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0128] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0129] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0130] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0131] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0134] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0135] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A private domain document question-answering method, characterized in that: The method comprises: Determine a private domain knowledge base including a keyword database and a vector database, wherein the keyword database includes a plurality of keywords having corresponding shard documents, and the vector database includes a plurality of vectors having corresponding shard documents; In response to receiving user question information, extracting a keyword list consisting of a corresponding plurality of question keywords and a corresponding question vector; According to the keyword list and the question vector, matching is performed in the keyword database and the vector database respectively to obtain a keyword matching result and a vector matching result; the keyword matching result includes the keyword similarity between the keyword list and at least one keyword corresponding to each of the shard documents in the keyword database, and the vector matching result includes the vector similarity between the question vector and the vector corresponding to each of the shard documents in the vector database; Determine relevant segment documents matching the user question information according to the keyword matching result and the vector matching result; Retrieve the relevant segmented documents through a pre-trained large language model to obtain answer information corresponding to the user's question information; The determining, according to the keyword matching result and the vector matching result, the relevant segment documents matching the user question information includes: Calculate a weighted sum of at least one keyword similarity of each of the shard documents in the keyword matching result and a vector similarity in the vector matching result to obtain a corresponding shard document matching degree; Determining relevant segment documents matching the user question information from the plurality of segment documents according to the corresponding segment document matching degrees; The pre-trained large language model is used to retrieve the relevant segmented documents to obtain answer information corresponding to the user's question information, including: Perform the following steps multiple times in an iterative manner: Determine at least one current relevant shard document; Input all relevant segment documents into a pre-trained large language model for retrieval to obtain at least one question-answer response, each of which has a corresponding relevant segment document; Performing consistency evaluation on each of the question-answer responses to obtain a corresponding evaluation score; In response to satisfying a preset stop condition, the iteration process is terminated and the question-and-answer reply with the highest evaluation score is determined as the answer information corresponding to the user's question information; In response to not meeting the preset stop condition, the relevant shard documents corresponding to the question and answer replies whose evaluation scores are less than the preset threshold are deleted.

2. The method according to claim 1, characterized in that The determining of a private domain knowledge base including a keyword database and a vector database comprises: determining a private domain corpus comprising a plurality of private domain documents; Slice each of the private domain documents to obtain a plurality of slice documents; Perform keyword extraction and vectorization processing on each of the shard documents according to the pre-trained language model to obtain a vector and at least one keyword corresponding to each of the shard documents; Determine a keyword database according to the keywords corresponding to the plurality of shard documents, and determine a vector database according to the vectors corresponding to the plurality of shard documents; A private domain knowledge base is determined according to the keyword database and the vector database.

3. The method according to claim 1, characterized in that The consistency evaluation of each question and answer response is performed to obtain a corresponding evaluation score, including: The question-and-answer reply, at least one corresponding relevant segment document, and the user question information are input into the trained natural language model, and a corresponding evaluation score is output.

4. The method according to any one of claims 1 to 3, characterized in that The stopping condition is that a preset number of iterations is reached, or there is no evaluation score less than the preset threshold.

5. A private domain document question and answer device, characterized in that: The device comprises: A knowledge base determination module, used to determine a private domain knowledge base including a keyword database and a vector database, wherein the keyword database includes a plurality of keywords having corresponding shard documents, and the vector database includes a plurality of vectors having corresponding shard documents; An information extraction module, configured to extract a keyword list consisting of a plurality of corresponding question keywords and a corresponding question vector in response to receiving user question information; A data search module, used to perform matching in a keyword database and a vector database according to the keyword list and the question vector, respectively, to obtain a keyword matching result and a vector matching result; the keyword matching result includes a keyword similarity between the keyword list and at least one keyword corresponding to each of the shard documents in the keyword database, and the vector matching result includes a vector similarity between the question vector and a vector corresponding to each of the shard documents in the vector database; A document determination module, configured to determine relevant segment documents matching the user question information according to the keyword matching result and the vector matching result; An answer retrieval module is used to retrieve the relevant segmented documents based on the pre-trained large language model to obtain answer information corresponding to the user's question information; The document determination module is further used for: Calculate a weighted sum of at least one keyword similarity of each of the shard documents in the keyword matching result and a vector similarity in the vector matching result to obtain a corresponding shard document matching degree; Determining relevant segment documents matching the user question information from the plurality of segment documents according to the corresponding segment document matching degrees; The answer retrieval module is further used for: Perform the following steps multiple times in an iterative manner: Determine at least one current relevant shard document; Input all relevant segment documents into a pre-trained large language model for retrieval to obtain at least one question-answer response, each of which has a corresponding relevant segment document; Performing consistency evaluation on each of the question-answer responses to obtain a corresponding evaluation score; In response to satisfying a preset stop condition, the iteration process is terminated and the question-and-answer reply with the highest evaluation score is determined as the answer information corresponding to the user's question information; In response to not meeting the preset stop condition, the relevant shard documents corresponding to the question and answer replies whose evaluation scores are less than the preset threshold are deleted.

6. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 4 when executing the instructions stored in the memory.

7. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images